Internal representation (IR) compiler and evaluator (#13330)

# Description

This PR adds an internal representation language to Nushell, offering an
alternative evaluator based on simple instructions, stream-containing
registers, and indexed control flow. The number of registers required is
determined statically at compile-time, and the fixed size required is
allocated upon entering the block.

Each instruction is associated with a span, which makes going backwards
from IR instructions to source code very easy.

Motivations for IR:

1. **Performance.** By simplifying the evaluation path and making it
more cache-friendly and branch predictor-friendly, code that does a lot
of computation in Nushell itself can be sped up a decent bit. Because
the IR is fairly easy to reason about, we can also implement
optimization passes in the future to eliminate and simplify code.
2. **Correctness.** The instructions mostly have very simple and
easily-specified behavior, so hopefully engine changes are a little bit
easier to reason about, and they can be specified in a more formal way
at some point. I have made an effort to document each of the
instructions in the docs for the enum itself in a reasonably specific
way. Some of the errors that would have happened during evaluation
before are now moved to the compilation step instead, because they don't
make sense to check during evaluation.
3. **As an intermediate target.** This is a good step for us to bring
the [`new-nu-parser`](https://github.com/nushell/new-nu-parser) in at
some point, as code generated from new AST can be directly compared to
code generated from old AST. If the IR code is functionally equivalent,
it will behave the exact same way.
4. **Debugging.** With a little bit more work, we can probably give
control over advancing the virtual machine that `IrBlock`s run on to
some sort of external driver, making things like breakpoints and single
stepping possible. Tools like `view ir` and [`explore
ir`](https://github.com/devyn/nu_plugin_explore_ir) make it easier than
before to see what exactly is going on with your Nushell code.

The goal is to eventually replace the AST evaluator entirely, once we're
sure it's working just as well. You can help dogfood this by running
Nushell with `$env.NU_USE_IR` set to some value. The environment
variable is checked when Nushell starts, so config runs with IR, or it
can also be set on a line at the REPL to change it dynamically. It is
also checked when running `do` in case within a script you want to just
run a specific piece of code with or without IR.

# Example

```nushell
view ir { |data|
  mut sum = 0
  for n in $data {
    $sum += $n
  }
  $sum
}
```
  
```gas
# 3 registers, 19 instructions, 0 bytes of data
   0: load-literal           %0, int(0)
   1: store-variable         var 904, %0 # let
   2: drain                  %0
   3: drop                   %0
   4: load-variable          %1, var 903
   5: iterate                %0, %1, end 15 # for, label(1), from(14:)
   6: store-variable         var 905, %0
   7: load-variable          %0, var 904
   8: load-variable          %2, var 905
   9: binary-op              %0, Math(Plus), %2
  10: span                   %0
  11: store-variable         var 904, %0
  12: load-literal           %0, nothing
  13: drain                  %0
  14: jump                   5
  15: drop                   %0          # label(0), from(5:)
  16: drain                  %0
  17: load-variable          %0, var 904
  18: return                 %0
```

# Benchmarks

All benchmarks run on a base model Mac Mini M1.

## Iterative Fibonacci sequence

This is about as best case as possible, making use of the much faster
control flow. Most code will not experience a speed improvement nearly
this large.

```nushell
def fib [n: int] {
  mut a = 0
  mut b = 1
  for _ in 2..=$n {
    let c = $a + $b
    $a = $b
    $b = $c
  }
  $b
}
use std bench
bench { 0..50 | each { |n| fib $n } }
```

IR disabled:

```
╭───────┬─────────────────╮
│ mean  │ 1ms 924µs 665ns │
│ min   │ 1ms 700µs 83ns  │
│ max   │ 3ms 450µs 125ns │
│ std   │ 395µs 759ns     │
│ times │ [list 50 items] │
╰───────┴─────────────────╯
```

IR enabled:

```
╭───────┬─────────────────╮
│ mean  │ 452µs 820ns     │
│ min   │ 427µs 417ns     │
│ max   │ 540µs 167ns     │
│ std   │ 17µs 158ns      │
│ times │ [list 50 items] │
╰───────┴─────────────────╯
```

![explore ir
view](https://github.com/nushell/nushell/assets/10729/d7bccc03-5222-461c-9200-0dce71b83b83)

##
[gradient_benchmark_no_check.nu](https://github.com/nushell/nu_scripts/blob/main/benchmarks/gradient_benchmark_no_check.nu)

IR disabled:

```
╭───┬──────────────────╮
│ 0 │ 27ms 929µs 958ns │
│ 1 │ 21ms 153µs 459ns │
│ 2 │ 18ms 639µs 666ns │
│ 3 │ 19ms 554µs 583ns │
│ 4 │ 13ms 383µs 375ns │
│ 5 │ 11ms 328µs 208ns │
│ 6 │  5ms 659µs 542ns │
╰───┴──────────────────╯
```

IR enabled:

```
╭───┬──────────────────╮
│ 0 │       22ms 662µs │
│ 1 │ 17ms 221µs 792ns │
│ 2 │ 14ms 786µs 708ns │
│ 3 │ 13ms 876µs 834ns │
│ 4 │  13ms 52µs 875ns │
│ 5 │ 11ms 269µs 666ns │
│ 6 │  6ms 942µs 500ns │
╰───┴──────────────────╯
```

##
[random-bytes.nu](https://github.com/nushell/nu_scripts/blob/main/benchmarks/random-bytes.nu)

I got pretty random results out of this benchmark so I decided not to
include it. Not clear why.

# User-Facing Changes
- IR compilation errors may appear even if the user isn't evaluating
with IR.
- IR evaluation can be enabled by setting the `NU_USE_IR` environment
variable to any value.
- New command `view ir` pretty-prints the IR for a block, and `view ir
--json` can be piped into an external tool like [`explore
ir`](https://github.com/devyn/nu_plugin_explore_ir).

# Tests + Formatting
All tests are passing with `NU_USE_IR=1`, and I've added some more eval
tests to compare the results for some very core operations. I will
probably want to add some more so we don't have to always check
`NU_USE_IR=1 toolkit test --workspace` on a regular basis.

# After Submitting
- [ ] release notes
- [ ] further documentation of instructions?
- [ ] post-release: publish `nu_plugin_explore_ir`

Cargo.lock

@@ -2900,6 +2900,7 @@ dependencies = [
  "openssl",
  "pretty_assertions",
  "reedline",
+ "regex",
  "rstest",
  "serde_json",
  "serial_test",
@@ -3151,6 +3152,7 @@ dependencies = [
 name = "nu-engine"
 version = "0.95.1"
 dependencies = [
+ "log",
  "nu-glob",
  "nu-path",
  "nu-protocol",
@@ -3339,6 +3341,7 @@ dependencies = [
  "convert_case",
  "fancy-regex",
  "indexmap",
+ "log",
  "lru",
  "miette",
  "nix",

Cargo.toml

@@ -232,6 +232,7 @@ assert_cmd = "2.0"
 dirs-next = { workspace = true }
 tango-bench = "0.5"
 pretty_assertions = { workspace = true }
+regex = { workspace = true }
 rstest = { workspace = true, default-features = false }
 serial_test = "3.1"
 tempfile = { workspace = true }
@@ -310,4 +311,4 @@ reedline = { git = "https://github.com/nushell/reedline", branch = "main" }
 # Run individual benchmarks like `cargo bench -- <regex>` e.g. `cargo bench -- parse`
 [[bench]]
 name = "benchmarks"
-harness = false
+harness = false

benches/benchmarks.rs

@@ -45,6 +45,10 @@ fn setup_stack_and_engine_from_command(command: &str) -> (Stack, EngineState) {
     };
 
     let mut stack = Stack::new();
+
+    // Support running benchmarks with IR mode
+    stack.use_ir = std::env::var_os("NU_USE_IR").is_some();
+
     evaluate_commands(
         &commands,
         &mut engine,

crates/nu-cli/src/commands/keybindings_list.rs

@@ -49,22 +49,24 @@ impl Command for KeybindingsList {
 
     fn run(
         &self,
-        _engine_state: &EngineState,
-        _stack: &mut Stack,
+        engine_state: &EngineState,
+        stack: &mut Stack,
         call: &Call,
         _input: PipelineData,
     ) -> Result<PipelineData, ShellError> {
-        let records = if call.named_len() == 0 {
-            let all_options = ["modifiers", "keycodes", "edits", "modes", "events"];
-            all_options
-                .iter()
-                .flat_map(|argument| get_records(argument, call.head))
-                .collect()
-        } else {
-            call.named_iter()
-                .flat_map(|(argument, _, _)| get_records(argument.item.as_str(), call.head))
-                .collect()
-        };
+        let all_options = ["modifiers", "keycodes", "edits", "modes", "events"];
+
+        let presence = all_options
+            .iter()
+            .map(|option| call.has_flag(engine_state, stack, option))
+            .collect::<Result<Vec<_>, ShellError>>()?;
+
+        let records = all_options
+            .iter()
+            .zip(presence)
+            .filter(|(_, present)| *present)
+            .flat_map(|(option, _)| get_records(option, call.head))
+            .collect();
 
         Ok(Value::list(records, call.head).into_pipeline_data())
     }

crates/nu-cli/src/eval_cmds.rs

@@ -70,6 +70,11 @@ pub fn evaluate_commands(
             std::process::exit(1);
         }
 
+        if let Some(err) = working_set.compile_errors.first() {
+            report_error(&working_set, err);
+            // Not a fatal error, for now
+        }
+
         (output, working_set.render())
     };
 

crates/nu-cli/src/repl.rs

@@ -268,6 +268,9 @@ fn loop_iteration(ctx: LoopContext) -> (bool, Stack, Reedline) {
     if let Err(err) = engine_state.merge_env(&mut stack, cwd) {
         report_error_new(engine_state, &err);
     }
+    // Check whether $env.NU_USE_IR is set, so that the user can change it in the REPL
+    // Temporary while IR eval is optional
+    stack.use_ir = stack.has_env_var(engine_state, "NU_USE_IR");
     perf!("merge env", start_time, use_color);
 
     start_time = std::time::Instant::now();

crates/nu-cli/src/util.rs

@@ -262,6 +262,11 @@ fn evaluate_source(
             return Ok(Some(1));
         }
 
+        if let Some(err) = working_set.compile_errors.first() {
+            report_error(&working_set, err);
+            // Not a fatal error, for now
+        }
+
         (output, working_set.render())
     };
 

crates/nu-cmd-lang/src/core_commands/const_.rs

@@ -46,6 +46,9 @@ impl Command for Const {
         call: &Call,
         _input: PipelineData,
     ) -> Result<PipelineData, ShellError> {
+        // This is compiled specially by the IR compiler. The code here is never used when
+        // running in IR mode.
+        let call = call.assert_ast_call()?;
         let var_id = if let Some(id) = call.positional_nth(0).and_then(|pos| pos.as_var()) {
             id
         } else {

crates/nu-cmd-lang/src/core_commands/do_.rs

@@ -81,6 +81,10 @@ impl Command for Do {
 
         bind_args_to(&mut callee_stack, &block.signature, rest, head)?;
         let eval_block_with_early_return = get_eval_block_with_early_return(engine_state);
+
+        // Applies to all block evaluation once set true
+        callee_stack.use_ir = caller_stack.has_env_var(engine_state, "NU_USE_IR");
+
         let result = eval_block_with_early_return(engine_state, &mut callee_stack, block, input);
 
         if has_env {

crates/nu-cmd-lang/src/core_commands/for_.rs

@@ -48,6 +48,9 @@ impl Command for For {
         call: &Call,
         _input: PipelineData,
     ) -> Result<PipelineData, ShellError> {
+        // This is compiled specially by the IR compiler. The code here is never used when
+        // running in IR mode.
+        let call = call.assert_ast_call()?;
         let head = call.head;
         let var_id = call
             .positional_nth(0)

crates/nu-cmd-lang/src/core_commands/if_.rs

@@ -60,6 +60,9 @@ impl Command for If {
         call: &Call,
         input: PipelineData,
     ) -> Result<PipelineData, ShellError> {
+        // This is compiled specially by the IR compiler. The code here is never used when
+        // running in IR mode.
+        let call = call.assert_ast_call()?;
         let cond = call.positional_nth(0).expect("checked through parser");
         let then_block = call
             .positional_nth(1)
@@ -99,6 +102,9 @@ impl Command for If {
         call: &Call,
         input: PipelineData,
     ) -> Result<PipelineData, ShellError> {
+        // This is compiled specially by the IR compiler. The code here is never used when
+        // running in IR mode.
+        let call = call.assert_ast_call()?;
         let cond = call.positional_nth(0).expect("checked through parser");
         let then_block = call
             .positional_nth(1)

crates/nu-cmd-lang/src/core_commands/let_.rs

@@ -46,6 +46,9 @@ impl Command for Let {
         call: &Call,
         input: PipelineData,
     ) -> Result<PipelineData, ShellError> {
+        // This is compiled specially by the IR compiler. The code here is never used when
+        // running in IR mode.
+        let call = call.assert_ast_call()?;
         let var_id = call
             .positional_nth(0)
             .expect("checked through parser")

crates/nu-cmd-lang/src/core_commands/loop_.rs

@@ -37,6 +37,9 @@ impl Command for Loop {
         call: &Call,
         _input: PipelineData,
     ) -> Result<PipelineData, ShellError> {
+        // This is compiled specially by the IR compiler. The code here is never used when
+        // running in IR mode.
+        let call = call.assert_ast_call()?;
         let head = call.head;
         let block_id = call
             .positional_nth(0)

crates/nu-cmd-lang/src/core_commands/match_.rs

@@ -43,6 +43,9 @@ impl Command for Match {
         call: &Call,
         input: PipelineData,
     ) -> Result<PipelineData, ShellError> {
+        // This is compiled specially by the IR compiler. The code here is never used when
+        // running in IR mode.
+        let call = call.assert_ast_call()?;
         let value: Value = call.req(engine_state, stack, 0)?;
         let matches = call
             .positional_nth(1)

crates/nu-cmd-lang/src/core_commands/mut_.rs

@@ -46,6 +46,9 @@ impl Command for Mut {
         call: &Call,
         input: PipelineData,
     ) -> Result<PipelineData, ShellError> {
+        // This is compiled specially by the IR compiler. The code here is never used when
+        // running in IR mode.
+        let call = call.assert_ast_call()?;
         let var_id = call
             .positional_nth(0)
             .expect("checked through parser")

crates/nu-cmd-lang/src/core_commands/overlay/use_.rs

@@ -65,9 +65,9 @@ impl Command for OverlayUse {
         name_arg.item = trim_quotes_str(&name_arg.item).to_string();
 
         let maybe_origin_module_id =
-            if let Some(overlay_expr) = call.get_parser_info("overlay_expr") {
+            if let Some(overlay_expr) = call.get_parser_info(caller_stack, "overlay_expr") {
                 if let Expr::Overlay(module_id) = &overlay_expr.expr {
-                    module_id
+                    *module_id
                 } else {
                     return Err(ShellError::NushellFailedSpanned {
                         msg: "Not an overlay".to_string(),
@@ -110,7 +110,7 @@ impl Command for OverlayUse {
             // a) adding a new overlay
             // b) refreshing an active overlay (the origin module changed)
 
-            let module = engine_state.get_module(*module_id);
+            let module = engine_state.get_module(module_id);
 
             // Evaluate the export-env block (if any) and keep its environment
             if let Some(block_id) = module.env_block {
@@ -118,7 +118,7 @@ impl Command for OverlayUse {
                     &name_arg.item,
                     engine_state,
                     caller_stack,
-                    get_dirs_var_from_call(call),
+                    get_dirs_var_from_call(caller_stack, call),
                 )?;
 
                 let block = engine_state.get_block(block_id);

crates/nu-cmd-lang/src/core_commands/try_.rs

@@ -47,6 +47,9 @@ impl Command for Try {
         call: &Call,
         input: PipelineData,
     ) -> Result<PipelineData, ShellError> {
+        // This is compiled specially by the IR compiler. The code here is never used when
+        // running in IR mode.
+        let call = call.assert_ast_call()?;
         let try_block = call
             .positional_nth(0)
             .expect("checked through parser")

crates/nu-cmd-lang/src/core_commands/use_.rs

@@ -57,7 +57,7 @@ This command is a parser keyword. For details, check:
         let Some(Expression {
             expr: Expr::ImportPattern(import_pattern),
             ..
-        }) = call.get_parser_info("import_pattern")
+        }) = call.get_parser_info(caller_stack, "import_pattern")
         else {
             return Err(ShellError::GenericError {
                 error: "Unexpected import".into(),
@@ -68,6 +68,9 @@ This command is a parser keyword. For details, check:
             });
         };
 
+        // Necessary so that we can modify the stack.
+        let import_pattern = import_pattern.clone();
+
         if let Some(module_id) = import_pattern.head.id {
             // Add constants
             for var_id in &import_pattern.constants {
@@ -99,7 +102,7 @@ This command is a parser keyword. For details, check:
                     &module_arg_str,
                     engine_state,
                     caller_stack,
-                    get_dirs_var_from_call(call),
+                    get_dirs_var_from_call(caller_stack, call),
                 )?;
                 let maybe_parent = maybe_file_path
                     .as_ref()

crates/nu-cmd-lang/src/core_commands/while_.rs

@@ -46,6 +46,9 @@ impl Command for While {
         call: &Call,
         _input: PipelineData,
     ) -> Result<PipelineData, ShellError> {
+        // This is compiled specially by the IR compiler. The code here is never used when
+        // running in IR mode.
+        let call = call.assert_ast_call()?;
         let head = call.head;
         let cond = call.positional_nth(0).expect("checked through parser");
         let block_id = call

crates/nu-command/src/bytes/build_.rs

@@ -49,10 +49,8 @@ impl Command for BytesBuild {
         _input: PipelineData,
     ) -> Result<PipelineData, ShellError> {
         let mut output = vec![];
-        for val in call.rest_iter_flattened(0, |expr| {
-            let eval_expression = get_eval_expression(engine_state);
-            eval_expression(engine_state, stack, expr)
-        })? {
+        let eval_expression = get_eval_expression(engine_state);
+        for val in call.rest_iter_flattened(engine_state, stack, eval_expression, 0)? {
             let val_span = val.span();
             match val {
                 Value::Binary { mut val, .. } => output.append(&mut val),

crates/nu-command/src/debug/explain.rs

@@ -1,6 +1,6 @@
 use nu_engine::{command_prelude::*, get_eval_expression};
 use nu_protocol::{
-    ast::{Argument, Block, Expr, Expression},
+    ast::{self, Argument, Block, Expr, Expression},
     engine::Closure,
 };
 
@@ -106,7 +106,7 @@ pub fn get_pipeline_elements(
 fn get_arguments(
     engine_state: &EngineState,
     stack: &mut Stack,
-    call: &Call,
+    call: &ast::Call,
     eval_expression_fn: fn(&EngineState, &mut Stack, &Expression) -> Result<Value, ShellError>,
 ) -> Vec<Value> {
     let mut arg_value = vec![];

crates/nu-command/src/debug/metadata.rs

@@ -28,6 +28,10 @@ impl Command for Metadata {
             .category(Category::Debug)
     }
 
+    fn requires_ast_for_arguments(&self) -> bool {
+        true
+    }
+
     fn run(
         &self,
         engine_state: &EngineState,
@@ -35,7 +39,7 @@ impl Command for Metadata {
         call: &Call,
         input: PipelineData,
     ) -> Result<PipelineData, ShellError> {
-        let arg = call.positional_nth(0);
+        let arg = call.positional_nth(stack, 0);
         let head = call.head;
 
         match arg {

crates/nu-command/src/debug/mod.rs

@@ -10,6 +10,7 @@ mod profile;
 mod timeit;
 mod view;
 mod view_files;
+mod view_ir;
 mod view_source;
 mod view_span;
 
@@ -25,5 +26,6 @@ pub use profile::DebugProfile;
 pub use timeit::TimeIt;
 pub use view::View;
 pub use view_files::ViewFiles;
+pub use view_ir::ViewIr;
 pub use view_source::ViewSource;
 pub use view_span::ViewSpan;

crates/nu-command/src/debug/timeit.rs

@@ -32,6 +32,10 @@ impl Command for TimeIt {
         vec!["timing", "timer", "benchmark", "measure"]
     }
 
+    fn requires_ast_for_arguments(&self) -> bool {
+        true
+    }
+
     fn run(
         &self,
         engine_state: &EngineState,
@@ -39,13 +43,14 @@ impl Command for TimeIt {
         call: &Call,
         input: PipelineData,
     ) -> Result<PipelineData, ShellError> {
-        let command_to_run = call.positional_nth(0);
+        // reset outdest, so the command can write to stdout and stderr.
+        let stack = &mut stack.push_redirection(None, None);
+
+        let command_to_run = call.positional_nth(stack, 0);
 
         // Get the start time after all other computation has been done.
         let start_time = Instant::now();
 
-        // reset outdest, so the command can write to stdout and stderr.
-        let stack = &mut stack.push_redirection(None, None);
         if let Some(command_to_run) = command_to_run {
             if let Some(block_id) = command_to_run.as_block() {
                 let eval_block = get_eval_block(engine_state);
@@ -53,7 +58,8 @@ impl Command for TimeIt {
                 eval_block(engine_state, stack, block, input)?
             } else {
                 let eval_expression_with_input = get_eval_expression_with_input(engine_state);
-                eval_expression_with_input(engine_state, stack, command_to_run, input)?.0
+                let expression = &command_to_run.clone();
+                eval_expression_with_input(engine_state, stack, expression, input)?.0
             }
         } else {
             PipelineData::empty()

crates/nu-command/src/debug/view_ir.rs

@@ -0,0 +1,83 @@
+use nu_engine::command_prelude::*;
+use nu_protocol::engine::Closure;
+
+#[derive(Clone)]
+pub struct ViewIr;
+
+impl Command for ViewIr {
+    fn name(&self) -> &str {
+        "view ir"
+    }
+
+    fn signature(&self) -> Signature {
+        Signature::new(self.name())
+            .required(
+                "closure",
+                SyntaxShape::Closure(None),
+                "The closure to see compiled code for.",
+            )
+            .switch(
+                "json",
+                "Dump the raw block data as JSON (unstable).",
+                Some('j'),
+            )
+            .input_output_type(Type::Nothing, Type::String)
+    }
+
+    fn usage(&self) -> &str {
+        "View the compiled IR code for a block of code."
+    }
+
+    fn run(
+        &self,
+        engine_state: &EngineState,
+        stack: &mut Stack,
+        call: &Call,
+        _input: PipelineData,
+    ) -> Result<PipelineData, ShellError> {
+        let closure: Closure = call.req(engine_state, stack, 0)?;
+        let json = call.has_flag(engine_state, stack, "json")?;
+
+        let block = engine_state.get_block(closure.block_id);
+        let ir_block = block
+            .ir_block
+            .as_ref()
+            .ok_or_else(|| ShellError::GenericError {
+                error: "Can't view IR for this block".into(),
+                msg: "block is missing compiled representation".into(),
+                span: block.span,
+                help: Some("the IrBlock is probably missing due to a compilation error".into()),
+                inner: vec![],
+            })?;
+
+        let formatted = if json {
+            let formatted_instructions = ir_block
+                .instructions
+                .iter()
+                .map(|instruction| {
+                    instruction
+                        .display(engine_state, &ir_block.data)
+                        .to_string()
+                })
+                .collect::<Vec<_>>();
+
+            serde_json::to_string_pretty(&serde_json::json!({
+                "block_id": closure.block_id,
+                "span": block.span,
+                "ir_block": ir_block,
+                "formatted_instructions": formatted_instructions,
+            }))
+            .map_err(|err| ShellError::GenericError {
+                error: "JSON serialization failed".into(),
+                msg: err.to_string(),
+                span: Some(call.head),
+                help: None,
+                inner: vec![],
+            })?
+        } else {
+            format!("{}", ir_block.display(engine_state))
+        };
+
+        Ok(Value::string(formatted, call.head).into_pipeline_data())
+    }
+}

crates/nu-command/src/default_context.rs

@@ -154,6 +154,7 @@ pub fn add_shell_command_context(mut engine_state: EngineState) -> EngineState {
             TimeIt,
             View,
             ViewFiles,
+            ViewIr,
             ViewSource,
             ViewSpan,
         };

crates/nu-command/src/env/export_env.rs

@@ -33,6 +33,10 @@ impl Command for ExportEnv {
         CommandType::Keyword
     }
 
+    fn requires_ast_for_arguments(&self) -> bool {
+        true
+    }
+
     fn run(
         &self,
         engine_state: &EngineState,
@@ -41,7 +45,7 @@ impl Command for ExportEnv {
         input: PipelineData,
     ) -> Result<PipelineData, ShellError> {
         let block_id = call
-            .positional_nth(0)
+            .positional_nth(caller_stack, 0)
             .expect("checked through parser")
             .as_block()
             .expect("internal error: missing block");

crates/nu-command/src/env/source_env.rs

@@ -56,7 +56,7 @@ impl Command for SourceEnv {
             &source_filename.item,
             engine_state,
             caller_stack,
-            get_dirs_var_from_call(call),
+            get_dirs_var_from_call(caller_stack, call),
         )? {
             PathBuf::from(&path)
         } else {

crates/nu-command/src/filesystem/du.rs

@@ -102,7 +102,7 @@ impl Command for Du {
         let current_dir = current_dir(engine_state, stack)?;
 
         let paths = get_rest_for_glob_pattern(engine_state, stack, call, 0)?;
-        let paths = if call.rest_iter(0).count() == 0 {
+        let paths = if !call.has_positional_args(stack, 0) {
             None
         } else {
             Some(paths)

crates/nu-command/src/filesystem/ls.rs

@@ -108,7 +108,7 @@ impl Command for Ls {
         };
 
         let pattern_arg = get_rest_for_glob_pattern(engine_state, stack, call, 0)?;
-        let input_pattern_arg = if call.rest_iter(0).count() == 0 {
+        let input_pattern_arg = if !call.has_positional_args(stack, 0) {
             None
         } else {
             Some(pattern_arg)

crates/nu-command/src/filesystem/open.rs

@@ -1,7 +1,7 @@
 use super::util::get_rest_for_glob_pattern;
 #[allow(deprecated)]
 use nu_engine::{command_prelude::*, current_dir, get_eval_block};
-use nu_protocol::{ByteStream, DataSource, NuGlob, PipelineMetadata};
+use nu_protocol::{ast, ByteStream, DataSource, NuGlob, PipelineMetadata};
 use std::path::Path;
 
 #[cfg(feature = "sqlite")]
@@ -56,7 +56,7 @@ impl Command for Open {
         let mut paths = get_rest_for_glob_pattern(engine_state, stack, call, 0)?;
         let eval_block = get_eval_block(engine_state);
 
-        if paths.is_empty() && call.rest_iter(0).next().is_none() {
+        if paths.is_empty() && !call.has_positional_args(stack, 0) {
             // try to use path from pipeline input if there were no positional or spread args
             let (filename, span) = match input {
                 PipelineData::Value(val, ..) => {
@@ -180,7 +180,8 @@ impl Command for Open {
                                 let block = engine_state.get_block(block_id);
                                 eval_block(engine_state, stack, block, stream)
                             } else {
-                                decl.run(engine_state, stack, &Call::new(call_span), stream)
+                                let call = ast::Call::new(call_span);
+                                decl.run(engine_state, stack, &(&call).into(), stream)
                             };
                             output.push(command_output.map_err(|inner| {
                                     ShellError::GenericError{

crates/nu-command/src/filesystem/save.rs

@@ -4,10 +4,8 @@ use nu_engine::get_eval_block;
 use nu_engine::{command_prelude::*, current_dir};
 use nu_path::expand_path_with;
 use nu_protocol::{
-    ast::{Expr, Expression},
-    byte_stream::copy_with_signals,
-    process::ChildPipe,
-    ByteStreamSource, DataSource, OutDest, PipelineMetadata, Signals,
+    ast, byte_stream::copy_with_signals, process::ChildPipe, ByteStreamSource, DataSource, OutDest,
+    PipelineMetadata, Signals,
 };
 use std::{
     fs::File,
@@ -69,24 +67,6 @@ impl Command for Save {
         let append = call.has_flag(engine_state, stack, "append")?;
         let force = call.has_flag(engine_state, stack, "force")?;
         let progress = call.has_flag(engine_state, stack, "progress")?;
-        let out_append = if let Some(Expression {
-            expr: Expr::Bool(out_append),
-            ..
-        }) = call.get_parser_info("out-append")
-        {
-            *out_append
-        } else {
-            false
-        };
-        let err_append = if let Some(Expression {
-            expr: Expr::Bool(err_append),
-            ..
-        }) = call.get_parser_info("err-append")
-        {
-            *err_append
-        } else {
-            false
-        };
 
         let span = call.head;
         #[allow(deprecated)]
@@ -109,14 +89,7 @@ impl Command for Save {
             PipelineData::ByteStream(stream, metadata) => {
                 check_saving_to_source_file(metadata.as_ref(), &path, stderr_path.as_ref())?;
 
-                let (file, stderr_file) = get_files(
-                    &path,
-                    stderr_path.as_ref(),
-                    append,
-                    out_append,
-                    err_append,
-                    force,
-                )?;
+                let (file, stderr_file) = get_files(&path, stderr_path.as_ref(), append, force)?;
 
                 let size = stream.known_size();
                 let signals = engine_state.signals();
@@ -221,14 +194,7 @@ impl Command for Save {
                     stderr_path.as_ref(),
                 )?;
 
-                let (mut file, _) = get_files(
-                    &path,
-                    stderr_path.as_ref(),
-                    append,
-                    out_append,
-                    err_append,
-                    force,
-                )?;
+                let (mut file, _) = get_files(&path, stderr_path.as_ref(), append, force)?;
                 for val in ls {
                     file.write_all(&value_to_bytes(val)?)
                         .map_err(|err| ShellError::IOError {
@@ -258,14 +224,7 @@ impl Command for Save {
                     input_to_bytes(input, Path::new(&path.item), raw, engine_state, stack, span)?;
 
                 // Only open file after successful conversion
-                let (mut file, _) = get_files(
-                    &path,
-                    stderr_path.as_ref(),
-                    append,
-                    out_append,
-                    err_append,
-                    force,
-                )?;
+                let (mut file, _) = get_files(&path, stderr_path.as_ref(), append, force)?;
 
                 file.write_all(&bytes).map_err(|err| ShellError::IOError {
                     msg: err.to_string(),
@@ -397,7 +356,8 @@ fn convert_to_extension(
             let eval_block = get_eval_block(engine_state);
             eval_block(engine_state, stack, block, input)
         } else {
-            decl.run(engine_state, stack, &Call::new(span), input)
+            let call = ast::Call::new(span);
+            decl.run(engine_state, stack, &(&call).into(), input)
         }
     } else {
         Ok(input)
@@ -473,19 +433,17 @@ fn get_files(
     path: &Spanned<PathBuf>,
     stderr_path: Option<&Spanned<PathBuf>>,
     append: bool,
-    out_append: bool,
-    err_append: bool,
     force: bool,
 ) -> Result<(File, Option<File>), ShellError> {
     // First check both paths
-    let (path, path_span) = prepare_path(path, append || out_append, force)?;
+    let (path, path_span) = prepare_path(path, append, force)?;
     let stderr_path_and_span = stderr_path
         .as_ref()
-        .map(|stderr_path| prepare_path(stderr_path, append || err_append, force))
+        .map(|stderr_path| prepare_path(stderr_path, append, force))
         .transpose()?;
 
     // Only if both files can be used open and possibly truncate them
-    let file = open_file(path, path_span, append || out_append)?;
+    let file = open_file(path, path_span, append)?;
 
     let stderr_file = stderr_path_and_span
         .map(|(stderr_path, stderr_path_span)| {
@@ -498,7 +456,7 @@ fn get_files(
                     inner: vec![],
                 })
             } else {
-                open_file(stderr_path, stderr_path_span, append || err_append)
+                open_file(stderr_path, stderr_path_span, append)
             }
         })
         .transpose()?;

crates/nu-command/src/filesystem/util.rs

@@ -1,6 +1,6 @@
 use dialoguer::Input;
 use nu_engine::{command_prelude::*, get_eval_expression};
-use nu_protocol::{ast::Expr, FromValue, NuGlob};
+use nu_protocol::{FromValue, NuGlob};
 use std::{
     error::Error,
     path::{Path, PathBuf},
@@ -92,42 +92,19 @@ pub fn is_older(src: &Path, dst: &Path) -> Option<bool> {
 
 /// Get rest arguments from given `call`, starts with `starting_pos`.
 ///
-/// It's similar to `call.rest`, except that it always returns NuGlob.  And if input argument has
-/// Type::Glob, the NuGlob is unquoted, which means it's required to expand.
+/// It's similar to `call.rest`, except that it always returns NuGlob.
 pub fn get_rest_for_glob_pattern(
     engine_state: &EngineState,
     stack: &mut Stack,
     call: &Call,
     starting_pos: usize,
 ) -> Result<Vec<Spanned<NuGlob>>, ShellError> {
-    let mut output = vec![];
     let eval_expression = get_eval_expression(engine_state);
 
-    for result in call.rest_iter_flattened(starting_pos, |expr| {
-        let result = eval_expression(engine_state, stack, expr);
-        match result {
-            Err(e) => Err(e),
-            Ok(result) => {
-                let span = result.span();
-                // convert from string to quoted string if expr is a variable
-                // or string interpolation
-                match result {
-                    Value::String { val, .. }
-                        if matches!(
-                            &expr.expr,
-                            Expr::FullCellPath(_) | Expr::StringInterpolation(_)
-                        ) =>
-                    {
-                        // should not expand if given input type is not glob.
-                        Ok(Value::glob(val, expr.ty != Type::Glob, span))
-                    }
-                    other => Ok(other),
-                }
-            }
-        }
-    })? {
-        output.push(FromValue::from_value(result)?);
-    }
-
-    Ok(output)
+    call.rest_iter_flattened(engine_state, stack, eval_expression, starting_pos)?
+        .into_iter()
+        // This used to be much more complex, but I think `FromValue` should be able to handle the
+        // nuance here.
+        .map(FromValue::from_value)
+        .collect()
 }

crates/nu-command/src/filters/transpose.rs

@@ -149,27 +149,27 @@ pub fn transpose(
     if !args.rest.is_empty() && args.header_row {
         return Err(ShellError::IncompatibleParametersSingle {
             msg: "Can not provide header names and use `--header-row`".into(),
-            span: call.get_named_arg("header-row").expect("has flag").span,
+            span: call.get_flag_span(stack, "header-row").expect("has flag"),
         });
     }
     if !args.header_row && args.keep_all {
         return Err(ShellError::IncompatibleParametersSingle {
             msg: "Can only be used with `--header-row`(`-r`)".into(),
-            span: call.get_named_arg("keep-all").expect("has flag").span,
+            span: call.get_flag_span(stack, "keep-all").expect("has flag"),
         });
     }
     if !args.header_row && args.keep_last {
         return Err(ShellError::IncompatibleParametersSingle {
             msg: "Can only be used with `--header-row`(`-r`)".into(),
-            span: call.get_named_arg("keep-last").expect("has flag").span,
+            span: call.get_flag_span(stack, "keep-last").expect("has flag"),
         });
     }
     if args.keep_all && args.keep_last {
         return Err(ShellError::IncompatibleParameters {
             left_message: "can't use `--keep-last` at the same time".into(),
-            left_span: call.get_named_arg("keep-last").expect("has flag").span,
+            left_span: call.get_flag_span(stack, "keep-last").expect("has flag"),
             right_message: "because of `--keep-all`".into(),
-            right_span: call.get_named_arg("keep-all").expect("has flag").span,
+            right_span: call.get_flag_span(stack, "keep-all").expect("has flag"),
         });
     }
 

crates/nu-command/src/filters/utils.rs

@@ -1,7 +1,6 @@
 use nu_engine::{CallExt, ClosureEval};
 use nu_protocol::{
-    ast::Call,
-    engine::{Closure, EngineState, Stack},
+    engine::{Call, Closure, EngineState, Stack},
     IntoPipelineData, PipelineData, ShellError, Span, Value,
 };
 

crates/nu-command/src/generators/cal.rs

@@ -1,7 +1,7 @@
 use chrono::{Datelike, Local, NaiveDate};
 use nu_color_config::StyleComputer;
 use nu_engine::command_prelude::*;
-use nu_protocol::ast::{Expr, Expression};
+use nu_protocol::ast::{self, Expr, Expression};
 
 use std::collections::VecDeque;
 
@@ -143,7 +143,7 @@ pub fn cal(
         style_computer,
     )?;
 
-    let mut table_no_index = Call::new(Span::unknown());
+    let mut table_no_index = ast::Call::new(Span::unknown());
     table_no_index.add_named((
         Spanned {
             item: "index".to_string(),
@@ -160,7 +160,12 @@ pub fn cal(
     let cal_table_output =
         Value::list(calendar_vec_deque.into_iter().collect(), tag).into_pipeline_data();
     if !arguments.as_table {
-        crate::Table.run(engine_state, stack, &table_no_index, cal_table_output)
+        crate::Table.run(
+            engine_state,
+            stack,
+            &(&table_no_index).into(),
+            cal_table_output,
+        )
     } else {
         Ok(cal_table_output)
     }

crates/nu-command/src/math/utils.rs

@@ -1,6 +1,6 @@
 use core::slice;
 use indexmap::IndexMap;
-use nu_protocol::{ast::Call, IntoPipelineData, PipelineData, ShellError, Signals, Span, Value};
+use nu_protocol::{engine::Call, IntoPipelineData, PipelineData, ShellError, Signals, Span, Value};
 
 pub fn run_with_function(
     call: &Call,

crates/nu-command/src/platform/ansi/ansi_.rs

@@ -676,7 +676,7 @@ Operating system commands:
             }
         };
 
-        let output = heavy_lifting(code, escape, osc, call)?;
+        let output = heavy_lifting(code, escape, osc, stack, call)?;
 
         Ok(Value::string(output, call.head).into_pipeline_data())
     }
@@ -713,26 +713,30 @@ Operating system commands:
             }
         };
 
-        let output = heavy_lifting(code, escape, osc, call)?;
+        let output = heavy_lifting(code, escape, osc, &Stack::new(), call)?;
 
         Ok(Value::string(output, call.head).into_pipeline_data())
     }
 }
 
-fn heavy_lifting(code: Value, escape: bool, osc: bool, call: &Call) -> Result<String, ShellError> {
+fn heavy_lifting(
+    code: Value,
+    escape: bool,
+    osc: bool,
+    stack: &Stack,
+    call: &Call,
+) -> Result<String, ShellError> {
     let param_is_string = matches!(code, Value::String { .. });
     if escape && osc {
         return Err(ShellError::IncompatibleParameters {
             left_message: "escape".into(),
             left_span: call
-                .get_named_arg("escape")
-                .expect("Unexpected missing argument")
-                .span,
+                .get_flag_span(stack, "escape")
+                .expect("Unexpected missing argument"),
             right_message: "osc".into(),
             right_span: call
-                .get_named_arg("osc")
-                .expect("Unexpected missing argument")
-                .span,
+                .get_flag_span(stack, "osc")
+                .expect("Unexpected missing argument"),
         });
     }
     let code_string = if param_is_string {
@@ -744,10 +748,7 @@ fn heavy_lifting(code: Value, escape: bool, osc: bool, call: &Call) -> Result<St
     if (escape || osc) && (param_is_valid_string) {
         let code_vec: Vec<char> = code_string.chars().collect();
         if code_vec[0] == '\\' {
-            let span = match call.get_flag_expr("escape") {
-                Some(expr) => expr.span,
-                None => call.head,
-            };
+            let span = call.get_flag_span(stack, "escape").unwrap_or(call.head);
 
             return Err(ShellError::TypeMismatch {
                 err_message: "no need for escape characters".into(),

crates/nu-command/src/platform/is_terminal.rs

@@ -58,7 +58,7 @@ impl Command for IsTerminal {
             _ => {
                 return Err(ShellError::IncompatibleParametersSingle {
                     msg: "Only one stream may be checked".into(),
-                    span: Span::merge_many(call.arguments.iter().map(|arg| arg.span())),
+                    span: call.arguments_span(),
                 });
             }
         };

crates/nu-command/src/platform/kill.rs

@@ -84,27 +84,26 @@ impl Command for Kill {
                 {
                     return Err(ShellError::IncompatibleParameters {
                         left_message: "force".to_string(),
-                        left_span: call
-                            .get_named_arg("force")
-                            .ok_or_else(|| ShellError::GenericError {
+                        left_span: call.get_flag_span(stack, "force").ok_or_else(|| {
+                            ShellError::GenericError {
                                 error: "Flag error".into(),
                                 msg: "flag force not found".into(),
                                 span: Some(call.head),
                                 help: None,
                                 inner: vec![],
-                            })?
-                            .span,
+                            }
+                        })?,
                         right_message: "signal".to_string(),
                         right_span: Span::merge(
-                            call.get_named_arg("signal")
-                                .ok_or_else(|| ShellError::GenericError {
+                            call.get_flag_span(stack, "signal").ok_or_else(|| {
+                                ShellError::GenericError {
                                     error: "Flag error".into(),
                                     msg: "flag signal not found".into(),
                                     span: Some(call.head),
                                     help: None,
                                     inner: vec![],
-                                })?
-                                .span,
+                                }
+                            })?,
                             signal_span,
                         ),
                     });

crates/nu-command/src/strings/encode_decode/base64.rs

@@ -8,8 +8,8 @@ use base64::{
 };
 use nu_cmd_base::input_handler::{operate as general_operate, CmdArgument};
 use nu_protocol::{
-    ast::{Call, CellPath},
-    engine::EngineState,
+    ast::CellPath,
+    engine::{Call, EngineState},
     PipelineData, ShellError, Span, Spanned, Value,
 };
 

crates/nu-command/src/strings/mod.rs

@@ -17,8 +17,7 @@ pub use str_::*;
 
 use nu_engine::CallExt;
 use nu_protocol::{
-    ast::Call,
-    engine::{EngineState, Stack, StateWorkingSet},
+    engine::{Call, EngineState, Stack, StateWorkingSet},
     ShellError,
 };
 

crates/nu-command/src/system/nu_check.rs

@@ -87,7 +87,7 @@ impl Command for NuCheck {
                         &path_str.item,
                         engine_state,
                         stack,
-                        get_dirs_var_from_call(call),
+                        get_dirs_var_from_call(stack, call),
                     ) {
                         Ok(path) => {
                             if let Some(path) = path {

crates/nu-command/src/system/run_external.rs

@@ -1,9 +1,7 @@
 use nu_cmd_base::hook::eval_hook;
 use nu_engine::{command_prelude::*, env_to_strings, get_eval_expression};
 use nu_path::{dots::expand_ndots, expand_tilde};
-use nu_protocol::{
-    ast::Expression, did_you_mean, process::ChildProcess, ByteStream, NuGlob, OutDest, Signals,
-};
+use nu_protocol::{did_you_mean, process::ChildProcess, ByteStream, NuGlob, OutDest, Signals};
 use nu_system::ForegroundChild;
 use nu_utils::IgnoreCaseExt;
 use pathdiff::diff_paths;
@@ -222,20 +220,21 @@ pub fn eval_arguments_from_call(
     call: &Call,
 ) -> Result<Vec<Spanned<OsString>>, ShellError> {
     let cwd = engine_state.cwd(Some(stack))?;
-    let mut args: Vec<Spanned<OsString>> = vec![];
-    for (expr, spread) in call.rest_iter(1) {
-        for arg in eval_argument(engine_state, stack, expr, spread)? {
-            match arg {
-                // Expand globs passed to run-external
-                Value::Glob { val, no_expand, .. } if !no_expand => args.extend(
-                    expand_glob(&val, cwd.as_ref(), expr.span, engine_state.signals())?
-                        .into_iter()
-                        .map(|s| s.into_spanned(expr.span)),
-                ),
-                other => {
-                    args.push(OsString::from(coerce_into_string(other)?).into_spanned(expr.span))
-                }
-            }
+    let eval_expression = get_eval_expression(engine_state);
+    let call_args = call.rest_iter_flattened(engine_state, stack, eval_expression, 1)?;
+    let mut args: Vec<Spanned<OsString>> = Vec::with_capacity(call_args.len());
+
+    for arg in call_args {
+        let span = arg.span();
+        match arg {
+            // Expand globs passed to run-external
+            Value::Glob { val, no_expand, .. } if !no_expand => args.extend(
+                expand_glob(&val, cwd.as_std_path(), span, engine_state.signals())?
+                    .into_iter()
+                    .map(|s| s.into_spanned(span)),
+            ),
+            other => args
+                .push(OsString::from(coerce_into_string(engine_state, other)?).into_spanned(span)),
         }
     }
     Ok(args)
@@ -243,42 +242,17 @@ pub fn eval_arguments_from_call(
 
 /// Custom `coerce_into_string()`, including globs, since those are often args to `run-external`
 /// as well
-fn coerce_into_string(val: Value) -> Result<String, ShellError> {
+fn coerce_into_string(engine_state: &EngineState, val: Value) -> Result<String, ShellError> {
     match val {
+        Value::List { .. } => Err(ShellError::CannotPassListToExternal {
+            arg: String::from_utf8_lossy(engine_state.get_span_contents(val.span())).into_owned(),
+            span: val.span(),
+        }),
         Value::Glob { val, .. } => Ok(val),
         _ => val.coerce_into_string(),
     }
 }
 
-/// Evaluate an argument, returning more than one value if it was a list to be spread.
-fn eval_argument(
-    engine_state: &EngineState,
-    stack: &mut Stack,
-    expr: &Expression,
-    spread: bool,
-) -> Result<Vec<Value>, ShellError> {
-    let eval = get_eval_expression(engine_state);
-    match eval(engine_state, stack, expr)? {
-        Value::List { vals, .. } => {
-            if spread {
-                Ok(vals)
-            } else {
-                Err(ShellError::CannotPassListToExternal {
-                    arg: String::from_utf8_lossy(engine_state.get_span_contents(expr.span)).into(),
-                    span: expr.span,
-                })
-            }
-        }
-        value => {
-            if spread {
-                Err(ShellError::CannotSpreadAsList { span: expr.span })
-            } else {
-                Ok(vec![value])
-            }
-        }
-    }
-}
-
 /// Performs glob expansion on `arg`. If the expansion found no matches or the pattern
 /// is not a valid glob, then this returns the original string as the expansion result.
 ///

crates/nu-command/src/system/uname.rs

@@ -1,10 +1,5 @@
-use nu_protocol::record;
-use nu_protocol::Value;
-use nu_protocol::{
-    ast::Call,
-    engine::{Command, EngineState, Stack},
-    Category, Example, PipelineData, ShellError, Signature, Type,
-};
+use nu_engine::command_prelude::*;
+use nu_protocol::{record, Value};
 
 #[derive(Clone)]
 pub struct UName;

crates/nu-command/src/viewers/table.rs

@@ -344,7 +344,7 @@ fn get_theme_flag(
 struct CmdInput<'a> {
     engine_state: &'a EngineState,
     stack: &'a mut Stack,
-    call: &'a Call,
+    call: &'a Call<'a>,
     data: PipelineData,
 }
 
@@ -352,7 +352,7 @@ impl<'a> CmdInput<'a> {
     fn new(
         engine_state: &'a EngineState,
         stack: &'a mut Stack,
-        call: &'a Call,
+        call: &'a Call<'a>,
         data: PipelineData,
     ) -> Self {
         Self {

crates/nu-engine/Cargo.toml

@@ -15,6 +15,7 @@ nu-protocol = { path = "../nu-protocol", features = ["plugin"], version = "0.95.
 nu-path = { path = "../nu-path", version = "0.95.1" }
 nu-glob = { path = "../nu-glob", version = "0.95.1" }
 nu-utils = { path = "../nu-utils", version = "0.95.1" }
+log = { workspace = true }
 
 [features]
-plugin = []
+plugin = []

crates/nu-engine/src/call_ext.rs

@@ -1,10 +1,10 @@
 use crate::eval_expression;
 use nu_protocol::{
-    ast::Call,
+    ast,
     debugger::WithoutDebug,
-    engine::{EngineState, Stack, StateWorkingSet},
+    engine::{self, EngineState, Stack, StateWorkingSet},
     eval_const::eval_constant,
-    FromValue, ShellError, Value,
+    ir, FromValue, ShellError, Span, Value,
 };
 
 pub trait CallExt {
@@ -23,6 +23,9 @@ pub trait CallExt {
         name: &str,
     ) -> Result<Option<T>, ShellError>;
 
+    /// Efficiently get the span of a flag argument
+    fn get_flag_span(&self, stack: &Stack, name: &str) -> Option<Span>;
+
     fn rest<T: FromValue>(
         &self,
         engine_state: &EngineState,
@@ -56,9 +59,12 @@ pub trait CallExt {
         stack: &mut Stack,
         name: &str,
     ) -> Result<T, ShellError>;
+
+    /// True if the command has any positional or rest arguments, excluding before the given index.
+    fn has_positional_args(&self, stack: &Stack, starting_pos: usize) -> bool;
 }
 
-impl CallExt for Call {
+impl CallExt for ast::Call {
     fn has_flag(
         &self,
         engine_state: &EngineState,
@@ -104,6 +110,10 @@ impl CallExt for Call {
         }
     }
 
+    fn get_flag_span(&self, _stack: &Stack, name: &str) -> Option<Span> {
+        self.get_named_arg(name).map(|arg| arg.span)
+    }
+
     fn rest<T: FromValue>(
         &self,
         engine_state: &EngineState,
@@ -189,4 +199,205 @@ impl CallExt for Call {
             })
         }
     }
+
+    fn has_positional_args(&self, _stack: &Stack, starting_pos: usize) -> bool {
+        self.rest_iter(starting_pos).next().is_some()
+    }
+}
+
+impl CallExt for ir::Call {
+    fn has_flag(
+        &self,
+        _engine_state: &EngineState,
+        stack: &mut Stack,
+        flag_name: &str,
+    ) -> Result<bool, ShellError> {
+        Ok(self
+            .named_iter(stack)
+            .find(|(name, _)| name.item == flag_name)
+            .is_some_and(|(_, value)| {
+                // Handle --flag=false
+                !matches!(value, Some(Value::Bool { val: false, .. }))
+            }))
+    }
+
+    fn get_flag<T: FromValue>(
+        &self,
+        _engine_state: &EngineState,
+        stack: &mut Stack,
+        name: &str,
+    ) -> Result<Option<T>, ShellError> {
+        if let Some(val) = self.get_named_arg(stack, name) {
+            T::from_value(val.clone()).map(Some)
+        } else {
+            Ok(None)
+        }
+    }
+
+    fn get_flag_span(&self, stack: &Stack, name: &str) -> Option<Span> {
+        self.named_iter(stack)
+            .find_map(|(i_name, _)| (i_name.item == name).then_some(i_name.span))
+    }
+
+    fn rest<T: FromValue>(
+        &self,
+        _engine_state: &EngineState,
+        stack: &mut Stack,
+        starting_pos: usize,
+    ) -> Result<Vec<T>, ShellError> {
+        self.rest_iter_flattened(stack, starting_pos)?
+            .into_iter()
+            .map(T::from_value)
+            .collect()
+    }
+
+    fn opt<T: FromValue>(
+        &self,
+        _engine_state: &EngineState,
+        stack: &mut Stack,
+        pos: usize,
+    ) -> Result<Option<T>, ShellError> {
+        self.positional_iter(stack)
+            .nth(pos)
+            .cloned()
+            .map(T::from_value)
+            .transpose()
+    }
+
+    fn opt_const<T: FromValue>(
+        &self,
+        _working_set: &StateWorkingSet,
+        _pos: usize,
+    ) -> Result<Option<T>, ShellError> {
+        Err(ShellError::IrEvalError {
+            msg: "const evaluation is not yet implemented on ir::Call".into(),
+            span: Some(self.head),
+        })
+    }
+
+    fn req<T: FromValue>(
+        &self,
+        engine_state: &EngineState,
+        stack: &mut Stack,
+        pos: usize,
+    ) -> Result<T, ShellError> {
+        if let Some(val) = self.opt(engine_state, stack, pos)? {
+            Ok(val)
+        } else if self.positional_len(stack) == 0 {
+            Err(ShellError::AccessEmptyContent { span: self.head })
+        } else {
+            Err(ShellError::AccessBeyondEnd {
+                max_idx: self.positional_len(stack) - 1,
+                span: self.head,
+            })
+        }
+    }
+
+    fn req_parser_info<T: FromValue>(
+        &self,
+        engine_state: &EngineState,
+        stack: &mut Stack,
+        name: &str,
+    ) -> Result<T, ShellError> {
+        // FIXME: this depends on the AST evaluator. We can fix this by making the parser info an
+        // enum rather than using expressions. It's not clear that evaluation of this is ever really
+        // needed.
+        if let Some(expr) = self.get_parser_info(stack, name) {
+            let expr = expr.clone();
+            let stack = &mut stack.use_call_arg_out_dest();
+            let result = eval_expression::<WithoutDebug>(engine_state, stack, &expr)?;
+            FromValue::from_value(result)
+        } else {
+            Err(ShellError::CantFindColumn {
+                col_name: name.into(),
+                span: None,
+                src_span: self.head,
+            })
+        }
+    }
+
+    fn has_positional_args(&self, stack: &Stack, starting_pos: usize) -> bool {
+        self.rest_iter(stack, starting_pos).next().is_some()
+    }
+}
+
+macro_rules! proxy {
+    ($self:ident . $method:ident ($($param:expr),*)) => (match &$self.inner {
+        engine::CallImpl::AstRef(call) => call.$method($($param),*),
+        engine::CallImpl::AstBox(call) => call.$method($($param),*),
+        engine::CallImpl::IrRef(call) => call.$method($($param),*),
+        engine::CallImpl::IrBox(call) => call.$method($($param),*),
+    })
+}
+
+impl CallExt for engine::Call<'_> {
+    fn has_flag(
+        &self,
+        engine_state: &EngineState,
+        stack: &mut Stack,
+        flag_name: &str,
+    ) -> Result<bool, ShellError> {
+        proxy!(self.has_flag(engine_state, stack, flag_name))
+    }
+
+    fn get_flag<T: FromValue>(
+        &self,
+        engine_state: &EngineState,
+        stack: &mut Stack,
+        name: &str,
+    ) -> Result<Option<T>, ShellError> {
+        proxy!(self.get_flag(engine_state, stack, name))
+    }
+
+    fn get_flag_span(&self, stack: &Stack, name: &str) -> Option<Span> {
+        proxy!(self.get_flag_span(stack, name))
+    }
+
+    fn rest<T: FromValue>(
+        &self,
+        engine_state: &EngineState,
+        stack: &mut Stack,
+        starting_pos: usize,
+    ) -> Result<Vec<T>, ShellError> {
+        proxy!(self.rest(engine_state, stack, starting_pos))
+    }
+
+    fn opt<T: FromValue>(
+        &self,
+        engine_state: &EngineState,
+        stack: &mut Stack,
+        pos: usize,
+    ) -> Result<Option<T>, ShellError> {
+        proxy!(self.opt(engine_state, stack, pos))
+    }
+
+    fn opt_const<T: FromValue>(
+        &self,
+        working_set: &StateWorkingSet,
+        pos: usize,
+    ) -> Result<Option<T>, ShellError> {
+        proxy!(self.opt_const(working_set, pos))
+    }
+
+    fn req<T: FromValue>(
+        &self,
+        engine_state: &EngineState,
+        stack: &mut Stack,
+        pos: usize,
+    ) -> Result<T, ShellError> {
+        proxy!(self.req(engine_state, stack, pos))
+    }
+
+    fn req_parser_info<T: FromValue>(
+        &self,
+        engine_state: &EngineState,
+        stack: &mut Stack,
+        name: &str,
+    ) -> Result<T, ShellError> {
+        proxy!(self.req_parser_info(engine_state, stack, name))
+    }
+
+    fn has_positional_args(&self, stack: &Stack, starting_pos: usize) -> bool {
+        proxy!(self.has_positional_args(stack, starting_pos))
+    }
 }

crates/nu-engine/src/command_prelude.rs

@@ -1,7 +1,7 @@
 pub use crate::CallExt;
 pub use nu_protocol::{
-    ast::{Call, CellPath},
-    engine::{Command, EngineState, Stack, StateWorkingSet},
+    ast::CellPath,
+    engine::{Call, Command, EngineState, Stack, StateWorkingSet},
     record, ByteStream, ByteStreamType, Category, ErrSpan, Example, IntoInterruptiblePipelineData,
     IntoPipelineData, IntoSpanned, PipelineData, Record, ShellError, Signature, Span, Spanned,
     SyntaxShape, Type, Value,

crates/nu-engine/src/compile/builder.rs

@@ -0,0 +1,575 @@
+use nu_protocol::{
+    ir::{DataSlice, Instruction, IrAstRef, IrBlock, Literal},
+    CompileError, IntoSpanned, RegId, Span, Spanned,
+};
+
+/// A label identifier. Only exists while building code. Replaced with the actual target.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(crate) struct LabelId(pub usize);
+
+/// Builds [`IrBlock`]s progressively by consuming instructions and handles register allocation.
+#[derive(Debug)]
+pub(crate) struct BlockBuilder {
+    pub(crate) block_span: Option<Span>,
+    pub(crate) instructions: Vec<Instruction>,
+    pub(crate) spans: Vec<Span>,
+    /// The actual instruction index that a label refers to. While building IR, branch targets are
+    /// specified as indices into this array rather than the true instruction index. This makes it
+    /// easier to make modifications to code, as just this array needs to be changed, and it's also
+    /// less error prone as during `finish()` we check to make sure all of the used labels have had
+    /// an index actually set.
+    pub(crate) labels: Vec<Option<usize>>,
+    pub(crate) data: Vec<u8>,
+    pub(crate) ast: Vec<Option<IrAstRef>>,
+    pub(crate) comments: Vec<String>,
+    pub(crate) register_allocation_state: Vec<bool>,
+    pub(crate) file_count: u32,
+    pub(crate) loop_stack: Vec<Loop>,
+}
+
+impl BlockBuilder {
+    /// Starts a new block, with the first register (`%0`) allocated as input.
+    pub(crate) fn new(block_span: Option<Span>) -> Self {
+        BlockBuilder {
+            block_span,
+            instructions: vec![],
+            spans: vec![],
+            labels: vec![],
+            data: vec![],
+            ast: vec![],
+            comments: vec![],
+            register_allocation_state: vec![true],
+            file_count: 0,
+            loop_stack: vec![],
+        }
+    }
+
+    /// Get the next unused register for code generation.
+    pub(crate) fn next_register(&mut self) -> Result<RegId, CompileError> {
+        if let Some(index) = self
+            .register_allocation_state
+            .iter_mut()
+            .position(|is_allocated| {
+                if !*is_allocated {
+                    *is_allocated = true;
+                    true
+                } else {
+                    false
+                }
+            })
+        {
+            Ok(RegId(index as u32))
+        } else if self.register_allocation_state.len() < (u32::MAX as usize - 2) {
+            let reg_id = RegId(self.register_allocation_state.len() as u32);
+            self.register_allocation_state.push(true);
+            Ok(reg_id)
+        } else {
+            Err(CompileError::RegisterOverflow {
+                block_span: self.block_span,
+            })
+        }
+    }
+
+    /// Check if a register is initialized with a value.
+    pub(crate) fn is_allocated(&self, reg_id: RegId) -> bool {
+        self.register_allocation_state
+            .get(reg_id.0 as usize)
+            .is_some_and(|state| *state)
+    }
+
+    /// Mark a register as initialized.
+    pub(crate) fn mark_register(&mut self, reg_id: RegId) -> Result<(), CompileError> {
+        if let Some(is_allocated) = self.register_allocation_state.get_mut(reg_id.0 as usize) {
+            *is_allocated = true;
+            Ok(())
+        } else {
+            Err(CompileError::RegisterOverflow {
+                block_span: self.block_span,
+            })
+        }
+    }
+
+    /// Mark a register as empty, so that it can be used again by something else.
+    #[track_caller]
+    pub(crate) fn free_register(&mut self, reg_id: RegId) -> Result<(), CompileError> {
+        let index = reg_id.0 as usize;
+
+        if self
+            .register_allocation_state
+            .get(index)
+            .is_some_and(|is_allocated| *is_allocated)
+        {
+            self.register_allocation_state[index] = false;
+            Ok(())
+        } else {
+            log::warn!("register {reg_id} uninitialized, builder = {self:#?}");
+            Err(CompileError::RegisterUninitialized {
+                reg_id,
+                caller: std::panic::Location::caller().to_string(),
+            })
+        }
+    }
+
+    /// Define a label, which can be used by branch instructions. The target can optionally be
+    /// specified now.
+    pub(crate) fn label(&mut self, target_index: Option<usize>) -> LabelId {
+        let label_id = self.labels.len();
+        self.labels.push(target_index);
+        LabelId(label_id)
+    }
+
+    /// Change the target of a label.
+    pub(crate) fn set_label(
+        &mut self,
+        label_id: LabelId,
+        target_index: usize,
+    ) -> Result<(), CompileError> {
+        *self
+            .labels
+            .get_mut(label_id.0)
+            .ok_or(CompileError::UndefinedLabel {
+                label_id: label_id.0,
+                span: None,
+            })? = Some(target_index);
+        Ok(())
+    }
+
+    /// Insert an instruction into the block, automatically marking any registers populated by
+    /// the instruction, and freeing any registers consumed by the instruction.
+    #[track_caller]
+    pub(crate) fn push(&mut self, instruction: Spanned<Instruction>) -> Result<(), CompileError> {
+        // Free read registers, and mark write registers.
+        //
+        // If a register is both read and written, it should be on both sides, so that we can verify
+        // that the register was in the right state beforehand.
+        let mut allocate = |read: &[RegId], write: &[RegId]| -> Result<(), CompileError> {
+            for reg in read {
+                self.free_register(*reg)?;
+            }
+            for reg in write {
+                self.mark_register(*reg)?;
+            }
+            Ok(())
+        };
+
+        let allocate_result = match &instruction.item {
+            Instruction::Unreachable => Ok(()),
+            Instruction::LoadLiteral { dst, lit } => {
+                allocate(&[], &[*dst]).and(
+                    // Free any registers on the literal
+                    match lit {
+                        Literal::Range {
+                            start,
+                            step,
+                            end,
+                            inclusion: _,
+                        } => allocate(&[*start, *step, *end], &[]),
+                        Literal::Bool(_)
+                        | Literal::Int(_)
+                        | Literal::Float(_)
+                        | Literal::Filesize(_)
+                        | Literal::Duration(_)
+                        | Literal::Binary(_)
+                        | Literal::Block(_)
+                        | Literal::Closure(_)
+                        | Literal::RowCondition(_)
+                        | Literal::List { capacity: _ }
+                        | Literal::Record { capacity: _ }
+                        | Literal::Filepath {
+                            val: _,
+                            no_expand: _,
+                        }
+                        | Literal::Directory {
+                            val: _,
+                            no_expand: _,
+                        }
+                        | Literal::GlobPattern {
+                            val: _,
+                            no_expand: _,
+                        }
+                        | Literal::String(_)
+                        | Literal::RawString(_)
+                        | Literal::CellPath(_)
+                        | Literal::Date(_)
+                        | Literal::Nothing => Ok(()),
+                    },
+                )
+            }
+            Instruction::LoadValue { dst, val: _ } => allocate(&[], &[*dst]),
+            Instruction::Move { dst, src } => allocate(&[*src], &[*dst]),
+            Instruction::Clone { dst, src } => allocate(&[*src], &[*dst, *src]),
+            Instruction::Collect { src_dst } => allocate(&[*src_dst], &[*src_dst]),
+            Instruction::Span { src_dst } => allocate(&[*src_dst], &[*src_dst]),
+            Instruction::Drop { src } => allocate(&[*src], &[]),
+            Instruction::Drain { src } => allocate(&[*src], &[]),
+            Instruction::LoadVariable { dst, var_id: _ } => allocate(&[], &[*dst]),
+            Instruction::StoreVariable { var_id: _, src } => allocate(&[*src], &[]),
+            Instruction::LoadEnv { dst, key: _ } => allocate(&[], &[*dst]),
+            Instruction::LoadEnvOpt { dst, key: _ } => allocate(&[], &[*dst]),
+            Instruction::StoreEnv { key: _, src } => allocate(&[*src], &[]),
+            Instruction::PushPositional { src } => allocate(&[*src], &[]),
+            Instruction::AppendRest { src } => allocate(&[*src], &[]),
+            Instruction::PushFlag { name: _ } => Ok(()),
+            Instruction::PushShortFlag { short: _ } => Ok(()),
+            Instruction::PushNamed { name: _, src } => allocate(&[*src], &[]),
+            Instruction::PushShortNamed { short: _, src } => allocate(&[*src], &[]),
+            Instruction::PushParserInfo { name: _, info: _ } => Ok(()),
+            Instruction::RedirectOut { mode: _ } => Ok(()),
+            Instruction::RedirectErr { mode: _ } => Ok(()),
+            Instruction::CheckErrRedirected { src } => allocate(&[*src], &[*src]),
+            Instruction::OpenFile {
+                file_num: _,
+                path,
+                append: _,
+            } => allocate(&[*path], &[]),
+            Instruction::WriteFile { file_num: _, src } => allocate(&[*src], &[]),
+            Instruction::CloseFile { file_num: _ } => Ok(()),
+            Instruction::Call {
+                decl_id: _,
+                src_dst,
+            } => allocate(&[*src_dst], &[*src_dst]),
+            Instruction::StringAppend { src_dst, val } => allocate(&[*src_dst, *val], &[*src_dst]),
+            Instruction::GlobFrom {
+                src_dst,
+                no_expand: _,
+            } => allocate(&[*src_dst], &[*src_dst]),
+            Instruction::ListPush { src_dst, item } => allocate(&[*src_dst, *item], &[*src_dst]),
+            Instruction::ListSpread { src_dst, items } => {
+                allocate(&[*src_dst, *items], &[*src_dst])
+            }
+            Instruction::RecordInsert { src_dst, key, val } => {
+                allocate(&[*src_dst, *key, *val], &[*src_dst])
+            }
+            Instruction::RecordSpread { src_dst, items } => {
+                allocate(&[*src_dst, *items], &[*src_dst])
+            }
+            Instruction::Not { src_dst } => allocate(&[*src_dst], &[*src_dst]),
+            Instruction::BinaryOp {
+                lhs_dst,
+                op: _,
+                rhs,
+            } => allocate(&[*lhs_dst, *rhs], &[*lhs_dst]),
+            Instruction::FollowCellPath { src_dst, path } => {
+                allocate(&[*src_dst, *path], &[*src_dst])
+            }
+            Instruction::CloneCellPath { dst, src, path } => {
+                allocate(&[*src, *path], &[*src, *dst])
+            }
+            Instruction::UpsertCellPath {
+                src_dst,
+                path,
+                new_value,
+            } => allocate(&[*src_dst, *path, *new_value], &[*src_dst]),
+            Instruction::Jump { index: _ } => Ok(()),
+            Instruction::BranchIf { cond, index: _ } => allocate(&[*cond], &[]),
+            Instruction::BranchIfEmpty { src, index: _ } => allocate(&[*src], &[*src]),
+            Instruction::Match {
+                pattern: _,
+                src,
+                index: _,
+            } => allocate(&[*src], &[*src]),
+            Instruction::CheckMatchGuard { src } => allocate(&[*src], &[*src]),
+            Instruction::Iterate {
+                dst,
+                stream,
+                end_index: _,
+            } => allocate(&[*stream], &[*dst, *stream]),
+            Instruction::OnError { index: _ } => Ok(()),
+            Instruction::OnErrorInto { index: _, dst } => allocate(&[], &[*dst]),
+            Instruction::PopErrorHandler => Ok(()),
+            Instruction::CheckExternalFailed { dst, src } => allocate(&[*src], &[*dst, *src]),
+            Instruction::ReturnEarly { src } => allocate(&[*src], &[]),
+            Instruction::Return { src } => allocate(&[*src], &[]),
+        };
+
+        // Add more context to the error
+        match allocate_result {
+            Ok(()) => (),
+            Err(CompileError::RegisterUninitialized { reg_id, caller }) => {
+                return Err(CompileError::RegisterUninitializedWhilePushingInstruction {
+                    reg_id,
+                    caller,
+                    instruction: format!("{:?}", instruction.item),
+                    span: instruction.span,
+                });
+            }
+            Err(err) => return Err(err),
+        }
+
+        self.instructions.push(instruction.item);
+        self.spans.push(instruction.span);
+        self.ast.push(None);
+        self.comments.push(String::new());
+        Ok(())
+    }
+
+    /// Set the AST of the last instruction. Separate method because it's rarely used.
+    pub(crate) fn set_last_ast(&mut self, ast_ref: Option<IrAstRef>) {
+        *self.ast.last_mut().expect("no last instruction") = ast_ref;
+    }
+
+    /// Add a comment to the last instruction.
+    pub(crate) fn add_comment(&mut self, comment: impl std::fmt::Display) {
+        add_comment(
+            self.comments.last_mut().expect("no last instruction"),
+            comment,
+        )
+    }
+
+    /// Load a register with a literal.
+    pub(crate) fn load_literal(
+        &mut self,
+        reg_id: RegId,
+        literal: Spanned<Literal>,
+    ) -> Result<(), CompileError> {
+        self.push(
+            Instruction::LoadLiteral {
+                dst: reg_id,
+                lit: literal.item,
+            }
+            .into_spanned(literal.span),
+        )?;
+        Ok(())
+    }
+
+    /// Allocate a new register and load a literal into it.
+    pub(crate) fn literal(&mut self, literal: Spanned<Literal>) -> Result<RegId, CompileError> {
+        let reg_id = self.next_register()?;
+        self.load_literal(reg_id, literal)?;
+        Ok(reg_id)
+    }
+
+    /// Deallocate a register and set it to `Empty`, if it is allocated
+    pub(crate) fn drop_reg(&mut self, reg_id: RegId) -> Result<(), CompileError> {
+        if self.is_allocated(reg_id) {
+            self.push(Instruction::Drop { src: reg_id }.into_spanned(Span::unknown()))?;
+        }
+        Ok(())
+    }
+
+    /// Set a register to `Empty`, but mark it as in-use, e.g. for input
+    pub(crate) fn load_empty(&mut self, reg_id: RegId) -> Result<(), CompileError> {
+        self.drop_reg(reg_id)?;
+        self.mark_register(reg_id)
+    }
+
+    /// Drain the stream in a register (fully consuming it)
+    pub(crate) fn drain(&mut self, src: RegId, span: Span) -> Result<(), CompileError> {
+        self.push(Instruction::Drain { src }.into_spanned(span))
+    }
+
+    /// Add data to the `data` array and return a [`DataSlice`] referencing it.
+    pub(crate) fn data(&mut self, data: impl AsRef<[u8]>) -> Result<DataSlice, CompileError> {
+        let data = data.as_ref();
+        let start = self.data.len();
+        if data.is_empty() {
+            Ok(DataSlice::empty())
+        } else if start + data.len() < u32::MAX as usize {
+            let slice = DataSlice {
+                start: start as u32,
+                len: data.len() as u32,
+            };
+            self.data.extend_from_slice(data);
+            Ok(slice)
+        } else {
+            Err(CompileError::DataOverflow {
+                block_span: self.block_span,
+            })
+        }
+    }
+
+    /// Clone a register with a `clone` instruction.
+    pub(crate) fn clone_reg(&mut self, src: RegId, span: Span) -> Result<RegId, CompileError> {
+        let dst = self.next_register()?;
+        self.push(Instruction::Clone { dst, src }.into_spanned(span))?;
+        Ok(dst)
+    }
+
+    /// Add a `branch-if` instruction
+    pub(crate) fn branch_if(
+        &mut self,
+        cond: RegId,
+        label_id: LabelId,
+        span: Span,
+    ) -> Result<(), CompileError> {
+        self.push(
+            Instruction::BranchIf {
+                cond,
+                index: label_id.0,
+            }
+            .into_spanned(span),
+        )
+    }
+
+    /// Add a `branch-if-empty` instruction
+    pub(crate) fn branch_if_empty(
+        &mut self,
+        src: RegId,
+        label_id: LabelId,
+        span: Span,
+    ) -> Result<(), CompileError> {
+        self.push(
+            Instruction::BranchIfEmpty {
+                src,
+                index: label_id.0,
+            }
+            .into_spanned(span),
+        )
+    }
+
+    /// Add a `jump` instruction
+    pub(crate) fn jump(&mut self, label_id: LabelId, span: Span) -> Result<(), CompileError> {
+        self.push(Instruction::Jump { index: label_id.0 }.into_spanned(span))
+    }
+
+    /// The index that the next instruction [`.push()`]ed will have.
+    pub(crate) fn here(&self) -> usize {
+        self.instructions.len()
+    }
+
+    /// Allocate a new file number, for redirection.
+    pub(crate) fn next_file_num(&mut self) -> Result<u32, CompileError> {
+        let next = self.file_count;
+        self.file_count = self
+            .file_count
+            .checked_add(1)
+            .ok_or(CompileError::FileOverflow {
+                block_span: self.block_span,
+            })?;
+        Ok(next)
+    }
+
+    /// Push a new loop state onto the builder. Creates new labels that must be set.
+    pub(crate) fn begin_loop(&mut self) -> Loop {
+        let loop_ = Loop {
+            break_label: self.label(None),
+            continue_label: self.label(None),
+        };
+        self.loop_stack.push(loop_);
+        loop_
+    }
+
+    /// True if we are currently in a loop.
+    pub(crate) fn is_in_loop(&self) -> bool {
+        !self.loop_stack.is_empty()
+    }
+
+    /// Add a loop breaking jump instruction.
+    pub(crate) fn push_break(&mut self, span: Span) -> Result<(), CompileError> {
+        let loop_ = self
+            .loop_stack
+            .last()
+            .ok_or_else(|| CompileError::NotInALoop {
+                msg: "`break` called from outside of a loop".into(),
+                span: Some(span),
+            })?;
+        self.jump(loop_.break_label, span)
+    }
+
+    /// Add a loop continuing jump instruction.
+    pub(crate) fn push_continue(&mut self, span: Span) -> Result<(), CompileError> {
+        let loop_ = self
+            .loop_stack
+            .last()
+            .ok_or_else(|| CompileError::NotInALoop {
+                msg: "`continue` called from outside of a loop".into(),
+                span: Some(span),
+            })?;
+        self.jump(loop_.continue_label, span)
+    }
+
+    /// Pop the loop state. Checks that the loop being ended is the same one that was expected.
+    pub(crate) fn end_loop(&mut self, loop_: Loop) -> Result<(), CompileError> {
+        let ended_loop = self
+            .loop_stack
+            .pop()
+            .ok_or_else(|| CompileError::NotInALoop {
+                msg: "end_loop() called outside of a loop".into(),
+                span: None,
+            })?;
+
+        if ended_loop == loop_ {
+            Ok(())
+        } else {
+            Err(CompileError::IncoherentLoopState {
+                block_span: self.block_span,
+            })
+        }
+    }
+
+    /// Mark an unreachable code path. Produces an error at runtime if executed.
+    pub(crate) fn unreachable(&mut self, span: Span) -> Result<(), CompileError> {
+        self.push(Instruction::Unreachable.into_spanned(span))
+    }
+
+    /// Consume the builder and produce the final [`IrBlock`].
+    pub(crate) fn finish(mut self) -> Result<IrBlock, CompileError> {
+        // Add comments to label targets
+        for (index, label_target) in self.labels.iter().enumerate() {
+            if let Some(label_target) = label_target {
+                add_comment(
+                    &mut self.comments[*label_target],
+                    format_args!("label({index})"),
+                );
+            }
+        }
+
+        // Populate the actual target indices of labels into the instructions
+        for ((index, instruction), span) in
+            self.instructions.iter_mut().enumerate().zip(&self.spans)
+        {
+            if let Some(label_id) = instruction.branch_target() {
+                let target_index = self.labels.get(label_id).cloned().flatten().ok_or(
+                    CompileError::UndefinedLabel {
+                        label_id,
+                        span: Some(*span),
+                    },
+                )?;
+                // Add a comment to the target index that we come from here
+                add_comment(
+                    &mut self.comments[target_index],
+                    format_args!("from({index}:)"),
+                );
+                instruction.set_branch_target(target_index).map_err(|_| {
+                    CompileError::SetBranchTargetOfNonBranchInstruction {
+                        instruction: format!("{:?}", instruction),
+                        span: *span,
+                    }
+                })?;
+            }
+        }
+
+        Ok(IrBlock {
+            instructions: self.instructions,
+            spans: self.spans,
+            data: self.data.into(),
+            ast: self.ast,
+            comments: self.comments.into_iter().map(|s| s.into()).collect(),
+            register_count: self
+                .register_allocation_state
+                .len()
+                .try_into()
+                .expect("register count overflowed in finish() despite previous checks"),
+            file_count: self.file_count,
+        })
+    }
+}
+
+/// Keeps track of the `break` and `continue` target labels for a loop.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(crate) struct Loop {
+    pub(crate) break_label: LabelId,
+    pub(crate) continue_label: LabelId,
+}
+
+/// Add a new comment to an existing one
+fn add_comment(comment: &mut String, new_comment: impl std::fmt::Display) {
+    use std::fmt::Write;
+    write!(
+        comment,
+        "{}{}",
+        if comment.is_empty() { "" } else { ", " },
+        new_comment
+    )
+    .expect("formatting failed");
+}

crates/nu-engine/src/compile/call.rs

@@ -0,0 +1,270 @@
+use std::sync::Arc;
+
+use nu_protocol::{
+    ast::{Argument, Call, Expression, ExternalArgument},
+    engine::StateWorkingSet,
+    ir::{Instruction, IrAstRef, Literal},
+    IntoSpanned, RegId, Span, Spanned,
+};
+
+use super::{compile_expression, keyword::*, BlockBuilder, CompileError, RedirectModes};
+
+pub(crate) fn compile_call(
+    working_set: &StateWorkingSet,
+    builder: &mut BlockBuilder,
+    call: &Call,
+    redirect_modes: RedirectModes,
+    io_reg: RegId,
+) -> Result<(), CompileError> {
+    let decl = working_set.get_decl(call.decl_id);
+
+    // Check if this call has --help - if so, just redirect to `help`
+    if call.named_iter().any(|(name, _, _)| name.item == "help") {
+        return compile_help(
+            working_set,
+            builder,
+            decl.name().into_spanned(call.head),
+            io_reg,
+        );
+    }
+
+    // Try to figure out if this is a keyword call like `if`, and handle those specially
+    if decl.is_keyword() {
+        match decl.name() {
+            "if" => {
+                return compile_if(working_set, builder, call, redirect_modes, io_reg);
+            }
+            "match" => {
+                return compile_match(working_set, builder, call, redirect_modes, io_reg);
+            }
+            "const" => {
+                // This differs from the behavior of the const command, which adds the const value
+                // to the stack. Since `load-variable` also checks `engine_state` for the variable
+                // and will get a const value though, is it really necessary to do that?
+                return builder.load_empty(io_reg);
+            }
+            "alias" => {
+                // Alias does nothing
+                return builder.load_empty(io_reg);
+            }
+            "let" | "mut" => {
+                return compile_let(working_set, builder, call, redirect_modes, io_reg);
+            }
+            "try" => {
+                return compile_try(working_set, builder, call, redirect_modes, io_reg);
+            }
+            "loop" => {
+                return compile_loop(working_set, builder, call, redirect_modes, io_reg);
+            }
+            "while" => {
+                return compile_while(working_set, builder, call, redirect_modes, io_reg);
+            }
+            "for" => {
+                return compile_for(working_set, builder, call, redirect_modes, io_reg);
+            }
+            "break" => {
+                return compile_break(working_set, builder, call, redirect_modes, io_reg);
+            }
+            "continue" => {
+                return compile_continue(working_set, builder, call, redirect_modes, io_reg);
+            }
+            "return" => {
+                return compile_return(working_set, builder, call, redirect_modes, io_reg);
+            }
+            _ => (),
+        }
+    }
+
+    // Keep AST if the decl needs it.
+    let requires_ast = decl.requires_ast_for_arguments();
+
+    // It's important that we evaluate the args first before trying to set up the argument
+    // state for the call.
+    //
+    // We could technically compile anything that isn't another call safely without worrying about
+    // the argument state, but we'd have to check all of that first and it just isn't really worth
+    // it.
+    enum CompiledArg<'a> {
+        Positional(RegId, Span, Option<IrAstRef>),
+        Named(
+            &'a str,
+            Option<&'a str>,
+            Option<RegId>,
+            Span,
+            Option<IrAstRef>,
+        ),
+        Spread(RegId, Span, Option<IrAstRef>),
+    }
+
+    let mut compiled_args = vec![];
+
+    for arg in &call.arguments {
+        let arg_reg = arg
+            .expr()
+            .map(|expr| {
+                let arg_reg = builder.next_register()?;
+
+                compile_expression(
+                    working_set,
+                    builder,
+                    expr,
+                    RedirectModes::capture_out(arg.span()),
+                    None,
+                    arg_reg,
+                )?;
+
+                Ok(arg_reg)
+            })
+            .transpose()?;
+
+        let ast_ref = arg
+            .expr()
+            .filter(|_| requires_ast)
+            .map(|expr| IrAstRef(Arc::new(expr.clone())));
+
+        match arg {
+            Argument::Positional(_) | Argument::Unknown(_) => {
+                compiled_args.push(CompiledArg::Positional(
+                    arg_reg.expect("expr() None in non-Named"),
+                    arg.span(),
+                    ast_ref,
+                ))
+            }
+            Argument::Named((name, short, _)) => compiled_args.push(CompiledArg::Named(
+                &name.item,
+                short.as_ref().map(|spanned| spanned.item.as_str()),
+                arg_reg,
+                arg.span(),
+                ast_ref,
+            )),
+            Argument::Spread(_) => compiled_args.push(CompiledArg::Spread(
+                arg_reg.expect("expr() None in non-Named"),
+                arg.span(),
+                ast_ref,
+            )),
+        }
+    }
+
+    // Now that the args are all compiled, set up the call state (argument stack and redirections)
+    for arg in compiled_args {
+        match arg {
+            CompiledArg::Positional(reg, span, ast_ref) => {
+                builder.push(Instruction::PushPositional { src: reg }.into_spanned(span))?;
+                builder.set_last_ast(ast_ref);
+            }
+            CompiledArg::Named(name, short, Some(reg), span, ast_ref) => {
+                if !name.is_empty() {
+                    let name = builder.data(name)?;
+                    builder.push(Instruction::PushNamed { name, src: reg }.into_spanned(span))?;
+                } else {
+                    let short = builder.data(short.unwrap_or(""))?;
+                    builder
+                        .push(Instruction::PushShortNamed { short, src: reg }.into_spanned(span))?;
+                }
+                builder.set_last_ast(ast_ref);
+            }
+            CompiledArg::Named(name, short, None, span, ast_ref) => {
+                if !name.is_empty() {
+                    let name = builder.data(name)?;
+                    builder.push(Instruction::PushFlag { name }.into_spanned(span))?;
+                } else {
+                    let short = builder.data(short.unwrap_or(""))?;
+                    builder.push(Instruction::PushShortFlag { short }.into_spanned(span))?;
+                }
+                builder.set_last_ast(ast_ref);
+            }
+            CompiledArg::Spread(reg, span, ast_ref) => {
+                builder.push(Instruction::AppendRest { src: reg }.into_spanned(span))?;
+                builder.set_last_ast(ast_ref);
+            }
+        }
+    }
+
+    // Add any parser info from the call
+    for (name, info) in &call.parser_info {
+        let name = builder.data(name)?;
+        let info = Box::new(info.clone());
+        builder.push(Instruction::PushParserInfo { name, info }.into_spanned(call.head))?;
+    }
+
+    if let Some(mode) = redirect_modes.out {
+        builder.push(mode.map(|mode| Instruction::RedirectOut { mode }))?;
+    }
+
+    if let Some(mode) = redirect_modes.err {
+        builder.push(mode.map(|mode| Instruction::RedirectErr { mode }))?;
+    }
+
+    // The state is set up, so we can do the call into io_reg
+    builder.push(
+        Instruction::Call {
+            decl_id: call.decl_id,
+            src_dst: io_reg,
+        }
+        .into_spanned(call.head),
+    )?;
+
+    Ok(())
+}
+
+pub(crate) fn compile_help(
+    working_set: &StateWorkingSet<'_>,
+    builder: &mut BlockBuilder,
+    decl_name: Spanned<&str>,
+    io_reg: RegId,
+) -> Result<(), CompileError> {
+    let help_command_id =
+        working_set
+            .find_decl(b"help")
+            .ok_or_else(|| CompileError::MissingRequiredDeclaration {
+                decl_name: "help".into(),
+                span: decl_name.span,
+            })?;
+
+    let name_data = builder.data(decl_name.item)?;
+    let name_literal = builder.literal(decl_name.map(|_| Literal::String(name_data)))?;
+
+    builder.push(Instruction::PushPositional { src: name_literal }.into_spanned(decl_name.span))?;
+
+    builder.push(
+        Instruction::Call {
+            decl_id: help_command_id,
+            src_dst: io_reg,
+        }
+        .into_spanned(decl_name.span),
+    )?;
+
+    Ok(())
+}
+
+pub(crate) fn compile_external_call(
+    working_set: &StateWorkingSet,
+    builder: &mut BlockBuilder,
+    head: &Expression,
+    args: &[ExternalArgument],
+    redirect_modes: RedirectModes,
+    io_reg: RegId,
+) -> Result<(), CompileError> {
+    // Pass everything to run-external
+    let run_external_id = working_set
+        .find_decl(b"run-external")
+        .ok_or(CompileError::RunExternalNotFound { span: head.span })?;
+
+    let mut call = Call::new(head.span);
+    call.decl_id = run_external_id;
+
+    call.arguments.push(Argument::Positional(head.clone()));
+
+    for arg in args {
+        match arg {
+            ExternalArgument::Regular(expr) => {
+                call.arguments.push(Argument::Positional(expr.clone()));
+            }
+            ExternalArgument::Spread(expr) => {
+                call.arguments.push(Argument::Spread(expr.clone()));
+            }
+        }
+    }
+
+    compile_call(working_set, builder, &call, redirect_modes, io_reg)
+}

crates/nu-engine/src/compile/expression.rs

@@ -0,0 +1,535 @@
+use super::{
+    compile_binary_op, compile_block, compile_call, compile_external_call, compile_load_env,
+    BlockBuilder, CompileError, RedirectModes,
+};
+
+use nu_protocol::{
+    ast::{CellPath, Expr, Expression, ListItem, RecordItem, ValueWithUnit},
+    engine::StateWorkingSet,
+    ir::{DataSlice, Instruction, Literal},
+    IntoSpanned, RegId, Span, Value, ENV_VARIABLE_ID,
+};
+
+pub(crate) fn compile_expression(
+    working_set: &StateWorkingSet,
+    builder: &mut BlockBuilder,
+    expr: &Expression,
+    redirect_modes: RedirectModes,
+    in_reg: Option<RegId>,
+    out_reg: RegId,
+) -> Result<(), CompileError> {
+    let drop_input = |builder: &mut BlockBuilder| {
+        if let Some(in_reg) = in_reg {
+            if in_reg != out_reg {
+                builder.drop_reg(in_reg)?;
+            }
+        }
+        Ok(())
+    };
+
+    let lit = |builder: &mut BlockBuilder, literal: Literal| {
+        drop_input(builder)?;
+
+        builder
+            .push(
+                Instruction::LoadLiteral {
+                    dst: out_reg,
+                    lit: literal,
+                }
+                .into_spanned(expr.span),
+            )
+            .map(|_| ())
+    };
+
+    let ignore = |builder: &mut BlockBuilder| {
+        drop_input(builder)?;
+        builder.load_empty(out_reg)
+    };
+
+    let unexpected = |expr_name: &str| CompileError::UnexpectedExpression {
+        expr_name: expr_name.into(),
+        span: expr.span,
+    };
+
+    let move_in_reg_to_out_reg = |builder: &mut BlockBuilder| {
+        // Ensure that out_reg contains the input value, because a call only uses one register
+        if let Some(in_reg) = in_reg {
+            if in_reg != out_reg {
+                // Have to move in_reg to out_reg so it can be used
+                builder.push(
+                    Instruction::Move {
+                        dst: out_reg,
+                        src: in_reg,
+                    }
+                    .into_spanned(expr.span),
+                )?;
+            }
+        } else {
+            // Will have to initialize out_reg with Empty first
+            builder.load_empty(out_reg)?;
+        }
+        Ok(())
+    };
+
+    match &expr.expr {
+        Expr::Bool(b) => lit(builder, Literal::Bool(*b)),
+        Expr::Int(i) => lit(builder, Literal::Int(*i)),
+        Expr::Float(f) => lit(builder, Literal::Float(*f)),
+        Expr::Binary(bin) => {
+            let data_slice = builder.data(bin)?;
+            lit(builder, Literal::Binary(data_slice))
+        }
+        Expr::Range(range) => {
+            // Compile the subexpressions of the range
+            let compile_part = |builder: &mut BlockBuilder,
+                                part_expr: Option<&Expression>|
+             -> Result<RegId, CompileError> {
+                let reg = builder.next_register()?;
+                if let Some(part_expr) = part_expr {
+                    compile_expression(
+                        working_set,
+                        builder,
+                        part_expr,
+                        RedirectModes::capture_out(part_expr.span),
+                        None,
+                        reg,
+                    )?;
+                } else {
+                    builder.load_literal(reg, Literal::Nothing.into_spanned(expr.span))?;
+                }
+                Ok(reg)
+            };
+
+            drop_input(builder)?;
+
+            let start = compile_part(builder, range.from.as_ref())?;
+            let step = compile_part(builder, range.next.as_ref())?;
+            let end = compile_part(builder, range.to.as_ref())?;
+
+            // Assemble the range
+            builder.load_literal(
+                out_reg,
+                Literal::Range {
+                    start,
+                    step,
+                    end,
+                    inclusion: range.operator.inclusion,
+                }
+                .into_spanned(expr.span),
+            )
+        }
+        Expr::Var(var_id) => {
+            drop_input(builder)?;
+            builder.push(
+                Instruction::LoadVariable {
+                    dst: out_reg,
+                    var_id: *var_id,
+                }
+                .into_spanned(expr.span),
+            )?;
+            Ok(())
+        }
+        Expr::VarDecl(_) => Err(unexpected("VarDecl")),
+        Expr::Call(call) => {
+            move_in_reg_to_out_reg(builder)?;
+
+            compile_call(working_set, builder, call, redirect_modes, out_reg)
+        }
+        Expr::ExternalCall(head, args) => {
+            move_in_reg_to_out_reg(builder)?;
+
+            compile_external_call(working_set, builder, head, args, redirect_modes, out_reg)
+        }
+        Expr::Operator(_) => Err(unexpected("Operator")),
+        Expr::RowCondition(block_id) => lit(builder, Literal::RowCondition(*block_id)),
+        Expr::UnaryNot(subexpr) => {
+            drop_input(builder)?;
+            compile_expression(
+                working_set,
+                builder,
+                subexpr,
+                RedirectModes::capture_out(subexpr.span),
+                None,
+                out_reg,
+            )?;
+            builder.push(Instruction::Not { src_dst: out_reg }.into_spanned(expr.span))?;
+            Ok(())
+        }
+        Expr::BinaryOp(lhs, op, rhs) => {
+            if let Expr::Operator(ref operator) = op.expr {
+                drop_input(builder)?;
+                compile_binary_op(
+                    working_set,
+                    builder,
+                    lhs,
+                    operator.clone().into_spanned(op.span),
+                    rhs,
+                    expr.span,
+                    out_reg,
+                )
+            } else {
+                Err(CompileError::UnsupportedOperatorExpression { span: op.span })
+            }
+        }
+        Expr::Subexpression(block_id) => {
+            let block = working_set.get_block(*block_id);
+            compile_block(working_set, builder, block, redirect_modes, in_reg, out_reg)
+        }
+        Expr::Block(block_id) => lit(builder, Literal::Block(*block_id)),
+        Expr::Closure(block_id) => lit(builder, Literal::Closure(*block_id)),
+        Expr::MatchBlock(_) => Err(unexpected("MatchBlock")), // only for `match` keyword
+        Expr::List(items) => {
+            // Guess capacity based on items (does not consider spread as more than 1)
+            lit(
+                builder,
+                Literal::List {
+                    capacity: items.len(),
+                },
+            )?;
+            for item in items {
+                // Compile the expression of the item / spread
+                let reg = builder.next_register()?;
+                let expr = match item {
+                    ListItem::Item(expr) | ListItem::Spread(_, expr) => expr,
+                };
+                compile_expression(
+                    working_set,
+                    builder,
+                    expr,
+                    RedirectModes::capture_out(expr.span),
+                    None,
+                    reg,
+                )?;
+
+                match item {
+                    ListItem::Item(_) => {
+                        // Add each item using list-push
+                        builder.push(
+                            Instruction::ListPush {
+                                src_dst: out_reg,
+                                item: reg,
+                            }
+                            .into_spanned(expr.span),
+                        )?;
+                    }
+                    ListItem::Spread(spread_span, _) => {
+                        // Spread the list using list-spread
+                        builder.push(
+                            Instruction::ListSpread {
+                                src_dst: out_reg,
+                                items: reg,
+                            }
+                            .into_spanned(*spread_span),
+                        )?;
+                    }
+                }
+            }
+            Ok(())
+        }
+        Expr::Table(table) => {
+            lit(
+                builder,
+                Literal::List {
+                    capacity: table.rows.len(),
+                },
+            )?;
+
+            // Evaluate the columns
+            let column_registers = table
+                .columns
+                .iter()
+                .map(|column| {
+                    let reg = builder.next_register()?;
+                    compile_expression(
+                        working_set,
+                        builder,
+                        column,
+                        RedirectModes::capture_out(column.span),
+                        None,
+                        reg,
+                    )?;
+                    Ok(reg)
+                })
+                .collect::<Result<Vec<RegId>, CompileError>>()?;
+
+            // Build records for each row
+            for row in table.rows.iter() {
+                let row_reg = builder.next_register()?;
+                builder.load_literal(
+                    row_reg,
+                    Literal::Record {
+                        capacity: table.columns.len(),
+                    }
+                    .into_spanned(expr.span),
+                )?;
+                for (column_reg, item) in column_registers.iter().zip(row.iter()) {
+                    let column_reg = builder.clone_reg(*column_reg, item.span)?;
+                    let item_reg = builder.next_register()?;
+                    compile_expression(
+                        working_set,
+                        builder,
+                        item,
+                        RedirectModes::capture_out(item.span),
+                        None,
+                        item_reg,
+                    )?;
+                    builder.push(
+                        Instruction::RecordInsert {
+                            src_dst: row_reg,
+                            key: column_reg,
+                            val: item_reg,
+                        }
+                        .into_spanned(item.span),
+                    )?;
+                }
+                builder.push(
+                    Instruction::ListPush {
+                        src_dst: out_reg,
+                        item: row_reg,
+                    }
+                    .into_spanned(expr.span),
+                )?;
+            }
+
+            // Free the column registers, since they aren't needed anymore
+            for reg in column_registers {
+                builder.drop_reg(reg)?;
+            }
+
+            Ok(())
+        }
+        Expr::Record(items) => {
+            lit(
+                builder,
+                Literal::Record {
+                    capacity: items.len(),
+                },
+            )?;
+
+            for item in items {
+                match item {
+                    RecordItem::Pair(key, val) => {
+                        // Add each item using record-insert
+                        let key_reg = builder.next_register()?;
+                        let val_reg = builder.next_register()?;
+                        compile_expression(
+                            working_set,
+                            builder,
+                            key,
+                            RedirectModes::capture_out(key.span),
+                            None,
+                            key_reg,
+                        )?;
+                        compile_expression(
+                            working_set,
+                            builder,
+                            val,
+                            RedirectModes::capture_out(val.span),
+                            None,
+                            val_reg,
+                        )?;
+                        builder.push(
+                            Instruction::RecordInsert {
+                                src_dst: out_reg,
+                                key: key_reg,
+                                val: val_reg,
+                            }
+                            .into_spanned(expr.span),
+                        )?;
+                    }
+                    RecordItem::Spread(spread_span, expr) => {
+                        // Spread the expression using record-spread
+                        let reg = builder.next_register()?;
+                        compile_expression(
+                            working_set,
+                            builder,
+                            expr,
+                            RedirectModes::capture_out(expr.span),
+                            None,
+                            reg,
+                        )?;
+                        builder.push(
+                            Instruction::RecordSpread {
+                                src_dst: out_reg,
+                                items: reg,
+                            }
+                            .into_spanned(*spread_span),
+                        )?;
+                    }
+                }
+            }
+            Ok(())
+        }
+        Expr::Keyword(kw) => {
+            // keyword: just pass through expr, since commands that use it and are not being
+            // specially handled already are often just positional anyway
+            compile_expression(
+                working_set,
+                builder,
+                &kw.expr,
+                redirect_modes,
+                in_reg,
+                out_reg,
+            )
+        }
+        Expr::ValueWithUnit(value_with_unit) => {
+            lit(builder, literal_from_value_with_unit(value_with_unit)?)
+        }
+        Expr::DateTime(dt) => lit(builder, Literal::Date(Box::new(*dt))),
+        Expr::Filepath(path, no_expand) => {
+            let val = builder.data(path)?;
+            lit(
+                builder,
+                Literal::Filepath {
+                    val,
+                    no_expand: *no_expand,
+                },
+            )
+        }
+        Expr::Directory(path, no_expand) => {
+            let val = builder.data(path)?;
+            lit(
+                builder,
+                Literal::Directory {
+                    val,
+                    no_expand: *no_expand,
+                },
+            )
+        }
+        Expr::GlobPattern(path, no_expand) => {
+            let val = builder.data(path)?;
+            lit(
+                builder,
+                Literal::GlobPattern {
+                    val,
+                    no_expand: *no_expand,
+                },
+            )
+        }
+        Expr::String(s) => {
+            let data_slice = builder.data(s)?;
+            lit(builder, Literal::String(data_slice))
+        }
+        Expr::RawString(rs) => {
+            let data_slice = builder.data(rs)?;
+            lit(builder, Literal::RawString(data_slice))
+        }
+        Expr::CellPath(path) => lit(builder, Literal::CellPath(Box::new(path.clone()))),
+        Expr::FullCellPath(full_cell_path) => {
+            if matches!(full_cell_path.head.expr, Expr::Var(ENV_VARIABLE_ID)) {
+                compile_load_env(builder, expr.span, &full_cell_path.tail, out_reg)
+            } else {
+                compile_expression(
+                    working_set,
+                    builder,
+                    &full_cell_path.head,
+                    RedirectModes::capture_out(expr.span),
+                    in_reg,
+                    out_reg,
+                )?;
+                // Only do the follow if this is actually needed
+                if !full_cell_path.tail.is_empty() {
+                    let cell_path_reg = builder.literal(
+                        Literal::CellPath(Box::new(CellPath {
+                            members: full_cell_path.tail.clone(),
+                        }))
+                        .into_spanned(expr.span),
+                    )?;
+                    builder.push(
+                        Instruction::FollowCellPath {
+                            src_dst: out_reg,
+                            path: cell_path_reg,
+                        }
+                        .into_spanned(expr.span),
+                    )?;
+                }
+                Ok(())
+            }
+        }
+        Expr::ImportPattern(_) => Err(unexpected("ImportPattern")),
+        Expr::Overlay(_) => Err(unexpected("Overlay")),
+        Expr::Signature(_) => ignore(builder), // no effect
+        Expr::StringInterpolation(exprs) | Expr::GlobInterpolation(exprs, _) => {
+            let mut exprs_iter = exprs.iter().peekable();
+
+            if exprs_iter
+                .peek()
+                .is_some_and(|e| matches!(e.expr, Expr::String(..) | Expr::RawString(..)))
+            {
+                // If the first expression is a string or raw string literal, just take it and build
+                // from that
+                compile_expression(
+                    working_set,
+                    builder,
+                    exprs_iter.next().expect("peek() was Some"),
+                    RedirectModes::capture_out(expr.span),
+                    None,
+                    out_reg,
+                )?;
+            } else {
+                // Start with an empty string
+                lit(builder, Literal::String(DataSlice::empty()))?;
+            }
+
+            // Compile each expression and append to out_reg
+            for expr in exprs_iter {
+                let scratch_reg = builder.next_register()?;
+                compile_expression(
+                    working_set,
+                    builder,
+                    expr,
+                    RedirectModes::capture_out(expr.span),
+                    None,
+                    scratch_reg,
+                )?;
+                builder.push(
+                    Instruction::StringAppend {
+                        src_dst: out_reg,
+                        val: scratch_reg,
+                    }
+                    .into_spanned(expr.span),
+                )?;
+            }
+
+            // If it's a glob interpolation, change it to a glob
+            if let Expr::GlobInterpolation(_, no_expand) = expr.expr {
+                builder.push(
+                    Instruction::GlobFrom {
+                        src_dst: out_reg,
+                        no_expand,
+                    }
+                    .into_spanned(expr.span),
+                )?;
+            }
+
+            Ok(())
+        }
+        Expr::Nothing => lit(builder, Literal::Nothing),
+        Expr::Garbage => Err(CompileError::Garbage { span: expr.span }),
+    }
+}
+
+fn literal_from_value_with_unit(value_with_unit: &ValueWithUnit) -> Result<Literal, CompileError> {
+    let Expr::Int(int_value) = value_with_unit.expr.expr else {
+        return Err(CompileError::UnexpectedExpression {
+            expr_name: format!("{:?}", value_with_unit.expr),
+            span: value_with_unit.expr.span,
+        });
+    };
+
+    match value_with_unit
+        .unit
+        .item
+        .build_value(int_value, Span::unknown())
+        .map_err(|err| CompileError::InvalidLiteral {
+            msg: err.to_string(),
+            span: value_with_unit.expr.span,
+        })? {
+        Value::Filesize { val, .. } => Ok(Literal::Filesize(val)),
+        Value::Duration { val, .. } => Ok(Literal::Duration(val)),
+        other => Err(CompileError::InvalidLiteral {
+            msg: format!("bad value returned by Unit::build_value(): {other:?}"),
+            span: value_with_unit.unit.span,
+        }),
+    }
+}

crates/nu-engine/src/compile/keyword.rs

@@ -0,0 +1,902 @@
+use nu_protocol::{
+    ast::{Block, Call, Expr, Expression},
+    engine::StateWorkingSet,
+    ir::Instruction,
+    IntoSpanned, RegId, Type, VarId,
+};
+
+use super::{compile_block, compile_expression, BlockBuilder, CompileError, RedirectModes};
+
+/// Compile a call to `if` as a branch-if
+pub(crate) fn compile_if(
+    working_set: &StateWorkingSet,
+    builder: &mut BlockBuilder,
+    call: &Call,
+    redirect_modes: RedirectModes,
+    io_reg: RegId,
+) -> Result<(), CompileError> {
+    // Pseudocode:
+    //
+    //        %io_reg <- <condition>
+    //        not %io_reg
+    //        branch-if %io_reg, FALSE
+    // TRUE:  ...<true_block>...
+    //        jump END
+    // FALSE: ...<else_expr>... OR drop %io_reg
+    // END:
+    let invalid = || CompileError::InvalidKeywordCall {
+        keyword: "if".into(),
+        span: call.head,
+    };
+
+    let condition = call.positional_nth(0).ok_or_else(invalid)?;
+    let true_block_arg = call.positional_nth(1).ok_or_else(invalid)?;
+    let else_arg = call.positional_nth(2);
+
+    let true_block_id = true_block_arg.as_block().ok_or_else(invalid)?;
+    let true_block = working_set.get_block(true_block_id);
+
+    let true_label = builder.label(None);
+    let false_label = builder.label(None);
+    let end_label = builder.label(None);
+
+    let not_condition_reg = {
+        // Compile the condition first
+        let condition_reg = builder.next_register()?;
+        compile_expression(
+            working_set,
+            builder,
+            condition,
+            RedirectModes::capture_out(condition.span),
+            None,
+            condition_reg,
+        )?;
+
+        // Negate the condition - we basically only want to jump if the condition is false
+        builder.push(
+            Instruction::Not {
+                src_dst: condition_reg,
+            }
+            .into_spanned(call.head),
+        )?;
+
+        condition_reg
+    };
+
+    // Set up a branch if the condition is false.
+    builder.branch_if(not_condition_reg, false_label, call.head)?;
+    builder.add_comment("if false");
+
+    // Compile the true case
+    builder.set_label(true_label, builder.here())?;
+    compile_block(
+        working_set,
+        builder,
+        true_block,
+        redirect_modes.clone(),
+        Some(io_reg),
+        io_reg,
+    )?;
+
+    // Add a jump over the false case
+    builder.jump(end_label, else_arg.map(|e| e.span).unwrap_or(call.head))?;
+    builder.add_comment("end if");
+
+    // On the else side now, assert that io_reg is still valid
+    builder.set_label(false_label, builder.here())?;
+    builder.mark_register(io_reg)?;
+
+    if let Some(else_arg) = else_arg {
+        let Expression {
+            expr: Expr::Keyword(else_keyword),
+            ..
+        } = else_arg
+        else {
+            return Err(invalid());
+        };
+
+        if else_keyword.keyword.as_ref() != b"else" {
+            return Err(invalid());
+        }
+
+        let else_expr = &else_keyword.expr;
+
+        match &else_expr.expr {
+            Expr::Block(block_id) => {
+                let false_block = working_set.get_block(*block_id);
+                compile_block(
+                    working_set,
+                    builder,
+                    false_block,
+                    redirect_modes,
+                    Some(io_reg),
+                    io_reg,
+                )?;
+            }
+            _ => {
+                // The else case supports bare expressions too, not only blocks
+                compile_expression(
+                    working_set,
+                    builder,
+                    else_expr,
+                    redirect_modes,
+                    Some(io_reg),
+                    io_reg,
+                )?;
+            }
+        }
+    } else {
+        // We don't have an else expression/block, so just set io_reg = Empty
+        builder.load_empty(io_reg)?;
+    }
+
+    // Set the end label
+    builder.set_label(end_label, builder.here())?;
+
+    Ok(())
+}
+
+/// Compile a call to `match`
+pub(crate) fn compile_match(
+    working_set: &StateWorkingSet,
+    builder: &mut BlockBuilder,
+    call: &Call,
+    redirect_modes: RedirectModes,
+    io_reg: RegId,
+) -> Result<(), CompileError> {
+    // Pseudocode:
+    //
+    //         %match_reg <- <match_expr>
+    //         collect %match_reg
+    //         match (pat1), %match_reg, PAT1
+    // MATCH2: match (pat2), %match_reg, PAT2
+    // FAIL:   drop %io_reg
+    //         drop %match_reg
+    //         jump END
+    // PAT1:   %guard_reg <- <guard_expr>
+    //         check-match-guard %guard_reg
+    //         not %guard_reg
+    //         branch-if %guard_reg, MATCH2
+    //         drop %match_reg
+    //         <...expr...>
+    //         jump END
+    // PAT2:   drop %match_reg
+    //         <...expr...>
+    //         jump END
+    // END:
+    let invalid = || CompileError::InvalidKeywordCall {
+        keyword: "match".into(),
+        span: call.head,
+    };
+
+    let match_expr = call.positional_nth(0).ok_or_else(invalid)?;
+
+    let match_block_arg = call.positional_nth(1).ok_or_else(invalid)?;
+    let match_block = match_block_arg.as_match_block().ok_or_else(invalid)?;
+
+    let match_reg = builder.next_register()?;
+
+    // Evaluate the match expression (patterns will be checked against this).
+    compile_expression(
+        working_set,
+        builder,
+        match_expr,
+        RedirectModes::capture_out(match_expr.span),
+        None,
+        match_reg,
+    )?;
+
+    // Important to collect it first
+    builder.push(Instruction::Collect { src_dst: match_reg }.into_spanned(match_expr.span))?;
+
+    // Generate the `match` instructions. Guards are not used at this stage.
+    let mut match_labels = Vec::with_capacity(match_block.len());
+    let mut next_labels = Vec::with_capacity(match_block.len());
+    let end_label = builder.label(None);
+
+    for (pattern, _) in match_block {
+        let match_label = builder.label(None);
+        match_labels.push(match_label);
+        builder.push(
+            Instruction::Match {
+                pattern: Box::new(pattern.pattern.clone()),
+                src: match_reg,
+                index: match_label.0,
+            }
+            .into_spanned(pattern.span),
+        )?;
+        // Also add a label for the next match instruction or failure case
+        next_labels.push(builder.label(Some(builder.here())));
+    }
+
+    // Match fall-through to jump to the end, if no match
+    builder.load_empty(io_reg)?;
+    builder.drop_reg(match_reg)?;
+    builder.jump(end_label, call.head)?;
+
+    // Generate each of the match expressions. Handle guards here, if present.
+    for (index, (pattern, expr)) in match_block.iter().enumerate() {
+        let match_label = match_labels[index];
+        let next_label = next_labels[index];
+
+        // `io_reg` and `match_reg` are still valid at each of these branch targets
+        builder.mark_register(io_reg)?;
+        builder.mark_register(match_reg)?;
+
+        // Set the original match instruction target here
+        builder.set_label(match_label, builder.here())?;
+
+        // Handle guard, if present
+        if let Some(guard) = &pattern.guard {
+            let guard_reg = builder.next_register()?;
+            compile_expression(
+                working_set,
+                builder,
+                guard,
+                RedirectModes::capture_out(guard.span),
+                None,
+                guard_reg,
+            )?;
+            builder
+                .push(Instruction::CheckMatchGuard { src: guard_reg }.into_spanned(guard.span))?;
+            builder.push(Instruction::Not { src_dst: guard_reg }.into_spanned(guard.span))?;
+            // Branch to the next match instruction if the branch fails to match
+            builder.branch_if(
+                guard_reg,
+                next_label,
+                // Span the branch with the next pattern, or the head if this is the end
+                match_block
+                    .get(index + 1)
+                    .map(|b| b.0.span)
+                    .unwrap_or(call.head),
+            )?;
+            builder.add_comment("if match guard false");
+        }
+
+        // match_reg no longer needed, successful match
+        builder.drop_reg(match_reg)?;
+
+        // Execute match right hand side expression
+        if let Some(block_id) = expr.as_block() {
+            let block = working_set.get_block(block_id);
+            compile_block(
+                working_set,
+                builder,
+                block,
+                redirect_modes.clone(),
+                Some(io_reg),
+                io_reg,
+            )?;
+        } else {
+            compile_expression(
+                working_set,
+                builder,
+                expr,
+                redirect_modes.clone(),
+                Some(io_reg),
+                io_reg,
+            )?;
+        }
+
+        // Jump to the end after the match logic is done
+        builder.jump(end_label, call.head)?;
+        builder.add_comment("end match");
+    }
+
+    // Set the end destination
+    builder.set_label(end_label, builder.here())?;
+
+    Ok(())
+}
+
+/// Compile a call to `let` or `mut` (just do store-variable)
+pub(crate) fn compile_let(
+    working_set: &StateWorkingSet,
+    builder: &mut BlockBuilder,
+    call: &Call,
+    _redirect_modes: RedirectModes,
+    io_reg: RegId,
+) -> Result<(), CompileError> {
+    // Pseudocode:
+    //
+    // %io_reg <- ...<block>... <- %io_reg
+    // store-variable $var, %io_reg
+    let invalid = || CompileError::InvalidKeywordCall {
+        keyword: "let".into(),
+        span: call.head,
+    };
+
+    let var_decl_arg = call.positional_nth(0).ok_or_else(invalid)?;
+    let block_arg = call.positional_nth(1).ok_or_else(invalid)?;
+
+    let var_id = var_decl_arg.as_var().ok_or_else(invalid)?;
+    let block_id = block_arg.as_block().ok_or_else(invalid)?;
+    let block = working_set.get_block(block_id);
+
+    let variable = working_set.get_variable(var_id);
+
+    compile_block(
+        working_set,
+        builder,
+        block,
+        RedirectModes::capture_out(call.head),
+        Some(io_reg),
+        io_reg,
+    )?;
+
+    // If the variable is a glob type variable, we should cast it with GlobFrom
+    if variable.ty == Type::Glob {
+        builder.push(
+            Instruction::GlobFrom {
+                src_dst: io_reg,
+                no_expand: true,
+            }
+            .into_spanned(call.head),
+        )?;
+    }
+
+    builder.push(
+        Instruction::StoreVariable {
+            var_id,
+            src: io_reg,
+        }
+        .into_spanned(call.head),
+    )?;
+    builder.add_comment("let");
+
+    // Don't forget to set io_reg to Empty afterward, as that's the result of an assignment
+    builder.load_empty(io_reg)?;
+
+    Ok(())
+}
+
+/// Compile a call to `try`, setting an error handler over the evaluated block
+pub(crate) fn compile_try(
+    working_set: &StateWorkingSet,
+    builder: &mut BlockBuilder,
+    call: &Call,
+    redirect_modes: RedirectModes,
+    io_reg: RegId,
+) -> Result<(), CompileError> {
+    // Pseudocode (literal block):
+    //
+    //       on-error-into ERR, %io_reg           // or without
+    //       %io_reg <- <...block...> <- %io_reg
+    //       check-external-failed %failed_reg, %io_reg
+    //       branch-if %failed_reg, FAIL
+    //       pop-error-handler
+    //       jump END
+    // FAIL: drain %io_reg
+    //       unreachable
+    // ERR:  clone %err_reg, %io_reg
+    //       store-variable $err_var, %err_reg         // or without
+    //       %io_reg <- <...catch block...> <- %io_reg // set to empty if no catch block
+    // END:
+    //
+    // with expression that can't be inlined:
+    //
+    //       %closure_reg <- <catch_expr>
+    //       on-error-into ERR, %io_reg
+    //       %io_reg <- <...block...> <- %io_reg
+    //       check-external-failed %failed_reg, %io_reg
+    //       branch-if %failed_reg, FAIL
+    //       pop-error-handler
+    //       jump END
+    // FAIL: drain %io_reg
+    //       unreachable
+    // ERR:  clone %err_reg, %io_reg
+    //       push-positional %closure_reg
+    //       push-positional %err_reg
+    //       call "do", %io_reg
+    // END:
+    let invalid = || CompileError::InvalidKeywordCall {
+        keyword: "try".into(),
+        span: call.head,
+    };
+
+    let block_arg = call.positional_nth(0).ok_or_else(invalid)?;
+    let block_id = block_arg.as_block().ok_or_else(invalid)?;
+    let block = working_set.get_block(block_id);
+
+    let catch_expr = match call.positional_nth(1) {
+        Some(kw_expr) => Some(kw_expr.as_keyword().ok_or_else(invalid)?),
+        None => None,
+    };
+    let catch_span = catch_expr.map(|e| e.span).unwrap_or(call.head);
+
+    let err_label = builder.label(None);
+    let failed_label = builder.label(None);
+    let end_label = builder.label(None);
+
+    // We have two ways of executing `catch`: if it was provided as a literal, we can inline it.
+    // Otherwise, we have to evaluate the expression and keep it as a register, and then call `do`.
+    enum CatchType<'a> {
+        Block {
+            block: &'a Block,
+            var_id: Option<VarId>,
+        },
+        Closure {
+            closure_reg: RegId,
+        },
+    }
+
+    let catch_type = catch_expr
+        .map(|catch_expr| match catch_expr.as_block() {
+            Some(block_id) => {
+                let block = working_set.get_block(block_id);
+                let var_id = block.signature.get_positional(0).and_then(|v| v.var_id);
+                Ok(CatchType::Block { block, var_id })
+            }
+            None => {
+                // We have to compile the catch_expr and use it as a closure
+                let closure_reg = builder.next_register()?;
+                compile_expression(
+                    working_set,
+                    builder,
+                    catch_expr,
+                    RedirectModes::capture_out(catch_expr.span),
+                    None,
+                    closure_reg,
+                )?;
+                Ok(CatchType::Closure { closure_reg })
+            }
+        })
+        .transpose()?;
+
+    // Put the error handler instruction. If we have a catch expression then we should capture the
+    // error.
+    if catch_type.is_some() {
+        builder.push(
+            Instruction::OnErrorInto {
+                index: err_label.0,
+                dst: io_reg,
+            }
+            .into_spanned(call.head),
+        )?
+    } else {
+        // Otherwise, we don't need the error value.
+        builder.push(Instruction::OnError { index: err_label.0 }.into_spanned(call.head))?
+    };
+
+    builder.add_comment("try");
+
+    // Compile the block
+    compile_block(
+        working_set,
+        builder,
+        block,
+        redirect_modes.clone(),
+        Some(io_reg),
+        io_reg,
+    )?;
+
+    // Check for external command exit code failure, and also redirect that to the catch handler
+    let failed_reg = builder.next_register()?;
+    builder.push(
+        Instruction::CheckExternalFailed {
+            dst: failed_reg,
+            src: io_reg,
+        }
+        .into_spanned(catch_span),
+    )?;
+    builder.branch_if(failed_reg, failed_label, catch_span)?;
+
+    // Successful case: pop the error handler
+    builder.push(Instruction::PopErrorHandler.into_spanned(call.head))?;
+
+    // Jump over the failure case
+    builder.jump(end_label, catch_span)?;
+
+    // Set up an error handler preamble for failed external.
+    // Draining the %io_reg results in the error handler being called with Empty, and sets
+    // $env.LAST_EXIT_CODE
+    builder.set_label(failed_label, builder.here())?;
+    builder.drain(io_reg, catch_span)?;
+    builder.add_comment("branches to err");
+    builder.unreachable(catch_span)?;
+
+    // This is the real error handler
+    builder.set_label(err_label, builder.here())?;
+
+    // Mark out register as likely not clean - state in error handler is not well defined
+    builder.mark_register(io_reg)?;
+
+    // Now compile whatever is necessary for the error handler
+    match catch_type {
+        Some(CatchType::Block { block, var_id }) => {
+            // Error will be in io_reg
+            builder.mark_register(io_reg)?;
+            if let Some(var_id) = var_id {
+                // Take a copy of the error as $err, since it will also be input
+                let err_reg = builder.next_register()?;
+                builder.push(
+                    Instruction::Clone {
+                        dst: err_reg,
+                        src: io_reg,
+                    }
+                    .into_spanned(catch_span),
+                )?;
+                builder.push(
+                    Instruction::StoreVariable {
+                        var_id,
+                        src: err_reg,
+                    }
+                    .into_spanned(catch_span),
+                )?;
+            }
+            // Compile the block, now that the variable is set
+            compile_block(
+                working_set,
+                builder,
+                block,
+                redirect_modes,
+                Some(io_reg),
+                io_reg,
+            )?;
+        }
+        Some(CatchType::Closure { closure_reg }) => {
+            // We should call `do`. Error will be in io_reg
+            let do_decl_id = working_set.find_decl(b"do").ok_or_else(|| {
+                CompileError::MissingRequiredDeclaration {
+                    decl_name: "do".into(),
+                    span: call.head,
+                }
+            })?;
+
+            // Take a copy of io_reg, because we pass it both as an argument and input
+            builder.mark_register(io_reg)?;
+            let err_reg = builder.next_register()?;
+            builder.push(
+                Instruction::Clone {
+                    dst: err_reg,
+                    src: io_reg,
+                }
+                .into_spanned(catch_span),
+            )?;
+
+            // Push the closure and the error
+            builder
+                .push(Instruction::PushPositional { src: closure_reg }.into_spanned(catch_span))?;
+            builder.push(Instruction::PushPositional { src: err_reg }.into_spanned(catch_span))?;
+
+            // Call `$err | do $closure $err`
+            builder.push(
+                Instruction::Call {
+                    decl_id: do_decl_id,
+                    src_dst: io_reg,
+                }
+                .into_spanned(catch_span),
+            )?;
+        }
+        None => {
+            // Just set out to empty.
+            builder.load_empty(io_reg)?;
+        }
+    }
+
+    // This is the end - if we succeeded, should jump here
+    builder.set_label(end_label, builder.here())?;
+
+    Ok(())
+}
+
+/// Compile a call to `loop` (via `jump`)
+pub(crate) fn compile_loop(
+    working_set: &StateWorkingSet,
+    builder: &mut BlockBuilder,
+    call: &Call,
+    _redirect_modes: RedirectModes,
+    io_reg: RegId,
+) -> Result<(), CompileError> {
+    // Pseudocode:
+    //
+    //       drop %io_reg
+    // LOOP: %io_reg <- ...<block>...
+    //       drain %io_reg
+    //       jump %LOOP
+    // END:  drop %io_reg
+    let invalid = || CompileError::InvalidKeywordCall {
+        keyword: "loop".into(),
+        span: call.head,
+    };
+
+    let block_arg = call.positional_nth(0).ok_or_else(invalid)?;
+    let block_id = block_arg.as_block().ok_or_else(invalid)?;
+    let block = working_set.get_block(block_id);
+
+    let loop_ = builder.begin_loop();
+    builder.load_empty(io_reg)?;
+
+    builder.set_label(loop_.continue_label, builder.here())?;
+
+    compile_block(
+        working_set,
+        builder,
+        block,
+        RedirectModes::default(),
+        None,
+        io_reg,
+    )?;
+
+    // Drain the output, just like for a semicolon
+    builder.drain(io_reg, call.head)?;
+
+    builder.jump(loop_.continue_label, call.head)?;
+    builder.add_comment("loop");
+
+    builder.set_label(loop_.break_label, builder.here())?;
+    builder.end_loop(loop_)?;
+
+    // State of %io_reg is not necessarily well defined here due to control flow, so make sure it's
+    // empty.
+    builder.mark_register(io_reg)?;
+    builder.load_empty(io_reg)?;
+
+    Ok(())
+}
+
+/// Compile a call to `while`, via branch instructions
+pub(crate) fn compile_while(
+    working_set: &StateWorkingSet,
+    builder: &mut BlockBuilder,
+    call: &Call,
+    _redirect_modes: RedirectModes,
+    io_reg: RegId,
+) -> Result<(), CompileError> {
+    // Pseudocode:
+    //
+    // LOOP:  %io_reg <- <condition>
+    //        branch-if %io_reg, TRUE
+    //        jump FALSE
+    // TRUE:  %io_reg <- ...<block>...
+    //        drain %io_reg
+    //        jump LOOP
+    // FALSE: drop %io_reg
+    let invalid = || CompileError::InvalidKeywordCall {
+        keyword: "while".into(),
+        span: call.head,
+    };
+
+    let cond_arg = call.positional_nth(0).ok_or_else(invalid)?;
+    let block_arg = call.positional_nth(1).ok_or_else(invalid)?;
+    let block_id = block_arg.as_block().ok_or_else(invalid)?;
+    let block = working_set.get_block(block_id);
+
+    let loop_ = builder.begin_loop();
+    builder.set_label(loop_.continue_label, builder.here())?;
+
+    let true_label = builder.label(None);
+
+    compile_expression(
+        working_set,
+        builder,
+        cond_arg,
+        RedirectModes::capture_out(call.head),
+        None,
+        io_reg,
+    )?;
+
+    builder.branch_if(io_reg, true_label, call.head)?;
+    builder.add_comment("while");
+    builder.jump(loop_.break_label, call.head)?;
+    builder.add_comment("end while");
+
+    builder.set_label(true_label, builder.here())?;
+
+    compile_block(
+        working_set,
+        builder,
+        block,
+        RedirectModes::default(),
+        None,
+        io_reg,
+    )?;
+
+    // Drain the result, just like for a semicolon
+    builder.drain(io_reg, call.head)?;
+
+    builder.jump(loop_.continue_label, call.head)?;
+    builder.add_comment("while");
+
+    builder.set_label(loop_.break_label, builder.here())?;
+    builder.end_loop(loop_)?;
+
+    // State of %io_reg is not necessarily well defined here due to control flow, so make sure it's
+    // empty.
+    builder.mark_register(io_reg)?;
+    builder.load_empty(io_reg)?;
+
+    Ok(())
+}
+
+/// Compile a call to `for` (via `iterate`)
+pub(crate) fn compile_for(
+    working_set: &StateWorkingSet,
+    builder: &mut BlockBuilder,
+    call: &Call,
+    _redirect_modes: RedirectModes,
+    io_reg: RegId,
+) -> Result<(), CompileError> {
+    // Pseudocode:
+    //
+    //       %stream_reg <- <in_expr>
+    // LOOP: iterate %io_reg, %stream_reg, END
+    //       store-variable $var, %io_reg
+    //       %io_reg <- <...block...>
+    //       drain %io_reg
+    //       jump LOOP
+    // END:  drop %io_reg
+    let invalid = || CompileError::InvalidKeywordCall {
+        keyword: "for".into(),
+        span: call.head,
+    };
+
+    if call.get_named_arg("numbered").is_some() {
+        // This is deprecated and we don't support it.
+        return Err(invalid());
+    }
+
+    let var_decl_arg = call.positional_nth(0).ok_or_else(invalid)?;
+    let var_id = var_decl_arg.as_var().ok_or_else(invalid)?;
+
+    let in_arg = call.positional_nth(1).ok_or_else(invalid)?;
+    let in_expr = in_arg.as_keyword().ok_or_else(invalid)?;
+
+    let block_arg = call.positional_nth(2).ok_or_else(invalid)?;
+    let block_id = block_arg.as_block().ok_or_else(invalid)?;
+    let block = working_set.get_block(block_id);
+
+    // Ensure io_reg is marked so we don't use it
+    builder.mark_register(io_reg)?;
+
+    let stream_reg = builder.next_register()?;
+
+    compile_expression(
+        working_set,
+        builder,
+        in_expr,
+        RedirectModes::capture_out(in_expr.span),
+        None,
+        stream_reg,
+    )?;
+
+    // Set up loop state
+    let loop_ = builder.begin_loop();
+    builder.set_label(loop_.continue_label, builder.here())?;
+
+    // This gets a value from the stream each time it's executed
+    // io_reg basically will act as our scratch register here
+    builder.push(
+        Instruction::Iterate {
+            dst: io_reg,
+            stream: stream_reg,
+            end_index: loop_.break_label.0,
+        }
+        .into_spanned(call.head),
+    )?;
+    builder.add_comment("for");
+
+    // Put the received value in the variable
+    builder.push(
+        Instruction::StoreVariable {
+            var_id,
+            src: io_reg,
+        }
+        .into_spanned(var_decl_arg.span),
+    )?;
+
+    // Do the body of the block
+    compile_block(
+        working_set,
+        builder,
+        block,
+        RedirectModes::default(),
+        None,
+        io_reg,
+    )?;
+
+    // Drain the output, just like for a semicolon
+    builder.drain(io_reg, call.head)?;
+
+    // Loop back to iterate to get the next value
+    builder.jump(loop_.continue_label, call.head)?;
+
+    // Set the end of the loop
+    builder.set_label(loop_.break_label, builder.here())?;
+    builder.end_loop(loop_)?;
+
+    // We don't need stream_reg anymore, after the loop
+    // io_reg may or may not be empty, so be sure it is
+    builder.free_register(stream_reg)?;
+    builder.mark_register(io_reg)?;
+    builder.load_empty(io_reg)?;
+
+    Ok(())
+}
+
+/// Compile a call to `break`.
+pub(crate) fn compile_break(
+    _working_set: &StateWorkingSet,
+    builder: &mut BlockBuilder,
+    call: &Call,
+    _redirect_modes: RedirectModes,
+    io_reg: RegId,
+) -> Result<(), CompileError> {
+    if builder.is_in_loop() {
+        builder.load_empty(io_reg)?;
+        builder.push_break(call.head)?;
+        builder.add_comment("break");
+    } else {
+        // Fall back to calling the command if we can't find the loop target statically
+        builder.push(
+            Instruction::Call {
+                decl_id: call.decl_id,
+                src_dst: io_reg,
+            }
+            .into_spanned(call.head),
+        )?;
+    }
+    Ok(())
+}
+
+/// Compile a call to `continue`.
+pub(crate) fn compile_continue(
+    _working_set: &StateWorkingSet,
+    builder: &mut BlockBuilder,
+    call: &Call,
+    _redirect_modes: RedirectModes,
+    io_reg: RegId,
+) -> Result<(), CompileError> {
+    if builder.is_in_loop() {
+        builder.load_empty(io_reg)?;
+        builder.push_continue(call.head)?;
+        builder.add_comment("continue");
+    } else {
+        // Fall back to calling the command if we can't find the loop target statically
+        builder.push(
+            Instruction::Call {
+                decl_id: call.decl_id,
+                src_dst: io_reg,
+            }
+            .into_spanned(call.head),
+        )?;
+    }
+    Ok(())
+}
+
+/// Compile a call to `return` as a `return-early` instruction.
+///
+/// This is not strictly necessary, but it is more efficient.
+pub(crate) fn compile_return(
+    working_set: &StateWorkingSet,
+    builder: &mut BlockBuilder,
+    call: &Call,
+    _redirect_modes: RedirectModes,
+    io_reg: RegId,
+) -> Result<(), CompileError> {
+    // Pseudocode:
+    //
+    // %io_reg <- <arg_expr>
+    // return-early %io_reg
+    if let Some(arg_expr) = call.positional_nth(0) {
+        compile_expression(
+            working_set,
+            builder,
+            arg_expr,
+            RedirectModes::capture_out(arg_expr.span),
+            None,
+            io_reg,
+        )?;
+    } else {
+        builder.load_empty(io_reg)?;
+    }
+
+    // TODO: It would be nice if this could be `return` instead, but there is a little bit of
+    // behaviour remaining that still depends on `ShellError::Return`
+    builder.push(Instruction::ReturnEarly { src: io_reg }.into_spanned(call.head))?;
+
+    // io_reg is supposed to remain allocated
+    builder.load_empty(io_reg)?;
+
+    Ok(())
+}

crates/nu-engine/src/compile/mod.rs

@@ -0,0 +1,204 @@
+use nu_protocol::{
+    ast::{Block, Pipeline, PipelineRedirection, RedirectionSource, RedirectionTarget},
+    engine::StateWorkingSet,
+    ir::{Instruction, IrBlock, RedirectMode},
+    CompileError, IntoSpanned, RegId, Span,
+};
+
+mod builder;
+mod call;
+mod expression;
+mod keyword;
+mod operator;
+mod redirect;
+
+use builder::BlockBuilder;
+use call::*;
+use expression::compile_expression;
+use operator::*;
+use redirect::*;
+
+const BLOCK_INPUT: RegId = RegId(0);
+
+/// Compile Nushell pipeline abstract syntax tree (AST) to internal representation (IR) instructions
+/// for evaluation.
+pub fn compile(working_set: &StateWorkingSet, block: &Block) -> Result<IrBlock, CompileError> {
+    let mut builder = BlockBuilder::new(block.span);
+
+    let span = block.span.unwrap_or(Span::unknown());
+
+    compile_block(
+        working_set,
+        &mut builder,
+        block,
+        RedirectModes::caller(span),
+        Some(BLOCK_INPUT),
+        BLOCK_INPUT,
+    )?;
+
+    // A complete block has to end with a `return`
+    builder.push(Instruction::Return { src: BLOCK_INPUT }.into_spanned(span))?;
+
+    builder.finish()
+}
+
+/// Compiles a [`Block`] in-place into an IR block. This can be used in a nested manner, for example
+/// by [`compile_if()`], where the instructions for the blocks for the if/else are inlined into the
+/// top-level IR block.
+fn compile_block(
+    working_set: &StateWorkingSet,
+    builder: &mut BlockBuilder,
+    block: &Block,
+    redirect_modes: RedirectModes,
+    in_reg: Option<RegId>,
+    out_reg: RegId,
+) -> Result<(), CompileError> {
+    let span = block.span.unwrap_or(Span::unknown());
+    let mut redirect_modes = Some(redirect_modes);
+    if !block.pipelines.is_empty() {
+        let last_index = block.pipelines.len() - 1;
+        for (index, pipeline) in block.pipelines.iter().enumerate() {
+            compile_pipeline(
+                working_set,
+                builder,
+                pipeline,
+                span,
+                // the redirect mode only applies to the last pipeline.
+                if index == last_index {
+                    redirect_modes
+                        .take()
+                        .expect("should only take redirect_modes once")
+                } else {
+                    RedirectModes::default()
+                },
+                // input is only passed to the first pipeline.
+                if index == 0 { in_reg } else { None },
+                out_reg,
+            )?;
+
+            if index != last_index {
+                // Explicitly drain the out reg after each non-final pipeline, because that's how
+                // the semicolon functions.
+                if builder.is_allocated(out_reg) {
+                    builder.push(Instruction::Drain { src: out_reg }.into_spanned(span))?;
+                }
+                builder.load_empty(out_reg)?;
+            }
+        }
+        Ok(())
+    } else if in_reg.is_none() {
+        builder.load_empty(out_reg)
+    } else {
+        Ok(())
+    }
+}
+
+fn compile_pipeline(
+    working_set: &StateWorkingSet,
+    builder: &mut BlockBuilder,
+    pipeline: &Pipeline,
+    fallback_span: Span,
+    redirect_modes: RedirectModes,
+    in_reg: Option<RegId>,
+    out_reg: RegId,
+) -> Result<(), CompileError> {
+    let mut iter = pipeline.elements.iter().peekable();
+    let mut in_reg = in_reg;
+    let mut redirect_modes = Some(redirect_modes);
+    while let Some(element) = iter.next() {
+        let span = element.pipe.unwrap_or(fallback_span);
+
+        // We have to get the redirection mode from either the explicit redirection in the pipeline
+        // element, or from the next expression if it's specified there. If this is the last
+        // element, then it's from whatever is passed in as the mode to use.
+
+        let next_redirect_modes = if let Some(next_element) = iter.peek() {
+            let mut modes = redirect_modes_of_expression(working_set, &next_element.expr, span)?;
+
+            // If there's a next element with no inherent redirection we always pipe out *unless*
+            // this is a single redirection of stderr to pipe (e>|)
+            if modes.out.is_none()
+                && !matches!(
+                    element.redirection,
+                    Some(PipelineRedirection::Single {
+                        source: RedirectionSource::Stderr,
+                        target: RedirectionTarget::Pipe { .. }
+                    })
+                )
+            {
+                let pipe_span = next_element.pipe.unwrap_or(next_element.expr.span);
+                modes.out = Some(RedirectMode::Pipe.into_spanned(pipe_span));
+            }
+
+            modes
+        } else {
+            redirect_modes
+                .take()
+                .expect("should only take redirect_modes once")
+        };
+
+        let spec_redirect_modes = match &element.redirection {
+            Some(PipelineRedirection::Single { source, target }) => {
+                let mode = redirection_target_to_mode(working_set, builder, target)?;
+                match source {
+                    RedirectionSource::Stdout => RedirectModes {
+                        out: Some(mode),
+                        err: None,
+                    },
+                    RedirectionSource::Stderr => RedirectModes {
+                        out: None,
+                        err: Some(mode),
+                    },
+                    RedirectionSource::StdoutAndStderr => RedirectModes {
+                        out: Some(mode),
+                        err: Some(mode),
+                    },
+                }
+            }
+            Some(PipelineRedirection::Separate { out, err }) => {
+                // In this case, out and err must not both be Pipe
+                assert!(
+                    !matches!(
+                        (out, err),
+                        (
+                            RedirectionTarget::Pipe { .. },
+                            RedirectionTarget::Pipe { .. }
+                        )
+                    ),
+                    "for Separate redirection, out and err targets must not both be Pipe"
+                );
+                let out = redirection_target_to_mode(working_set, builder, out)?;
+                let err = redirection_target_to_mode(working_set, builder, err)?;
+                RedirectModes {
+                    out: Some(out),
+                    err: Some(err),
+                }
+            }
+            None => RedirectModes {
+                out: None,
+                err: None,
+            },
+        };
+
+        let redirect_modes = RedirectModes {
+            out: spec_redirect_modes.out.or(next_redirect_modes.out),
+            err: spec_redirect_modes.err.or(next_redirect_modes.err),
+        };
+
+        compile_expression(
+            working_set,
+            builder,
+            &element.expr,
+            redirect_modes.clone(),
+            in_reg,
+            out_reg,
+        )?;
+
+        // Clean up the redirection
+        finish_redirection(builder, redirect_modes, out_reg)?;
+
+        // The next pipeline element takes input from this output
+        in_reg = Some(out_reg);
+    }
+    Ok(())
+}

crates/nu-engine/src/compile/operator.rs

@@ -0,0 +1,378 @@
+use nu_protocol::{
+    ast::{Assignment, Boolean, CellPath, Expr, Expression, Math, Operator, PathMember},
+    engine::StateWorkingSet,
+    ir::{Instruction, Literal},
+    IntoSpanned, RegId, Span, Spanned, ENV_VARIABLE_ID,
+};
+use nu_utils::IgnoreCaseExt;
+
+use super::{compile_expression, BlockBuilder, CompileError, RedirectModes};
+
+pub(crate) fn compile_binary_op(
+    working_set: &StateWorkingSet,
+    builder: &mut BlockBuilder,
+    lhs: &Expression,
+    op: Spanned<Operator>,
+    rhs: &Expression,
+    span: Span,
+    out_reg: RegId,
+) -> Result<(), CompileError> {
+    if let Operator::Assignment(assign_op) = op.item {
+        if let Some(decomposed_op) = decompose_assignment(assign_op) {
+            // Compiling an assignment that uses a binary op with the existing value
+            compile_binary_op(
+                working_set,
+                builder,
+                lhs,
+                decomposed_op.into_spanned(op.span),
+                rhs,
+                span,
+                out_reg,
+            )?;
+        } else {
+            // Compiling a plain assignment, where the current left-hand side value doesn't matter
+            compile_expression(
+                working_set,
+                builder,
+                rhs,
+                RedirectModes::capture_out(rhs.span),
+                None,
+                out_reg,
+            )?;
+        }
+
+        compile_assignment(working_set, builder, lhs, op.span, out_reg)?;
+
+        // Load out_reg with Nothing, as that's the result of an assignment
+        builder.load_literal(out_reg, Literal::Nothing.into_spanned(op.span))
+    } else {
+        // Not an assignment: just do the binary op
+        let lhs_reg = out_reg;
+
+        compile_expression(
+            working_set,
+            builder,
+            lhs,
+            RedirectModes::capture_out(lhs.span),
+            None,
+            lhs_reg,
+        )?;
+
+        match op.item {
+            // `and` / `or` are short-circuiting, and we can get by with one register and a branch
+            Operator::Boolean(Boolean::And) => {
+                let true_label = builder.label(None);
+                builder.branch_if(lhs_reg, true_label, op.span)?;
+
+                // If the branch was not taken it's false, so short circuit to load false
+                let false_label = builder.label(None);
+                builder.jump(false_label, op.span)?;
+
+                builder.set_label(true_label, builder.here())?;
+                compile_expression(
+                    working_set,
+                    builder,
+                    rhs,
+                    RedirectModes::capture_out(rhs.span),
+                    None,
+                    lhs_reg,
+                )?;
+
+                let end_label = builder.label(None);
+                builder.jump(end_label, op.span)?;
+
+                // Consumed by `branch-if`, so we have to set it false again
+                builder.set_label(false_label, builder.here())?;
+                builder.load_literal(lhs_reg, Literal::Bool(false).into_spanned(lhs.span))?;
+
+                builder.set_label(end_label, builder.here())?;
+            }
+            Operator::Boolean(Boolean::Or) => {
+                let true_label = builder.label(None);
+                builder.branch_if(lhs_reg, true_label, op.span)?;
+
+                // If the branch was not taken it's false, so do the right-side expression
+                compile_expression(
+                    working_set,
+                    builder,
+                    rhs,
+                    RedirectModes::capture_out(rhs.span),
+                    None,
+                    lhs_reg,
+                )?;
+
+                let end_label = builder.label(None);
+                builder.jump(end_label, op.span)?;
+
+                // Consumed by `branch-if`, so we have to set it true again
+                builder.set_label(true_label, builder.here())?;
+                builder.load_literal(lhs_reg, Literal::Bool(true).into_spanned(lhs.span))?;
+
+                builder.set_label(end_label, builder.here())?;
+            }
+            _ => {
+                // Any other operator, via `binary-op`
+                let rhs_reg = builder.next_register()?;
+
+                compile_expression(
+                    working_set,
+                    builder,
+                    rhs,
+                    RedirectModes::capture_out(rhs.span),
+                    None,
+                    rhs_reg,
+                )?;
+
+                builder.push(
+                    Instruction::BinaryOp {
+                        lhs_dst: lhs_reg,
+                        op: op.item,
+                        rhs: rhs_reg,
+                    }
+                    .into_spanned(op.span),
+                )?;
+            }
+        }
+
+        if lhs_reg != out_reg {
+            builder.push(
+                Instruction::Move {
+                    dst: out_reg,
+                    src: lhs_reg,
+                }
+                .into_spanned(op.span),
+            )?;
+        }
+
+        builder.push(Instruction::Span { src_dst: out_reg }.into_spanned(span))?;
+
+        Ok(())
+    }
+}
+
+/// The equivalent plain operator to use for an assignment, if any
+pub(crate) fn decompose_assignment(assignment: Assignment) -> Option<Operator> {
+    match assignment {
+        Assignment::Assign => None,
+        Assignment::PlusAssign => Some(Operator::Math(Math::Plus)),
+        Assignment::AppendAssign => Some(Operator::Math(Math::Append)),
+        Assignment::MinusAssign => Some(Operator::Math(Math::Minus)),
+        Assignment::MultiplyAssign => Some(Operator::Math(Math::Multiply)),
+        Assignment::DivideAssign => Some(Operator::Math(Math::Divide)),
+    }
+}
+
+/// Compile assignment of the value in a register to a left-hand expression
+pub(crate) fn compile_assignment(
+    working_set: &StateWorkingSet,
+    builder: &mut BlockBuilder,
+    lhs: &Expression,
+    assignment_span: Span,
+    rhs_reg: RegId,
+) -> Result<(), CompileError> {
+    match lhs.expr {
+        Expr::Var(var_id) => {
+            // Double check that the variable is supposed to be mutable
+            if !working_set.get_variable(var_id).mutable {
+                return Err(CompileError::AssignmentRequiresMutableVar { span: lhs.span });
+            }
+
+            builder.push(
+                Instruction::StoreVariable {
+                    var_id,
+                    src: rhs_reg,
+                }
+                .into_spanned(assignment_span),
+            )?;
+            Ok(())
+        }
+        Expr::FullCellPath(ref path) => match (&path.head, &path.tail) {
+            (
+                Expression {
+                    expr: Expr::Var(var_id),
+                    ..
+                },
+                _,
+            ) if *var_id == ENV_VARIABLE_ID => {
+                // This will be an assignment to an environment variable.
+                let Some(PathMember::String { val: key, .. }) = path.tail.first() else {
+                    return Err(CompileError::CannotReplaceEnv { span: lhs.span });
+                };
+
+                // Some env vars can't be set by Nushell code.
+                const AUTOMATIC_NAMES: &[&str] = &["PWD", "FILE_PWD", "CURRENT_FILE"];
+                if AUTOMATIC_NAMES.iter().any(|name| key.eq_ignore_case(name)) {
+                    return Err(CompileError::AutomaticEnvVarSetManually {
+                        envvar_name: "PWD".into(),
+                        span: lhs.span,
+                    });
+                }
+
+                let key_data = builder.data(key)?;
+
+                let val_reg = if path.tail.len() > 1 {
+                    // Get the current value of the head and first tail of the path, from env
+                    let head_reg = builder.next_register()?;
+
+                    // We could use compile_load_env, but this shares the key data...
+                    // Always use optional, because it doesn't matter if it's already there
+                    builder.push(
+                        Instruction::LoadEnvOpt {
+                            dst: head_reg,
+                            key: key_data,
+                        }
+                        .into_spanned(lhs.span),
+                    )?;
+
+                    // Default to empty record so we can do further upserts
+                    let default_label = builder.label(None);
+                    let upsert_label = builder.label(None);
+                    builder.branch_if_empty(head_reg, default_label, assignment_span)?;
+                    builder.jump(upsert_label, assignment_span)?;
+
+                    builder.set_label(default_label, builder.here())?;
+                    builder.load_literal(
+                        head_reg,
+                        Literal::Record { capacity: 0 }.into_spanned(lhs.span),
+                    )?;
+
+                    // Do the upsert on the current value to incorporate rhs
+                    builder.set_label(upsert_label, builder.here())?;
+                    compile_upsert_cell_path(
+                        builder,
+                        (&path.tail[1..]).into_spanned(lhs.span),
+                        head_reg,
+                        rhs_reg,
+                        assignment_span,
+                    )?;
+
+                    head_reg
+                } else {
+                    // Path has only one tail, so we don't need the current value to do an upsert,
+                    // just set it directly to rhs
+                    rhs_reg
+                };
+
+                // Finally, store the modified env variable
+                builder.push(
+                    Instruction::StoreEnv {
+                        key: key_data,
+                        src: val_reg,
+                    }
+                    .into_spanned(assignment_span),
+                )?;
+                Ok(())
+            }
+            (_, tail) if tail.is_empty() => {
+                // If the path tail is empty, we can really just treat this as if it were an
+                // assignment to the head
+                compile_assignment(working_set, builder, &path.head, assignment_span, rhs_reg)
+            }
+            _ => {
+                // Just a normal assignment to some path
+                let head_reg = builder.next_register()?;
+
+                // Compile getting current value of the head expression
+                compile_expression(
+                    working_set,
+                    builder,
+                    &path.head,
+                    RedirectModes::capture_out(path.head.span),
+                    None,
+                    head_reg,
+                )?;
+
+                // Upsert the tail of the path into the old value of the head expression
+                compile_upsert_cell_path(
+                    builder,
+                    path.tail.as_slice().into_spanned(lhs.span),
+                    head_reg,
+                    rhs_reg,
+                    assignment_span,
+                )?;
+
+                // Now compile the assignment of the updated value to the head
+                compile_assignment(working_set, builder, &path.head, assignment_span, head_reg)
+            }
+        },
+        Expr::Garbage => Err(CompileError::Garbage { span: lhs.span }),
+        _ => Err(CompileError::AssignmentRequiresVar { span: lhs.span }),
+    }
+}
+
+/// Compile an upsert-cell-path instruction, with known literal members
+pub(crate) fn compile_upsert_cell_path(
+    builder: &mut BlockBuilder,
+    members: Spanned<&[PathMember]>,
+    src_dst: RegId,
+    new_value: RegId,
+    span: Span,
+) -> Result<(), CompileError> {
+    let path_reg = builder.literal(
+        Literal::CellPath(
+            CellPath {
+                members: members.item.to_vec(),
+            }
+            .into(),
+        )
+        .into_spanned(members.span),
+    )?;
+    builder.push(
+        Instruction::UpsertCellPath {
+            src_dst,
+            path: path_reg,
+            new_value,
+        }
+        .into_spanned(span),
+    )?;
+    Ok(())
+}
+
+/// Compile the correct sequence to get an environment variable + follow a path on it
+pub(crate) fn compile_load_env(
+    builder: &mut BlockBuilder,
+    span: Span,
+    path: &[PathMember],
+    out_reg: RegId,
+) -> Result<(), CompileError> {
+    if path.is_empty() {
+        builder.push(
+            Instruction::LoadVariable {
+                dst: out_reg,
+                var_id: ENV_VARIABLE_ID,
+            }
+            .into_spanned(span),
+        )?;
+    } else {
+        let (key, optional) = match &path[0] {
+            PathMember::String { val, optional, .. } => (builder.data(val)?, *optional),
+            PathMember::Int { span, .. } => {
+                return Err(CompileError::AccessEnvByInt { span: *span })
+            }
+        };
+        let tail = &path[1..];
+
+        if optional {
+            builder.push(Instruction::LoadEnvOpt { dst: out_reg, key }.into_spanned(span))?;
+        } else {
+            builder.push(Instruction::LoadEnv { dst: out_reg, key }.into_spanned(span))?;
+        }
+
+        if !tail.is_empty() {
+            let path = builder.literal(
+                Literal::CellPath(Box::new(CellPath {
+                    members: tail.to_vec(),
+                }))
+                .into_spanned(span),
+            )?;
+            builder.push(
+                Instruction::FollowCellPath {
+                    src_dst: out_reg,
+                    path,
+                }
+                .into_spanned(span),
+            )?;
+        }
+    }
+    Ok(())
+}

crates/nu-engine/src/compile/redirect.rs

@@ -0,0 +1,157 @@
+use nu_protocol::{
+    ast::{Expression, RedirectionTarget},
+    engine::StateWorkingSet,
+    ir::{Instruction, RedirectMode},
+    IntoSpanned, OutDest, RegId, Span, Spanned,
+};
+
+use super::{compile_expression, BlockBuilder, CompileError};
+
+#[derive(Default, Clone)]
+pub(crate) struct RedirectModes {
+    pub(crate) out: Option<Spanned<RedirectMode>>,
+    pub(crate) err: Option<Spanned<RedirectMode>>,
+}
+
+impl RedirectModes {
+    pub(crate) fn capture_out(span: Span) -> Self {
+        RedirectModes {
+            out: Some(RedirectMode::Capture.into_spanned(span)),
+            err: None,
+        }
+    }
+
+    pub(crate) fn caller(span: Span) -> RedirectModes {
+        RedirectModes {
+            out: Some(RedirectMode::Caller.into_spanned(span)),
+            err: Some(RedirectMode::Caller.into_spanned(span)),
+        }
+    }
+}
+
+pub(crate) fn redirection_target_to_mode(
+    working_set: &StateWorkingSet,
+    builder: &mut BlockBuilder,
+    target: &RedirectionTarget,
+) -> Result<Spanned<RedirectMode>, CompileError> {
+    Ok(match target {
+        RedirectionTarget::File {
+            expr,
+            append,
+            span: redir_span,
+        } => {
+            let file_num = builder.next_file_num()?;
+            let path_reg = builder.next_register()?;
+            compile_expression(
+                working_set,
+                builder,
+                expr,
+                RedirectModes::capture_out(*redir_span),
+                None,
+                path_reg,
+            )?;
+            builder.push(
+                Instruction::OpenFile {
+                    file_num,
+                    path: path_reg,
+                    append: *append,
+                }
+                .into_spanned(*redir_span),
+            )?;
+            RedirectMode::File { file_num }.into_spanned(*redir_span)
+        }
+        RedirectionTarget::Pipe { span } => RedirectMode::Pipe.into_spanned(*span),
+    })
+}
+
+pub(crate) fn redirect_modes_of_expression(
+    working_set: &StateWorkingSet,
+    expression: &Expression,
+    redir_span: Span,
+) -> Result<RedirectModes, CompileError> {
+    let (out, err) = expression.expr.pipe_redirection(working_set);
+    Ok(RedirectModes {
+        out: out
+            .map(|r| r.into_spanned(redir_span))
+            .map(out_dest_to_redirect_mode)
+            .transpose()?,
+        err: err
+            .map(|r| r.into_spanned(redir_span))
+            .map(out_dest_to_redirect_mode)
+            .transpose()?,
+    })
+}
+
+/// Finish the redirection for an expression, writing to and closing files as necessary
+pub(crate) fn finish_redirection(
+    builder: &mut BlockBuilder,
+    modes: RedirectModes,
+    out_reg: RegId,
+) -> Result<(), CompileError> {
+    if let Some(Spanned {
+        item: RedirectMode::File { file_num },
+        span,
+    }) = modes.out
+    {
+        // If out is a file and err is a pipe, we must not consume the expression result -
+        // that is actually the err, in that case.
+        if !matches!(
+            modes.err,
+            Some(Spanned {
+                item: RedirectMode::Pipe { .. },
+                ..
+            })
+        ) {
+            builder.push(
+                Instruction::WriteFile {
+                    file_num,
+                    src: out_reg,
+                }
+                .into_spanned(span),
+            )?;
+            builder.load_empty(out_reg)?;
+        }
+        builder.push(Instruction::CloseFile { file_num }.into_spanned(span))?;
+    }
+
+    match modes.err {
+        Some(Spanned {
+            item: RedirectMode::File { file_num },
+            span,
+        }) => {
+            // Close the file, unless it's the same as out (in which case it was already closed)
+            if !modes.out.is_some_and(|out_mode| match out_mode.item {
+                RedirectMode::File {
+                    file_num: out_file_num,
+                } => file_num == out_file_num,
+                _ => false,
+            }) {
+                builder.push(Instruction::CloseFile { file_num }.into_spanned(span))?;
+            }
+        }
+        Some(Spanned {
+            item: RedirectMode::Pipe,
+            span,
+        }) => {
+            builder.push(Instruction::CheckErrRedirected { src: out_reg }.into_spanned(span))?;
+        }
+        _ => (),
+    }
+
+    Ok(())
+}
+
+pub(crate) fn out_dest_to_redirect_mode(
+    out_dest: Spanned<OutDest>,
+) -> Result<Spanned<RedirectMode>, CompileError> {
+    let span = out_dest.span;
+    out_dest
+        .map(|out_dest| match out_dest {
+            OutDest::Pipe => Ok(RedirectMode::Pipe),
+            OutDest::Capture => Ok(RedirectMode::Capture),
+            OutDest::Null => Ok(RedirectMode::Null),
+            OutDest::Inherit => Ok(RedirectMode::Inherit),
+            OutDest::File(_) => Err(CompileError::InvalidRedirectMode { span }),
+        })
+        .transpose()
+}

crates/nu-engine/src/documentation.rs

@@ -45,10 +45,12 @@ fn nu_highlight_string(code_string: &str, engine_state: &EngineState, stack: &mu
     if let Some(highlighter) = engine_state.find_decl(b"nu-highlight", &[]) {
         let decl = engine_state.get_decl(highlighter);
 
+        let call = Call::new(Span::unknown());
+
         if let Ok(output) = decl.run(
             engine_state,
             stack,
-            &Call::new(Span::unknown()),
+            &(&call).into(),
             Value::string(code_string, Span::unknown()).into_pipeline_data(),
         ) {
             let result = output.into_value(Span::unknown());
@@ -269,11 +271,12 @@ fn get_documentation(
             let _ = write!(long_desc, "\n  > {}\n", example.example);
         } else if let Some(highlighter) = engine_state.find_decl(b"nu-highlight", &[]) {
             let decl = engine_state.get_decl(highlighter);
+            let call = Call::new(Span::unknown());
 
             match decl.run(
                 engine_state,
                 stack,
-                &Call::new(Span::unknown()),
+                &(&call).into(),
                 Value::string(example.example, Span::unknown()).into_pipeline_data(),
             ) {
                 Ok(output) => {
@@ -326,7 +329,7 @@ fn get_documentation(
                         .run(
                             engine_state,
                             stack,
-                            &table_call,
+                            &(&table_call).into(),
                             PipelineData::Value(result.clone(), None),
                         )
                         .ok()

crates/nu-engine/src/env.rs

@@ -1,8 +1,8 @@
 use crate::ClosureEvalOnce;
 use nu_path::canonicalize_with;
 use nu_protocol::{
-    ast::{Call, Expr},
-    engine::{EngineState, Stack, StateWorkingSet},
+    ast::Expr,
+    engine::{Call, EngineState, Stack, StateWorkingSet},
     Config, ShellError, Span, Value, VarId,
 };
 use std::{
@@ -244,14 +244,15 @@ pub fn path_str(
 }
 
 pub const DIR_VAR_PARSER_INFO: &str = "dirs_var";
-pub fn get_dirs_var_from_call(call: &Call) -> Option<VarId> {
-    call.get_parser_info(DIR_VAR_PARSER_INFO).and_then(|x| {
-        if let Expr::Var(id) = x.expr {
-            Some(id)
-        } else {
-            None
-        }
-    })
+pub fn get_dirs_var_from_call(stack: &Stack, call: &Call) -> Option<VarId> {
+    call.get_parser_info(stack, DIR_VAR_PARSER_INFO)
+        .and_then(|x| {
+            if let Expr::Var(id) = x.expr {
+                Some(id)
+            } else {
+                None
+            }
+        })
 }
 
 /// This helper function is used to find files during eval

crates/nu-engine/src/eval.rs

@@ -1,3 +1,4 @@
+use crate::eval_ir_block;
 #[allow(deprecated)]
 use crate::{current_dir, get_config, get_full_help};
 use nu_path::{expand_path_with, AbsolutePathBuf};
@@ -7,7 +8,7 @@ use nu_protocol::{
         PipelineRedirection, RedirectionSource, RedirectionTarget,
     },
     debugger::DebugContext,
-    engine::{Closure, EngineState, Redirection, Stack},
+    engine::{Closure, EngineState, Redirection, Stack, StateWorkingSet},
     eval_base::Eval,
     ByteStreamSource, Config, FromValue, IntoPipelineData, OutDest, PipelineData, ShellError, Span,
     Spanned, Type, Value, VarId, ENV_VARIABLE_ID,
@@ -174,7 +175,7 @@ pub fn eval_call<D: DebugContext>(
         // We pass caller_stack here with the knowledge that internal commands
         // are going to be specifically looking for global state in the stack
         // rather than any local state.
-        decl.run(engine_state, caller_stack, call, input)
+        decl.run(engine_state, caller_stack, &call.into(), input)
     }
 }
 
@@ -223,7 +224,7 @@ fn eval_external(
         }
     }
 
-    command.run(engine_state, stack, &call, input)
+    command.run(engine_state, stack, &(&call).into(), input)
 }
 
 pub fn eval_expression<D: DebugContext>(
@@ -507,6 +508,11 @@ pub fn eval_block<D: DebugContext>(
     block: &Block,
     mut input: PipelineData,
 ) -> Result<PipelineData, ShellError> {
+    // Remove once IR is the default.
+    if stack.use_ir {
+        return eval_ir_block::<D>(engine_state, stack, block, input);
+    }
+
     D::enter_block(engine_state, block);
 
     let num_pipelines = block.len();
@@ -521,7 +527,7 @@ pub fn eval_block<D: DebugContext>(
 
         for (i, element) in elements.iter().enumerate() {
             let next = elements.get(i + 1).unwrap_or(last);
-            let (next_out, next_err) = next.pipe_redirection(engine_state);
+            let (next_out, next_err) = next.pipe_redirection(&StateWorkingSet::new(engine_state));
             let (stdout, stderr) = eval_element_redirection::<D>(
                 engine_state,
                 stack,
@@ -903,7 +909,7 @@ impl Eval for EvalRuntime {
 ///
 /// An automatic environment variable cannot be assigned to by user code.
 /// Current there are three of them: $env.PWD, $env.FILE_PWD, $env.CURRENT_FILE
-fn is_automatic_env_var(var: &str) -> bool {
+pub(crate) fn is_automatic_env_var(var: &str) -> bool {
     let names = ["PWD", "FILE_PWD", "CURRENT_FILE"];
     names.iter().any(|&name| {
         if cfg!(windows) {

crates/nu-engine/src/eval_helpers.rs

@@ -1,6 +1,6 @@
 use crate::{
     eval_block, eval_block_with_early_return, eval_expression, eval_expression_with_input,
-    eval_subexpression,
+    eval_ir_block, eval_subexpression,
 };
 use nu_protocol::{
     ast::{Block, Expression},
@@ -13,6 +13,10 @@ use nu_protocol::{
 pub type EvalBlockFn =
     fn(&EngineState, &mut Stack, &Block, PipelineData) -> Result<PipelineData, ShellError>;
 
+/// Type of eval_ir_block() function
+pub type EvalIrBlockFn =
+    fn(&EngineState, &mut Stack, &Block, PipelineData) -> Result<PipelineData, ShellError>;
+
 /// Type of eval_block_with_early_return() function
 pub type EvalBlockWithEarlyReturnFn =
     fn(&EngineState, &mut Stack, &Block, PipelineData) -> Result<PipelineData, ShellError>;
@@ -42,6 +46,16 @@ pub fn get_eval_block(engine_state: &EngineState) -> EvalBlockFn {
     }
 }
 
+/// Helper function to fetch `eval_ir_block()` with the correct type parameter based on whether
+/// engine_state is configured with or without a debugger.
+pub fn get_eval_ir_block(engine_state: &EngineState) -> EvalIrBlockFn {
+    if engine_state.is_debugging() {
+        eval_ir_block::<WithDebug>
+    } else {
+        eval_ir_block::<WithoutDebug>
+    }
+}
+
 /// Helper function to fetch `eval_block_with_early_return()` with the correct type parameter based
 /// on whether engine_state is configured with or without a debugger.
 pub fn get_eval_block_with_early_return(engine_state: &EngineState) -> EvalBlockWithEarlyReturnFn {

crates/nu-engine/src/lib.rs

@@ -2,16 +2,19 @@ mod call_ext;
 mod closure_eval;
 pub mod column;
 pub mod command_prelude;
+mod compile;
 pub mod documentation;
 pub mod env;
 mod eval;
 mod eval_helpers;
+mod eval_ir;
 mod glob_from;
 pub mod scope;
 
 pub use call_ext::CallExt;
 pub use closure_eval::*;
 pub use column::get_columns;
+pub use compile::compile;
 pub use documentation::get_full_help;
 pub use env::*;
 pub use eval::{
@@ -19,4 +22,5 @@ pub use eval::{
     eval_expression_with_input, eval_subexpression, eval_variable, redirect_env,
 };
 pub use eval_helpers::*;
+pub use eval_ir::eval_ir_block;
 pub use glob_from::glob_from;

crates/nu-parser/src/known_external.rs

@@ -1,7 +1,8 @@
 use nu_engine::command_prelude::*;
 use nu_protocol::{
-    ast::{Argument, Expr, Expression},
-    engine::{CommandType, UNKNOWN_SPAN_ID},
+    ast::{self, Expr, Expression},
+    engine::{self, CallImpl, CommandType, UNKNOWN_SPAN_ID},
+    ir::{self, DataSlice},
 };
 
 #[derive(Clone)]
@@ -43,8 +44,6 @@ impl Command for KnownExternal {
 
         let command = engine_state.get_decl(decl_id);
 
-        let mut extern_call = Call::new(head_span);
-
         let extern_name = if let Some(name_bytes) = engine_state.find_decl_name(call.decl_id, &[]) {
             String::from_utf8_lossy(name_bytes)
         } else {
@@ -56,59 +55,166 @@ impl Command for KnownExternal {
         };
 
         let extern_name: Vec<_> = extern_name.split(' ').collect();
-        let call_head_id = engine_state
-            .find_span_id(call.head)
-            .unwrap_or(UNKNOWN_SPAN_ID);
 
-        let arg_extern_name = Expression::new_existing(
-            Expr::String(extern_name[0].to_string()),
+        match &call.inner {
+            CallImpl::AstRef(call) => {
+                let extern_call = ast_call_to_extern_call(engine_state, call, &extern_name)?;
+                command.run(engine_state, stack, &(&extern_call).into(), input)
+            }
+            CallImpl::AstBox(call) => {
+                let extern_call = ast_call_to_extern_call(engine_state, call, &extern_name)?;
+                command.run(engine_state, stack, &(&extern_call).into(), input)
+            }
+            CallImpl::IrRef(call) => {
+                let extern_call = ir_call_to_extern_call(stack, call, &extern_name)?;
+                command.run(engine_state, stack, &(&extern_call).into(), input)
+            }
+            CallImpl::IrBox(call) => {
+                let extern_call = ir_call_to_extern_call(stack, call, &extern_name)?;
+                command.run(engine_state, stack, &(&extern_call).into(), input)
+            }
+        }
+    }
+}
+
+/// Transform the args from an `ast::Call` onto a `run-external` call
+fn ast_call_to_extern_call(
+    engine_state: &EngineState,
+    call: &ast::Call,
+    extern_name: &[&str],
+) -> Result<ast::Call, ShellError> {
+    let head_span = call.head;
+
+    let mut extern_call = ast::Call::new(head_span);
+
+    let call_head_id = engine_state
+        .find_span_id(call.head)
+        .unwrap_or(UNKNOWN_SPAN_ID);
+
+    let arg_extern_name = Expression::new_existing(
+        Expr::String(extern_name[0].to_string()),
+        call.head,
+        call_head_id,
+        Type::String,
+    );
+
+    extern_call.add_positional(arg_extern_name);
+
+    for subcommand in extern_name.iter().skip(1) {
+        extern_call.add_positional(Expression::new_existing(
+            Expr::String(subcommand.to_string()),
             call.head,
             call_head_id,
             Type::String,
-        );
+        ));
+    }
 
-        extern_call.add_positional(arg_extern_name);
-
-        for subcommand in extern_name.into_iter().skip(1) {
-            extern_call.add_positional(Expression::new_existing(
-                Expr::String(subcommand.to_string()),
-                call.head,
-                call_head_id,
-                Type::String,
-            ));
-        }
-
-        for arg in &call.arguments {
-            match arg {
-                Argument::Positional(positional) => extern_call.add_positional(positional.clone()),
-                Argument::Named(named) => {
-                    let named_span_id = engine_state
-                        .find_span_id(named.0.span)
-                        .unwrap_or(UNKNOWN_SPAN_ID);
-                    if let Some(short) = &named.1 {
-                        extern_call.add_positional(Expression::new_existing(
-                            Expr::String(format!("-{}", short.item)),
-                            named.0.span,
-                            named_span_id,
-                            Type::String,
-                        ));
-                    } else {
-                        extern_call.add_positional(Expression::new_existing(
-                            Expr::String(format!("--{}", named.0.item)),
-                            named.0.span,
-                            named_span_id,
-                            Type::String,
-                        ));
-                    }
-                    if let Some(arg) = &named.2 {
-                        extern_call.add_positional(arg.clone());
-                    }
+    for arg in &call.arguments {
+        match arg {
+            ast::Argument::Positional(positional) => extern_call.add_positional(positional.clone()),
+            ast::Argument::Named(named) => {
+                let named_span_id = engine_state
+                    .find_span_id(named.0.span)
+                    .unwrap_or(UNKNOWN_SPAN_ID);
+                if let Some(short) = &named.1 {
+                    extern_call.add_positional(Expression::new_existing(
+                        Expr::String(format!("-{}", short.item)),
+                        named.0.span,
+                        named_span_id,
+                        Type::String,
+                    ));
+                } else {
+                    extern_call.add_positional(Expression::new_existing(
+                        Expr::String(format!("--{}", named.0.item)),
+                        named.0.span,
+                        named_span_id,
+                        Type::String,
+                    ));
                 }
-                Argument::Unknown(unknown) => extern_call.add_unknown(unknown.clone()),
-                Argument::Spread(args) => extern_call.add_spread(args.clone()),
+                if let Some(arg) = &named.2 {
+                    extern_call.add_positional(arg.clone());
+                }
+            }
+            ast::Argument::Unknown(unknown) => extern_call.add_unknown(unknown.clone()),
+            ast::Argument::Spread(args) => extern_call.add_spread(args.clone()),
+        }
+    }
+
+    Ok(extern_call)
+}
+
+/// Transform the args from an `ir::Call` onto a `run-external` call
+fn ir_call_to_extern_call(
+    stack: &mut Stack,
+    call: &ir::Call,
+    extern_name: &[&str],
+) -> Result<ir::Call, ShellError> {
+    let mut extern_call = ir::Call::build(call.decl_id, call.head);
+
+    // Add the command and subcommands
+    for name in extern_name {
+        extern_call.add_positional(stack, call.head, Value::string(*name, call.head));
+    }
+
+    // Add the arguments, reformatting named arguments into string positionals
+    for index in 0..call.args_len {
+        match &call.arguments(stack)[index] {
+            engine::Argument::Flag {
+                data,
+                name,
+                short,
+                span,
+            } => {
+                let name_arg = engine::Argument::Positional {
+                    span: *span,
+                    val: Value::string(known_external_option_name(data, *name, *short), *span),
+                    ast: None,
+                };
+                extern_call.add_argument(stack, name_arg);
+            }
+            engine::Argument::Named {
+                data,
+                name,
+                short,
+                span,
+                val,
+                ..
+            } => {
+                let name_arg = engine::Argument::Positional {
+                    span: *span,
+                    val: Value::string(known_external_option_name(data, *name, *short), *span),
+                    ast: None,
+                };
+                let val_arg = engine::Argument::Positional {
+                    span: *span,
+                    val: val.clone(),
+                    ast: None,
+                };
+                extern_call.add_argument(stack, name_arg);
+                extern_call.add_argument(stack, val_arg);
+            }
+            a @ (engine::Argument::Positional { .. }
+            | engine::Argument::Spread { .. }
+            | engine::Argument::ParserInfo { .. }) => {
+                let argument = a.clone();
+                extern_call.add_argument(stack, argument);
             }
         }
+    }
 
-        command.run(engine_state, stack, &extern_call, input)
+    Ok(extern_call.finish())
+}
+
+fn known_external_option_name(data: &[u8], name: DataSlice, short: DataSlice) -> String {
+    if !data[name].is_empty() {
+        format!(
+            "--{}",
+            std::str::from_utf8(&data[name]).expect("invalid utf-8 in flag name")
+        )
+    } else {
+        format!(
+            "-{}",
+            std::str::from_utf8(&data[short]).expect("invalid utf-8 in flag short name")
+        )
     }
 }

crates/nu-parser/src/parse_patterns.rs

@@ -39,7 +39,7 @@ pub fn parse_pattern(working_set: &mut StateWorkingSet, span: Span) -> MatchPatt
         let value = parse_value(working_set, span, &SyntaxShape::Any);
 
         MatchPattern {
-            pattern: Pattern::Value(value),
+            pattern: Pattern::Value(Box::new(value)),
             guard: None,
             span,
         }

crates/nu-parser/src/parser.rs

@@ -3302,6 +3302,8 @@ pub fn parse_row_condition(working_set: &mut StateWorkingSet, spans: &[Span]) ->
                 default_value: None,
             });
 
+            compile_block(working_set, &mut block);
+
             working_set.add_block(Arc::new(block))
         }
     };
@@ -4445,7 +4447,7 @@ pub fn parse_match_block_expression(working_set: &mut StateWorkingSet, span: Spa
                 &SyntaxShape::MathExpression,
             );
 
-            pattern.guard = Some(guard);
+            pattern.guard = Some(Box::new(guard));
             position += if found { start + 1 } else { start };
             connector = working_set.get_span_contents(output[position].span);
         }
@@ -5298,6 +5300,8 @@ pub fn parse_expression(working_set: &mut StateWorkingSet, spans: &[Span]) -> Ex
             let ty = output.ty.clone();
             block.pipelines = vec![Pipeline::from_vec(vec![output])];
 
+            compile_block(working_set, &mut block);
+
             let block_id = working_set.add_block(Arc::new(block));
 
             let mut env_vars = vec![];
@@ -5853,9 +5857,25 @@ pub fn parse_block(
         working_set.parse_errors.extend_from_slice(&errors);
     }
 
+    // Do not try to compile blocks that are subexpressions, or when we've already had a parse
+    // failure as that definitely will fail to compile
+    if !is_subexpression && working_set.parse_errors.is_empty() {
+        compile_block(working_set, &mut block);
+    }
+
     block
 }
 
+/// Compile an IR block for the `Block`, adding a compile error on failure
+fn compile_block(working_set: &mut StateWorkingSet<'_>, block: &mut Block) {
+    match nu_engine::compile(working_set, block) {
+        Ok(ir_block) => {
+            block.ir_block = Some(ir_block);
+        }
+        Err(err) => working_set.compile_errors.push(err),
+    }
+}
+
 pub fn discover_captures_in_closure(
     working_set: &StateWorkingSet,
     block: &Block,
@@ -6298,12 +6318,14 @@ fn wrap_expr_with_collect(working_set: &mut StateWorkingSet, expr: &Expression)
             default_value: None,
         });
 
-        let block = Block {
+        let mut block = Block {
             pipelines: vec![Pipeline::from_vec(vec![expr.clone()])],
             signature: Box::new(signature),
             ..Default::default()
         };
 
+        compile_block(working_set, &mut block);
+
         let block_id = working_set.add_block(Arc::new(block));
 
         output.push(Argument::Positional(Expression::new(

crates/nu-parser/tests/test_parser.rs

@@ -1,7 +1,7 @@
 use nu_parser::*;
 use nu_protocol::{
-    ast::{Argument, Call, Expr, Expression, ExternalArgument, PathMember, Range},
-    engine::{Command, EngineState, Stack, StateWorkingSet},
+    ast::{Argument, Expr, Expression, ExternalArgument, PathMember, Range},
+    engine::{Call, Command, EngineState, Stack, StateWorkingSet},
     ParseError, PipelineData, ShellError, Signature, Span, SyntaxShape, Type,
 };
 use rstest::rstest;
@@ -1759,10 +1759,7 @@ mod range {
 #[cfg(test)]
 mod input_types {
     use super::*;
-    use nu_protocol::{
-        ast::{Argument, Call},
-        Category, PipelineData, ShellError, Type,
-    };
+    use nu_protocol::{ast::Argument, engine::Call, Category, PipelineData, ShellError, Type};
 
     #[derive(Clone)]
     pub struct LsTest;

crates/nu-plugin-engine/src/context.rs

@@ -1,8 +1,7 @@
 use crate::util::MutableCow;
 use nu_engine::{get_eval_block_with_early_return, get_full_help, ClosureEvalOnce};
 use nu_protocol::{
-    ast::Call,
-    engine::{Closure, EngineState, Redirection, Stack},
+    engine::{Call, Closure, EngineState, Redirection, Stack},
     Config, IntoSpanned, OutDest, PipelineData, PluginIdentity, ShellError, Signals, Span, Spanned,
     Value,
 };
@@ -54,7 +53,7 @@ pub struct PluginExecutionCommandContext<'a> {
     identity: Arc<PluginIdentity>,
     engine_state: Cow<'a, EngineState>,
     stack: MutableCow<'a, Stack>,
-    call: Cow<'a, Call>,
+    call: Call<'a>,
 }
 
 impl<'a> PluginExecutionCommandContext<'a> {
@@ -62,13 +61,13 @@ impl<'a> PluginExecutionCommandContext<'a> {
         identity: Arc<PluginIdentity>,
         engine_state: &'a EngineState,
         stack: &'a mut Stack,
-        call: &'a Call,
+        call: &'a Call<'a>,
     ) -> PluginExecutionCommandContext<'a> {
         PluginExecutionCommandContext {
             identity,
             engine_state: Cow::Borrowed(engine_state),
             stack: MutableCow::Borrowed(stack),
-            call: Cow::Borrowed(call),
+            call: call.clone(),
         }
     }
 }
@@ -217,7 +216,7 @@ impl<'a> PluginExecutionContext for PluginExecutionCommandContext<'a> {
             identity: self.identity.clone(),
             engine_state: Cow::Owned(self.engine_state.clone().into_owned()),
             stack: self.stack.owned(),
-            call: Cow::Owned(self.call.clone().into_owned()),
+            call: self.call.to_owned(),
         })
     }
 }

crates/nu-plugin-protocol/src/evaluated_call.rs

@@ -1,7 +1,7 @@
 use nu_protocol::{
-    ast::{Call, Expression},
-    engine::{EngineState, Stack},
-    FromValue, ShellError, Span, Spanned, Value,
+    ast::{self, Expression},
+    engine::{Call, CallImpl, EngineState, Stack},
+    ir, FromValue, ShellError, Span, Spanned, Value,
 };
 use serde::{Deserialize, Serialize};
 
@@ -33,6 +33,24 @@ impl EvaluatedCall {
         engine_state: &EngineState,
         stack: &mut Stack,
         eval_expression_fn: fn(&EngineState, &mut Stack, &Expression) -> Result<Value, ShellError>,
+    ) -> Result<Self, ShellError> {
+        match &call.inner {
+            CallImpl::AstRef(call) => {
+                Self::try_from_ast_call(call, engine_state, stack, eval_expression_fn)
+            }
+            CallImpl::AstBox(call) => {
+                Self::try_from_ast_call(call, engine_state, stack, eval_expression_fn)
+            }
+            CallImpl::IrRef(call) => Self::try_from_ir_call(call, stack),
+            CallImpl::IrBox(call) => Self::try_from_ir_call(call, stack),
+        }
+    }
+
+    fn try_from_ast_call(
+        call: &ast::Call,
+        engine_state: &EngineState,
+        stack: &mut Stack,
+        eval_expression_fn: fn(&EngineState, &mut Stack, &Expression) -> Result<Value, ShellError>,
     ) -> Result<Self, ShellError> {
         let positional =
             call.rest_iter_flattened(0, |expr| eval_expression_fn(engine_state, stack, expr))?;
@@ -54,6 +72,22 @@ impl EvaluatedCall {
         })
     }
 
+    fn try_from_ir_call(call: &ir::Call, stack: &Stack) -> Result<Self, ShellError> {
+        let positional = call.rest_iter_flattened(stack, 0)?;
+
+        let mut named = Vec::with_capacity(call.named_len(stack));
+        named.extend(
+            call.named_iter(stack)
+                .map(|(name, value)| (name.map(|s| s.to_owned()), value.cloned())),
+        );
+
+        Ok(Self {
+            head: call.head,
+            positional,
+            named,
+        })
+    }
+
     /// Check if a flag (named parameter that does not take a value) is set
     /// Returns Ok(true) if flag is set or passed true value
     /// Returns Ok(false) if flag is not set or passed false value

crates/nu-protocol/Cargo.toml

@@ -33,6 +33,7 @@ serde = { workspace = true, default-features = false }
 thiserror = "1.0"
 typetag = "0.2"
 os_pipe = { workspace = true, features = ["io_safety"] }
+log = { workspace = true }
 
 [target.'cfg(unix)'.dependencies]
 nix = { workspace = true, default-features = false, features = ["signal"] }
@@ -54,4 +55,4 @@ tempfile = { workspace = true }
 os_pipe = { workspace = true }
 
 [package.metadata.docs.rs]
-all-features = true
+all-features = true

crates/nu-protocol/src/alias.rs

@@ -1,6 +1,6 @@
 use crate::{
-    ast::{Call, Expression},
-    engine::{Command, CommandType, EngineState, Stack},
+    ast::Expression,
+    engine::{Call, Command, CommandType, EngineState, Stack},
     PipelineData, ShellError, Signature,
 };
 

crates/nu-protocol/src/ast/block.rs

@@ -1,5 +1,5 @@
 use super::Pipeline;
-use crate::{engine::EngineState, OutDest, Signature, Span, Type, VarId};
+use crate::{engine::StateWorkingSet, ir::IrBlock, OutDest, Signature, Span, Type, VarId};
 use serde::{Deserialize, Serialize};
 
 #[derive(Debug, Clone, Serialize, Deserialize)]
@@ -8,6 +8,8 @@ pub struct Block {
     pub pipelines: Vec<Pipeline>,
     pub captures: Vec<VarId>,
     pub redirect_env: bool,
+    /// The block compiled to IR instructions. Not available for subexpressions.
+    pub ir_block: Option<IrBlock>,
     pub span: Option<Span>, // None option encodes no span to avoid using test_span()
 }
 
@@ -22,10 +24,10 @@ impl Block {
 
     pub fn pipe_redirection(
         &self,
-        engine_state: &EngineState,
+        working_set: &StateWorkingSet,
     ) -> (Option<OutDest>, Option<OutDest>) {
         if let Some(first) = self.pipelines.first() {
-            first.pipe_redirection(engine_state)
+            first.pipe_redirection(working_set)
         } else {
             (None, None)
         }
@@ -45,6 +47,7 @@ impl Block {
             pipelines: vec![],
             captures: vec![],
             redirect_env: false,
+            ir_block: None,
             span: None,
         }
     }
@@ -55,6 +58,7 @@ impl Block {
             pipelines: Vec::with_capacity(capacity),
             captures: vec![],
             redirect_env: false,
+            ir_block: None,
             span: None,
         }
     }
@@ -86,6 +90,7 @@ where
             pipelines: pipelines.collect(),
             captures: vec![],
             redirect_env: false,
+            ir_block: None,
             span: None,
         }
     }

crates/nu-protocol/src/ast/expr.rs

@@ -5,7 +5,9 @@ use super::{
     Call, CellPath, Expression, ExternalArgument, FullCellPath, Keyword, MatchPattern, Operator,
     Range, Table, ValueWithUnit,
 };
-use crate::{ast::ImportPattern, engine::EngineState, BlockId, OutDest, Signature, Span, VarId};
+use crate::{
+    ast::ImportPattern, engine::StateWorkingSet, BlockId, OutDest, Signature, Span, VarId,
+};
 
 #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
 pub enum Expr {
@@ -60,17 +62,17 @@ const _: () = assert!(std::mem::size_of::<Expr>() <= 40);
 impl Expr {
     pub fn pipe_redirection(
         &self,
-        engine_state: &EngineState,
+        working_set: &StateWorkingSet,
     ) -> (Option<OutDest>, Option<OutDest>) {
         // Usages of `$in` will be wrapped by a `collect` call by the parser,
         // so we do not have to worry about that when considering
         // which of the expressions below may consume pipeline output.
         match self {
-            Expr::Call(call) => engine_state.get_decl(call.decl_id).pipe_redirection(),
-            Expr::Subexpression(block_id) | Expr::Block(block_id) => engine_state
+            Expr::Call(call) => working_set.get_decl(call.decl_id).pipe_redirection(),
+            Expr::Subexpression(block_id) | Expr::Block(block_id) => working_set
                 .get_block(*block_id)
-                .pipe_redirection(engine_state),
-            Expr::FullCellPath(cell_path) => cell_path.head.expr.pipe_redirection(engine_state),
+                .pipe_redirection(working_set),
+            Expr::FullCellPath(cell_path) => cell_path.head.expr.pipe_redirection(working_set),
             Expr::Bool(_)
             | Expr::Int(_)
             | Expr::Float(_)

crates/nu-protocol/src/ast/match_pattern.rs

@@ -5,7 +5,7 @@ use serde::{Deserialize, Serialize};
 #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
 pub struct MatchPattern {
     pub pattern: Pattern,
-    pub guard: Option<Expression>,
+    pub guard: Option<Box<Expression>>,
     pub span: Span,
 }
 
@@ -19,7 +19,9 @@ impl MatchPattern {
 pub enum Pattern {
     Record(Vec<(String, MatchPattern)>),
     List(Vec<MatchPattern>),
-    Value(Expression),
+    // TODO: it would be nice if this didn't depend on AST
+    // maybe const evaluation can get us to a Value instead?
+    Value(Box<Expression>),
     Variable(VarId),
     Or(Vec<MatchPattern>),
     Rest(VarId), // the ..$foo pattern

crates/nu-protocol/src/ast/pipeline.rs

@@ -1,8 +1,4 @@
-use crate::{
-    ast::Expression,
-    engine::{EngineState, StateWorkingSet},
-    OutDest, Span,
-};
+use crate::{ast::Expression, engine::StateWorkingSet, OutDest, Span};
 use serde::{Deserialize, Serialize};
 use std::fmt::Display;
 
@@ -120,9 +116,9 @@ impl PipelineElement {
 
     pub fn pipe_redirection(
         &self,
-        engine_state: &EngineState,
+        working_set: &StateWorkingSet,
     ) -> (Option<OutDest>, Option<OutDest>) {
-        self.expr.expr.pipe_redirection(engine_state)
+        self.expr.expr.pipe_redirection(working_set)
     }
 }
 
@@ -166,10 +162,10 @@ impl Pipeline {
 
     pub fn pipe_redirection(
         &self,
-        engine_state: &EngineState,
+        working_set: &StateWorkingSet,
     ) -> (Option<OutDest>, Option<OutDest>) {
         if let Some(first) = self.elements.first() {
-            first.pipe_redirection(engine_state)
+            first.pipe_redirection(working_set)
         } else {
             (None, None)
         }

crates/nu-protocol/src/engine/argument.rs

@@ -0,0 +1,124 @@
+use std::sync::Arc;
+
+use crate::{ast::Expression, ir::DataSlice, Span, Value};
+
+/// Represents a fully evaluated argument to a call.
+#[derive(Debug, Clone)]
+pub enum Argument {
+    /// A positional argument
+    Positional {
+        span: Span,
+        val: Value,
+        ast: Option<Arc<Expression>>,
+    },
+    /// A spread argument, e.g. `...$args`
+    Spread {
+        span: Span,
+        vals: Value,
+        ast: Option<Arc<Expression>>,
+    },
+    /// A named argument with no value, e.g. `--flag`
+    Flag {
+        data: Arc<[u8]>,
+        name: DataSlice,
+        short: DataSlice,
+        span: Span,
+    },
+    /// A named argument with a value, e.g. `--flag value` or `--flag=`
+    Named {
+        data: Arc<[u8]>,
+        name: DataSlice,
+        short: DataSlice,
+        span: Span,
+        val: Value,
+        ast: Option<Arc<Expression>>,
+    },
+    /// Information generated by the parser for use by certain keyword commands
+    ParserInfo {
+        data: Arc<[u8]>,
+        name: DataSlice,
+        // TODO: rather than `Expression`, this would probably be best served by a specific enum
+        // type for this purpose.
+        info: Box<Expression>,
+    },
+}
+
+impl Argument {
+    /// The span encompassing the argument's usage within the call, distinct from the span of the
+    /// actual value of the argument.
+    pub fn span(&self) -> Option<Span> {
+        match self {
+            Argument::Positional { span, .. } => Some(*span),
+            Argument::Spread { span, .. } => Some(*span),
+            Argument::Flag { span, .. } => Some(*span),
+            Argument::Named { span, .. } => Some(*span),
+            // Because `ParserInfo` is generated, its span shouldn't be used
+            Argument::ParserInfo { .. } => None,
+        }
+    }
+
+    /// The original AST [`Expression`] for the argument's value. This is not usually available;
+    /// declarations have to opt-in if they require this.
+    pub fn ast_expression(&self) -> Option<&Arc<Expression>> {
+        match self {
+            Argument::Positional { ast, .. } => ast.as_ref(),
+            Argument::Spread { ast, .. } => ast.as_ref(),
+            Argument::Flag { .. } => None,
+            Argument::Named { ast, .. } => ast.as_ref(),
+            Argument::ParserInfo { .. } => None,
+        }
+    }
+}
+
+/// Stores the argument context for calls in IR evaluation.
+#[derive(Debug, Clone)]
+pub struct ArgumentStack {
+    arguments: Vec<Argument>,
+}
+
+impl ArgumentStack {
+    /// Create a new, empty argument stack.
+    pub const fn new() -> Self {
+        ArgumentStack { arguments: vec![] }
+    }
+
+    /// Returns the index of the end of the argument stack. Call and save this before adding
+    /// arguments.
+    pub fn get_base(&self) -> usize {
+        self.arguments.len()
+    }
+
+    /// Calculates the number of arguments past the given [previously retrieved](.get_base) base
+    /// pointer.
+    pub fn get_len(&self, base: usize) -> usize {
+        self.arguments.len().checked_sub(base).unwrap_or_else(|| {
+            panic!(
+                "base ({}) is beyond the end of the arguments stack ({})",
+                base,
+                self.arguments.len()
+            );
+        })
+    }
+
+    /// Push an argument onto the end of the argument stack.
+    pub fn push(&mut self, argument: Argument) {
+        self.arguments.push(argument);
+    }
+
+    /// Clear all of the arguments after the given base index, to prepare for the next frame.
+    pub fn leave_frame(&mut self, base: usize) {
+        self.arguments.truncate(base);
+    }
+
+    /// Get arguments for the frame based on the given [`base`](`.get_base()`) and
+    /// [`len`](`.get_len()`) parameters.
+    pub fn get_args(&self, base: usize, len: usize) -> &[Argument] {
+        &self.arguments[base..(base + len)]
+    }
+
+    /// Move arguments for the frame based on the given [`base`](`.get_base()`) and
+    /// [`len`](`.get_len()`) parameters.
+    pub fn drain_args(&mut self, base: usize, len: usize) -> impl Iterator<Item = Argument> + '_ {
+        self.arguments.drain(base..(base + len))
+    }
+}

crates/nu-protocol/src/engine/call.rs

@@ -0,0 +1,223 @@
+use crate::{
+    ast::{self, Expression},
+    ir, DeclId, FromValue, ShellError, Span, Value,
+};
+
+use super::{EngineState, Stack, StateWorkingSet};
+
+/// This is a HACK to help [`Command`](super::Command) support both the old AST evaluator and the
+/// new IR evaluator at the same time. It should be removed once we are satisfied with the new
+/// evaluator.
+#[derive(Debug, Clone)]
+pub struct Call<'a> {
+    pub head: Span,
+    pub decl_id: DeclId,
+    pub inner: CallImpl<'a>,
+}
+
+#[derive(Debug, Clone)]
+pub enum CallImpl<'a> {
+    AstRef(&'a ast::Call),
+    AstBox(Box<ast::Call>),
+    IrRef(&'a ir::Call),
+    IrBox(Box<ir::Call>),
+}
+
+impl Call<'_> {
+    /// Returns a new AST call with the given span. This is often used by commands that need an
+    /// empty call to pass to a command. It's not easily possible to add anything to this.
+    pub fn new(span: Span) -> Self {
+        // this is using the boxed variant, which isn't so efficient... but this is only temporary
+        // anyway.
+        Call {
+            head: span,
+            decl_id: 0,
+            inner: CallImpl::AstBox(Box::new(ast::Call::new(span))),
+        }
+    }
+
+    /// Convert the `Call` from any lifetime into `'static`, by cloning the data within onto the
+    /// heap.
+    pub fn to_owned(&self) -> Call<'static> {
+        Call {
+            head: self.head,
+            decl_id: self.decl_id,
+            inner: self.inner.to_owned(),
+        }
+    }
+
+    /// Assert that the call is `ast::Call`, and fail with an error if it isn't.
+    ///
+    /// Provided as a stop-gap for commands that can't work with `ir::Call`, or just haven't been
+    /// implemented yet. Eventually these issues should be resolved and then this can be removed.
+    pub fn assert_ast_call(&self) -> Result<&ast::Call, ShellError> {
+        match &self.inner {
+            CallImpl::AstRef(call) => Ok(call),
+            CallImpl::AstBox(call) => Ok(call),
+            _ => Err(ShellError::NushellFailedSpanned {
+                msg: "Can't be used in IR context".into(),
+                label: "this command is not yet supported by IR evaluation".into(),
+                span: self.head,
+            }),
+        }
+    }
+
+    /// FIXME: implementation asserts `ast::Call` and proxies to that
+    pub fn has_flag_const(
+        &self,
+        working_set: &StateWorkingSet,
+        flag_name: &str,
+    ) -> Result<bool, ShellError> {
+        self.assert_ast_call()?
+            .has_flag_const(working_set, flag_name)
+    }
+
+    /// FIXME: implementation asserts `ast::Call` and proxies to that
+    pub fn get_flag_const<T: FromValue>(
+        &self,
+        working_set: &StateWorkingSet,
+        name: &str,
+    ) -> Result<Option<T>, ShellError> {
+        self.assert_ast_call()?.get_flag_const(working_set, name)
+    }
+
+    /// FIXME: implementation asserts `ast::Call` and proxies to that
+    pub fn req_const<T: FromValue>(
+        &self,
+        working_set: &StateWorkingSet,
+        pos: usize,
+    ) -> Result<T, ShellError> {
+        self.assert_ast_call()?.req_const(working_set, pos)
+    }
+
+    /// FIXME: implementation asserts `ast::Call` and proxies to that
+    pub fn rest_const<T: FromValue>(
+        &self,
+        working_set: &StateWorkingSet,
+        starting_pos: usize,
+    ) -> Result<Vec<T>, ShellError> {
+        self.assert_ast_call()?
+            .rest_const(working_set, starting_pos)
+    }
+
+    /// Returns a span covering the call's arguments.
+    pub fn arguments_span(&self) -> Span {
+        match &self.inner {
+            CallImpl::AstRef(call) => call.arguments_span(),
+            CallImpl::AstBox(call) => call.arguments_span(),
+            CallImpl::IrRef(call) => call.arguments_span(),
+            CallImpl::IrBox(call) => call.arguments_span(),
+        }
+    }
+
+    /// Returns a span covering the whole call.
+    pub fn span(&self) -> Span {
+        match &self.inner {
+            CallImpl::AstRef(call) => call.span(),
+            CallImpl::AstBox(call) => call.span(),
+            CallImpl::IrRef(call) => call.span(),
+            CallImpl::IrBox(call) => call.span(),
+        }
+    }
+
+    /// Get a parser info argument by name.
+    pub fn get_parser_info<'a>(&'a self, stack: &'a Stack, name: &str) -> Option<&'a Expression> {
+        match &self.inner {
+            CallImpl::AstRef(call) => call.get_parser_info(name),
+            CallImpl::AstBox(call) => call.get_parser_info(name),
+            CallImpl::IrRef(call) => call.get_parser_info(stack, name),
+            CallImpl::IrBox(call) => call.get_parser_info(stack, name),
+        }
+    }
+
+    /// Evaluator-agnostic implementation of `rest_iter_flattened()`. Evaluates or gets all of the
+    /// positional and spread arguments, flattens spreads, and then returns one list of values.
+    pub fn rest_iter_flattened(
+        &self,
+        engine_state: &EngineState,
+        stack: &mut Stack,
+        eval_expression: fn(
+            &EngineState,
+            &mut Stack,
+            &ast::Expression,
+        ) -> Result<Value, ShellError>,
+        starting_pos: usize,
+    ) -> Result<Vec<Value>, ShellError> {
+        fn by_ast(
+            call: &ast::Call,
+            engine_state: &EngineState,
+            stack: &mut Stack,
+            eval_expression: fn(
+                &EngineState,
+                &mut Stack,
+                &ast::Expression,
+            ) -> Result<Value, ShellError>,
+            starting_pos: usize,
+        ) -> Result<Vec<Value>, ShellError> {
+            call.rest_iter_flattened(starting_pos, |expr| {
+                eval_expression(engine_state, stack, expr)
+            })
+        }
+
+        fn by_ir(
+            call: &ir::Call,
+            stack: &Stack,
+            starting_pos: usize,
+        ) -> Result<Vec<Value>, ShellError> {
+            call.rest_iter_flattened(stack, starting_pos)
+        }
+
+        match &self.inner {
+            CallImpl::AstRef(call) => {
+                by_ast(call, engine_state, stack, eval_expression, starting_pos)
+            }
+            CallImpl::AstBox(call) => {
+                by_ast(call, engine_state, stack, eval_expression, starting_pos)
+            }
+            CallImpl::IrRef(call) => by_ir(call, stack, starting_pos),
+            CallImpl::IrBox(call) => by_ir(call, stack, starting_pos),
+        }
+    }
+
+    /// Get the original AST expression for a positional argument. Does not usually work for IR
+    /// unless the decl specified `requires_ast_for_arguments()`
+    pub fn positional_nth<'a>(&'a self, stack: &'a Stack, index: usize) -> Option<&'a Expression> {
+        match &self.inner {
+            CallImpl::AstRef(call) => call.positional_nth(index),
+            CallImpl::AstBox(call) => call.positional_nth(index),
+            CallImpl::IrRef(call) => call.positional_ast(stack, index).map(|arc| arc.as_ref()),
+            CallImpl::IrBox(call) => call.positional_ast(stack, index).map(|arc| arc.as_ref()),
+        }
+    }
+}
+
+impl CallImpl<'_> {
+    pub fn to_owned(&self) -> CallImpl<'static> {
+        match self {
+            CallImpl::AstRef(call) => CallImpl::AstBox(Box::new((*call).clone())),
+            CallImpl::AstBox(call) => CallImpl::AstBox(call.clone()),
+            CallImpl::IrRef(call) => CallImpl::IrBox(Box::new((*call).clone())),
+            CallImpl::IrBox(call) => CallImpl::IrBox(call.clone()),
+        }
+    }
+}
+
+impl<'a> From<&'a ast::Call> for Call<'a> {
+    fn from(call: &'a ast::Call) -> Self {
+        Call {
+            head: call.head,
+            decl_id: call.decl_id,
+            inner: CallImpl::AstRef(call),
+        }
+    }
+}
+
+impl<'a> From<&'a ir::Call> for Call<'a> {
+    fn from(call: &'a ir::Call) -> Self {
+        Call {
+            head: call.head,
+            decl_id: call.decl_id,
+            inner: CallImpl::IrRef(call),
+        }
+    }
+}

crates/nu-protocol/src/engine/command.rs

@@ -1,5 +1,5 @@
 use super::{EngineState, Stack, StateWorkingSet};
-use crate::{ast::Call, Alias, BlockId, Example, OutDest, PipelineData, ShellError, Signature};
+use crate::{engine::Call, Alias, BlockId, Example, OutDest, PipelineData, ShellError, Signature};
 use std::fmt::Display;
 
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
@@ -124,6 +124,12 @@ pub trait Command: Send + Sync + CommandClone {
     fn pipe_redirection(&self) -> (Option<OutDest>, Option<OutDest>) {
         (None, None)
     }
+
+    /// Return true if the AST nodes for the arguments are required for IR evaluation. This is
+    /// currently inefficient so is not generally done.
+    fn requires_ast_for_arguments(&self) -> bool {
+        false
+    }
 }
 
 pub trait CommandClone {

crates/nu-protocol/src/engine/error_handler.rs

@@ -0,0 +1,55 @@
+use crate::RegId;
+
+/// Describes an error handler stored during IR evaluation.
+#[derive(Debug, Clone, Copy)]
+pub struct ErrorHandler {
+    /// Instruction index within the block that will handle the error
+    pub handler_index: usize,
+    /// Register to put the error information into, when an error occurs
+    pub error_register: Option<RegId>,
+}
+
+/// Keeps track of error handlers pushed during evaluation of an IR block.
+#[derive(Debug, Clone)]
+pub struct ErrorHandlerStack {
+    handlers: Vec<ErrorHandler>,
+}
+
+impl ErrorHandlerStack {
+    pub const fn new() -> ErrorHandlerStack {
+        ErrorHandlerStack { handlers: vec![] }
+    }
+
+    /// Get the current base of the stack, which establishes a frame.
+    pub fn get_base(&self) -> usize {
+        self.handlers.len()
+    }
+
+    /// Push a new error handler onto the stack.
+    pub fn push(&mut self, handler: ErrorHandler) {
+        self.handlers.push(handler);
+    }
+
+    /// Try to pop an error handler from the stack. Won't go below `base`, to avoid retrieving a
+    /// handler belonging to a parent frame.
+    pub fn pop(&mut self, base: usize) -> Option<ErrorHandler> {
+        if self.handlers.len() > base {
+            self.handlers.pop()
+        } else {
+            None
+        }
+    }
+
+    /// Reset the stack to the state it was in at the beginning of the frame, in preparation to
+    /// return control to the parent frame.
+    pub fn leave_frame(&mut self, base: usize) {
+        if self.handlers.len() >= base {
+            self.handlers.truncate(base);
+        } else {
+            panic!(
+                "ErrorHandlerStack bug: tried to leave frame at {base}, but current base is {}",
+                self.get_base()
+            )
+        }
+    }
+}

crates/nu-protocol/src/engine/mod.rs

@@ -1,8 +1,11 @@
+mod argument;
 mod cached_file;
+mod call;
 mod call_info;
 mod capture_block;
 mod command;
 mod engine_state;
+mod error_handler;
 mod overlay;
 mod pattern_match;
 mod stack;
@@ -14,10 +17,13 @@ mod variable;
 
 pub use cached_file::CachedFile;
 
+pub use argument::*;
+pub use call::*;
 pub use call_info::*;
 pub use capture_block::*;
 pub use command::*;
 pub use engine_state::*;
+pub use error_handler::*;
 pub use overlay::*;
 pub use pattern_match::*;
 pub use stack::*;

crates/nu-protocol/src/engine/stack.rs

@@ -1,7 +1,7 @@
 use crate::{
     engine::{
-        EngineState, Redirection, StackCallArgGuard, StackCaptureGuard, StackIoGuard, StackOutDest,
-        DEFAULT_OVERLAY_NAME,
+        ArgumentStack, EngineState, ErrorHandlerStack, Redirection, StackCallArgGuard,
+        StackCaptureGuard, StackIoGuard, StackOutDest, DEFAULT_OVERLAY_NAME,
     },
     OutDest, ShellError, Span, Value, VarId, ENV_VARIABLE_ID, NU_VARIABLE_ID,
 };
@@ -41,6 +41,12 @@ pub struct Stack {
     pub env_hidden: HashMap<String, HashSet<String>>,
     /// List of active overlays
     pub active_overlays: Vec<String>,
+    /// Argument stack for IR evaluation
+    pub arguments: ArgumentStack,
+    /// Error handler stack for IR evaluation
+    pub error_handlers: ErrorHandlerStack,
+    /// Set true to always use IR mode
+    pub use_ir: bool,
     pub recursion_count: u64,
     pub parent_stack: Option<Arc<Stack>>,
     /// Variables that have been deleted (this is used to hide values from parent stack lookups)
@@ -68,6 +74,9 @@ impl Stack {
             env_vars: Vec::new(),
             env_hidden: HashMap::new(),
             active_overlays: vec![DEFAULT_OVERLAY_NAME.to_string()],
+            arguments: ArgumentStack::new(),
+            error_handlers: ErrorHandlerStack::new(),
+            use_ir: false,
             recursion_count: 0,
             parent_stack: None,
             parent_deletions: vec![],
@@ -85,6 +94,9 @@ impl Stack {
             env_vars: parent.env_vars.clone(),
             env_hidden: parent.env_hidden.clone(),
             active_overlays: parent.active_overlays.clone(),
+            arguments: ArgumentStack::new(),
+            error_handlers: ErrorHandlerStack::new(),
+            use_ir: parent.use_ir,
             recursion_count: parent.recursion_count,
             vars: vec![],
             parent_deletions: vec![],
@@ -254,6 +266,9 @@ impl Stack {
             env_vars,
             env_hidden: self.env_hidden.clone(),
             active_overlays: self.active_overlays.clone(),
+            arguments: ArgumentStack::new(),
+            error_handlers: ErrorHandlerStack::new(),
+            use_ir: self.use_ir,
             recursion_count: self.recursion_count,
             parent_stack: None,
             parent_deletions: vec![],
@@ -284,6 +299,9 @@ impl Stack {
             env_vars,
             env_hidden: self.env_hidden.clone(),
             active_overlays: self.active_overlays.clone(),
+            arguments: ArgumentStack::new(),
+            error_handlers: ErrorHandlerStack::new(),
+            use_ir: self.use_ir,
             recursion_count: self.recursion_count,
             parent_stack: None,
             parent_deletions: vec![],

crates/nu-protocol/src/engine/state_working_set.rs

@@ -4,8 +4,8 @@ use crate::{
         usage::build_usage, CachedFile, Command, CommandType, EngineState, OverlayFrame,
         StateDelta, Variable, VirtualPath, Visibility,
     },
-    BlockId, Category, Config, DeclId, FileId, GetSpan, Module, ModuleId, ParseError, ParseWarning,
-    Span, SpanId, Type, Value, VarId, VirtualPathId,
+    BlockId, Category, CompileError, Config, DeclId, FileId, GetSpan, Module, ModuleId, ParseError,
+    ParseWarning, Span, SpanId, Type, Value, VarId, VirtualPathId,
 };
 use core::panic;
 use std::{
@@ -31,6 +31,7 @@ pub struct StateWorkingSet<'a> {
     pub search_predecls: bool,
     pub parse_errors: Vec<ParseError>,
     pub parse_warnings: Vec<ParseWarning>,
+    pub compile_errors: Vec<CompileError>,
 }
 
 impl<'a> StateWorkingSet<'a> {
@@ -50,6 +51,7 @@ impl<'a> StateWorkingSet<'a> {
             search_predecls: true,
             parse_errors: vec![],
             parse_warnings: vec![],
+            compile_errors: vec![],
         }
     }
 
@@ -260,6 +262,12 @@ impl<'a> StateWorkingSet<'a> {
     }
 
     pub fn add_block(&mut self, block: Arc<Block>) -> BlockId {
+        log::trace!(
+            "block id={} added, has IR = {:?}",
+            self.num_blocks(),
+            block.ir_block.is_some()
+        );
+
         self.delta.blocks.push(block);
 
         self.num_blocks() - 1

crates/nu-protocol/src/errors/cli_error.rs

@@ -107,4 +107,8 @@ impl<'src> miette::Diagnostic for CliError<'src> {
     fn related<'a>(&'a self) -> Option<Box<dyn Iterator<Item = &'a dyn miette::Diagnostic> + 'a>> {
         self.0.related()
     }
+
+    fn diagnostic_source(&self) -> Option<&dyn miette::Diagnostic> {
+        self.0.diagnostic_source()
+    }
 }

crates/nu-protocol/src/errors/compile_error.rs

@@ -0,0 +1,238 @@
+use crate::{RegId, Span};
+use miette::Diagnostic;
+use serde::{Deserialize, Serialize};
+use thiserror::Error;
+
+/// An internal compiler error, generally means a Nushell bug rather than an issue with user error
+/// since parsing and typechecking has already passed.
+#[derive(Debug, Clone, Error, Diagnostic, PartialEq, Serialize, Deserialize)]
+pub enum CompileError {
+    #[error("Register overflow.")]
+    #[diagnostic(code(nu::compile::register_overflow))]
+    RegisterOverflow {
+        #[label("the code being compiled is probably too large")]
+        block_span: Option<Span>,
+    },
+
+    #[error("Register {reg_id} was uninitialized when used, possibly reused.")]
+    #[diagnostic(
+        code(nu::compile::register_uninitialized),
+        help("this is a compiler bug. Please report it at https://github.com/nushell/nushell/issues/new\nfrom: {caller}"),
+    )]
+    RegisterUninitialized { reg_id: RegId, caller: String },
+
+    #[error("Register {reg_id} was uninitialized when used, possibly reused.")]
+    #[diagnostic(
+        code(nu::compile::register_uninitialized),
+        help("this is a compiler bug. Please report it at https://github.com/nushell/nushell/issues/new\nfrom: {caller}"),
+    )]
+    RegisterUninitializedWhilePushingInstruction {
+        reg_id: RegId,
+        caller: String,
+        instruction: String,
+        #[label("while adding this instruction: {instruction}")]
+        span: Span,
+    },
+
+    #[error("Block contains too much string data: maximum 4 GiB exceeded.")]
+    #[diagnostic(
+        code(nu::compile::data_overflow),
+        help("try loading the string data from a file instead")
+    )]
+    DataOverflow {
+        #[label("while compiling this block")]
+        block_span: Option<Span>,
+    },
+
+    #[error("Block contains too many files.")]
+    #[diagnostic(
+        code(nu::compile::register_overflow),
+        help("try using fewer file redirections")
+    )]
+    FileOverflow {
+        #[label("while compiling this block")]
+        block_span: Option<Span>,
+    },
+
+    #[error("Invalid redirect mode: File should not be specified by commands.")]
+    #[diagnostic(
+        code(nu::compile::invalid_redirect_mode),
+        help("this is a command bug. Please report it at https://github.com/nushell/nushell/issues/new")
+    )]
+    InvalidRedirectMode {
+        #[label("while compiling this expression")]
+        span: Span,
+    },
+
+    #[error("Encountered garbage, likely due to parse error.")]
+    #[diagnostic(code(nu::compile::garbage))]
+    Garbage {
+        #[label("garbage found here")]
+        span: Span,
+    },
+
+    #[error("Unsupported operator expression.")]
+    #[diagnostic(code(nu::compile::unsupported_operator_expression))]
+    UnsupportedOperatorExpression {
+        #[label("this expression is in operator position but is not an operator")]
+        span: Span,
+    },
+
+    #[error("Attempted access of $env by integer path.")]
+    #[diagnostic(code(nu::compile::access_env_by_int))]
+    AccessEnvByInt {
+        #[label("$env keys should be strings")]
+        span: Span,
+    },
+
+    #[error("Encountered invalid `{keyword}` keyword call.")]
+    #[diagnostic(code(nu::compile::invalid_keyword_call))]
+    InvalidKeywordCall {
+        keyword: String,
+        #[label("this call is not properly formed")]
+        span: Span,
+    },
+
+    #[error("Attempted to set branch target of non-branch instruction.")]
+    #[diagnostic(
+        code(nu::compile::set_branch_target_of_non_branch_instruction),
+        help("this is a compiler bug. Please report it at https://github.com/nushell/nushell/issues/new"),
+    )]
+    SetBranchTargetOfNonBranchInstruction {
+        instruction: String,
+        #[label("tried to modify: {instruction}")]
+        span: Span,
+    },
+
+    /// You're trying to run an unsupported external command.
+    ///
+    /// ## Resolution
+    ///
+    /// Make sure there's an appropriate `run-external` declaration for this external command.
+    #[error("External calls are not supported.")]
+    #[diagnostic(
+        code(nu::compile::run_external_not_found),
+        help("`run-external` was not found in scope")
+    )]
+    RunExternalNotFound {
+        #[label("can't be run in this context")]
+        span: Span,
+    },
+
+    /// Invalid assignment left-hand side
+    ///
+    /// ## Resolution
+    ///
+    /// Assignment requires that you assign to a variable or variable cell path.
+    #[error("Assignment operations require a variable.")]
+    #[diagnostic(
+        code(nu::compile::assignment_requires_variable),
+        help("try assigning to a variable or a cell path of a variable")
+    )]
+    AssignmentRequiresVar {
+        #[label("needs to be a variable")]
+        span: Span,
+    },
+
+    /// Invalid assignment left-hand side
+    ///
+    /// ## Resolution
+    ///
+    /// Assignment requires that you assign to a mutable variable or cell path.
+    #[error("Assignment to an immutable variable.")]
+    #[diagnostic(
+        code(nu::compile::assignment_requires_mutable_variable),
+        help("declare the variable with `mut`, or shadow it again with `let`")
+    )]
+    AssignmentRequiresMutableVar {
+        #[label("needs to be a mutable variable")]
+        span: Span,
+    },
+
+    /// This environment variable cannot be set manually.
+    ///
+    /// ## Resolution
+    ///
+    /// This environment variable is set automatically by Nushell and cannot not be set manually.
+    #[error("{envvar_name} cannot be set manually.")]
+    #[diagnostic(
+        code(nu::compile::automatic_env_var_set_manually),
+        help(
+            r#"The environment variable '{envvar_name}' is set automatically by Nushell and cannot be set manually."#
+        )
+    )]
+    AutomaticEnvVarSetManually {
+        envvar_name: String,
+        #[label("cannot set '{envvar_name}' manually")]
+        span: Span,
+    },
+
+    /// It is not possible to replace the entire environment at once
+    ///
+    /// ## Resolution
+    ///
+    /// Setting the entire environment is not allowed. Change environment variables individually
+    /// instead.
+    #[error("Cannot replace environment.")]
+    #[diagnostic(
+        code(nu::compile::cannot_replace_env),
+        help("Assigning a value to '$env' is not allowed.")
+    )]
+    CannotReplaceEnv {
+        #[label("setting '$env' not allowed")]
+        span: Span,
+    },
+
+    #[error("Unexpected expression.")]
+    #[diagnostic(code(nu::compile::unexpected_expression))]
+    UnexpectedExpression {
+        expr_name: String,
+        #[label("{expr_name} is not allowed in this context")]
+        span: Span,
+    },
+
+    #[error("Missing required declaration: `{decl_name}`")]
+    #[diagnostic(code(nu::compile::missing_required_declaration))]
+    MissingRequiredDeclaration {
+        decl_name: String,
+        #[label("`{decl_name}` must be in scope to compile this expression")]
+        span: Span,
+    },
+
+    #[error("Invalid literal")]
+    #[diagnostic(code(nu::compile::invalid_literal))]
+    InvalidLiteral {
+        msg: String,
+        #[label("{msg}")]
+        span: Span,
+    },
+
+    #[error("{msg}")]
+    #[diagnostic(code(nu::compile::not_in_a_loop))]
+    NotInALoop {
+        msg: String,
+        #[label("can't be used outside of a loop")]
+        span: Option<Span>,
+    },
+
+    #[error("Incoherent loop state: the loop that ended was not the one we were expecting.")]
+    #[diagnostic(
+        code(nu::compile::incoherent_loop_state),
+        help("this is a compiler bug. Please report it at https://github.com/nushell/nushell/issues/new"),
+    )]
+    IncoherentLoopState {
+        #[label("while compiling this block")]
+        block_span: Option<Span>,
+    },
+
+    #[error("Undefined label `{label_id}`.")]
+    #[diagnostic(
+        code(nu::compile::undefined_label),
+        help("this is a compiler bug. Please report it at https://github.com/nushell/nushell/issues/new"),
+    )]
+    UndefinedLabel {
+        label_id: usize,
+        #[label("label was used while compiling this code")]
+        span: Option<Span>,
+    },
+}

crates/nu-protocol/src/errors/mod.rs

@@ -1,10 +1,12 @@
 pub mod cli_error;
+mod compile_error;
 mod labeled_error;
 mod parse_error;
 mod parse_warning;
 mod shell_error;
 
 pub use cli_error::{format_error, report_error, report_error_new};
+pub use compile_error::CompileError;
 pub use labeled_error::{ErrorLabel, LabeledError};
 pub use parse_error::{DidYouMean, ParseError};
 pub use parse_warning::ParseWarning;

crates/nu-protocol/src/errors/shell_error.rs

@@ -1376,6 +1376,23 @@ On Windows, this would be %USERPROFILE%\AppData\Roaming"#
         help("Set XDG_CONFIG_HOME to an absolute path, or set it to an empty string to ignore it")
     )]
     InvalidXdgConfig { xdg: String, default: String },
+
+    /// An unexpected error occurred during IR evaluation.
+    ///
+    /// ## Resolution
+    ///
+    /// This is most likely a correctness issue with the IR compiler or evaluator. Please file a
+    /// bug with the minimum code needed to reproduce the issue, if possible.
+    #[error("IR evaluation error: {msg}")]
+    #[diagnostic(
+        code(nu::shell::ir_eval_error),
+        help("this is a bug, please report it at https://github.com/nushell/nushell/issues/new along with the code you were running if able")
+    )]
+    IrEvalError {
+        msg: String,
+        #[label = "while running this code"]
+        span: Option<Span>,
+    },
 }
 
 // TODO: Implement as From trait

crates/nu-protocol/src/eval_const.rs

@@ -307,7 +307,7 @@ fn eval_const_call(
         return Err(ShellError::NotAConstHelp { span: call.head });
     }
 
-    decl.run_const(working_set, call, input)
+    decl.run_const(working_set, &call.into(), input)
 }
 
 pub fn eval_const_subexpression(

crates/nu-protocol/src/id.rs

@@ -7,5 +7,19 @@ pub type ModuleId = usize;
 pub type OverlayId = usize;
 pub type FileId = usize;
 pub type VirtualPathId = usize;
+
 #[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
 pub struct SpanId(pub usize); // more robust ID style used in the new parser
+
+/// An ID for an [IR](crate::ir) register. `%n` is a common shorthand for `RegId(n)`.
+///
+/// Note: `%0` is allocated with the block input at the beginning of a compiled block.
+#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
+#[repr(transparent)]
+pub struct RegId(pub u32);
+
+impl std::fmt::Display for RegId {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "%{}", self.0)
+    }
+}

crates/nu-protocol/src/ir/call.rs

@@ -0,0 +1,351 @@
+use std::sync::Arc;
+
+use crate::{
+    ast::Expression,
+    engine::{self, Argument, Stack},
+    DeclId, ShellError, Span, Spanned, Value,
+};
+
+use super::DataSlice;
+
+/// Contains the information for a call being made to a declared command.
+#[derive(Debug, Clone)]
+pub struct Call {
+    /// The declaration ID of the command to be invoked.
+    pub decl_id: DeclId,
+    /// The span encompassing the command name, before the arguments.
+    pub head: Span,
+    /// The span encompassing the command name and all arguments.
+    pub span: Span,
+    /// The base index of the arguments for this call within the
+    /// [argument stack](crate::engine::ArgumentStack).
+    pub args_base: usize,
+    /// The number of [`Argument`]s for the call. Note that this just counts the number of
+    /// `Argument` entries on the stack, and has nothing to do with the actual number of positional
+    /// or spread arguments.
+    pub args_len: usize,
+}
+
+impl Call {
+    /// Build a new call with arguments.
+    pub fn build(decl_id: DeclId, head: Span) -> CallBuilder {
+        CallBuilder {
+            inner: Call {
+                decl_id,
+                head,
+                span: head,
+                args_base: 0,
+                args_len: 0,
+            },
+        }
+    }
+
+    /// Get the arguments for this call from the arguments stack.
+    pub fn arguments<'a>(&self, stack: &'a Stack) -> &'a [Argument] {
+        stack.arguments.get_args(self.args_base, self.args_len)
+    }
+
+    /// The span encompassing the arguments
+    ///
+    /// If there are no arguments the span covers where the first argument would exist
+    ///
+    /// If there are one or more arguments the span encompasses the start of the first argument to
+    /// end of the last argument
+    pub fn arguments_span(&self) -> Span {
+        let past = self.head.past();
+        Span::new(past.start, self.span.end)
+    }
+
+    /// The number of named arguments, with or without values.
+    pub fn named_len(&self, stack: &Stack) -> usize {
+        self.arguments(stack)
+            .iter()
+            .filter(|arg| matches!(arg, Argument::Named { .. } | Argument::Flag { .. }))
+            .count()
+    }
+
+    /// Iterate through named arguments, with or without values.
+    pub fn named_iter<'a>(
+        &'a self,
+        stack: &'a Stack,
+    ) -> impl Iterator<Item = (Spanned<&'a str>, Option<&'a Value>)> + 'a {
+        self.arguments(stack).iter().filter_map(
+            |arg: &Argument| -> Option<(Spanned<&str>, Option<&Value>)> {
+                match arg {
+                    Argument::Flag {
+                        data, name, span, ..
+                    } => Some((
+                        Spanned {
+                            item: std::str::from_utf8(&data[*name]).expect("invalid arg name"),
+                            span: *span,
+                        },
+                        None,
+                    )),
+                    Argument::Named {
+                        data,
+                        name,
+                        span,
+                        val,
+                        ..
+                    } => Some((
+                        Spanned {
+                            item: std::str::from_utf8(&data[*name]).expect("invalid arg name"),
+                            span: *span,
+                        },
+                        Some(val),
+                    )),
+                    _ => None,
+                }
+            },
+        )
+    }
+
+    /// Get a named argument's value by name. Returns [`None`] for named arguments with no value as
+    /// well.
+    pub fn get_named_arg<'a>(&self, stack: &'a Stack, flag_name: &str) -> Option<&'a Value> {
+        // Optimized to avoid str::from_utf8()
+        self.arguments(stack)
+            .iter()
+            .find_map(|arg: &Argument| -> Option<Option<&Value>> {
+                match arg {
+                    Argument::Flag { data, name, .. } if &data[*name] == flag_name.as_bytes() => {
+                        Some(None)
+                    }
+                    Argument::Named {
+                        data, name, val, ..
+                    } if &data[*name] == flag_name.as_bytes() => Some(Some(val)),
+                    _ => None,
+                }
+            })
+            .flatten()
+    }
+
+    /// The number of positional arguments, excluding spread arguments.
+    pub fn positional_len(&self, stack: &Stack) -> usize {
+        self.arguments(stack)
+            .iter()
+            .filter(|arg| matches!(arg, Argument::Positional { .. }))
+            .count()
+    }
+
+    /// Iterate through positional arguments. Does not include spread arguments.
+    pub fn positional_iter<'a>(&self, stack: &'a Stack) -> impl Iterator<Item = &'a Value> {
+        self.arguments(stack).iter().filter_map(|arg| match arg {
+            Argument::Positional { val, .. } => Some(val),
+            _ => None,
+        })
+    }
+
+    /// Get a positional argument by index. Does not include spread arguments.
+    pub fn positional_nth<'a>(&self, stack: &'a Stack, index: usize) -> Option<&'a Value> {
+        self.positional_iter(stack).nth(index)
+    }
+
+    /// Get the AST node for a positional argument by index. Not usually available unless the decl
+    /// required it.
+    pub fn positional_ast<'a>(
+        &self,
+        stack: &'a Stack,
+        index: usize,
+    ) -> Option<&'a Arc<Expression>> {
+        self.arguments(stack)
+            .iter()
+            .filter_map(|arg| match arg {
+                Argument::Positional { ast, .. } => Some(ast),
+                _ => None,
+            })
+            .nth(index)
+            .and_then(|option| option.as_ref())
+    }
+
+    /// Returns every argument to the rest parameter, as well as whether each argument
+    /// is spread or a normal positional argument (true for spread, false for normal)
+    pub fn rest_iter<'a>(
+        &self,
+        stack: &'a Stack,
+        start: usize,
+    ) -> impl Iterator<Item = (&'a Value, bool)> + 'a {
+        self.arguments(stack)
+            .iter()
+            .filter_map(|arg| match arg {
+                Argument::Positional { val, .. } => Some((val, false)),
+                Argument::Spread { vals, .. } => Some((vals, true)),
+                _ => None,
+            })
+            .skip(start)
+    }
+
+    /// Returns all of the positional arguments including and after `start`, with spread arguments
+    /// flattened into a single `Vec`.
+    pub fn rest_iter_flattened(
+        &self,
+        stack: &Stack,
+        start: usize,
+    ) -> Result<Vec<Value>, ShellError> {
+        let mut acc = vec![];
+        for (rest_val, spread) in self.rest_iter(stack, start) {
+            if spread {
+                match rest_val {
+                    Value::List { vals, .. } => acc.extend(vals.iter().cloned()),
+                    Value::Error { error, .. } => return Err(ShellError::clone(error)),
+                    _ => {
+                        return Err(ShellError::CannotSpreadAsList {
+                            span: rest_val.span(),
+                        })
+                    }
+                }
+            } else {
+                acc.push(rest_val.clone());
+            }
+        }
+        Ok(acc)
+    }
+
+    /// Get a parser info argument by name.
+    pub fn get_parser_info<'a>(&self, stack: &'a Stack, name: &str) -> Option<&'a Expression> {
+        self.arguments(stack)
+            .iter()
+            .find_map(|argument| match argument {
+                Argument::ParserInfo {
+                    data,
+                    name: name_slice,
+                    info: expr,
+                } if &data[*name_slice] == name.as_bytes() => Some(expr.as_ref()),
+                _ => None,
+            })
+    }
+
+    /// Returns a span encompassing the entire call.
+    pub fn span(&self) -> Span {
+        self.span
+    }
+
+    /// Resets the [`Stack`] to its state before the call was made.
+    pub fn leave(&self, stack: &mut Stack) {
+        stack.arguments.leave_frame(self.args_base);
+    }
+}
+
+/// Utility struct for building a [`Call`] with arguments on the [`Stack`].
+pub struct CallBuilder {
+    inner: Call,
+}
+
+impl CallBuilder {
+    /// Add an argument to the [`Stack`] and reference it from the [`Call`].
+    pub fn add_argument(&mut self, stack: &mut Stack, argument: Argument) -> &mut Self {
+        if self.inner.args_len == 0 {
+            self.inner.args_base = stack.arguments.get_base();
+        }
+        self.inner.args_len += 1;
+        if let Some(span) = argument.span() {
+            self.inner.span = self.inner.span.append(span);
+        }
+        stack.arguments.push(argument);
+        self
+    }
+
+    /// Add a positional argument to the [`Stack`] and reference it from the [`Call`].
+    pub fn add_positional(&mut self, stack: &mut Stack, span: Span, val: Value) -> &mut Self {
+        self.add_argument(
+            stack,
+            Argument::Positional {
+                span,
+                val,
+                ast: None,
+            },
+        )
+    }
+
+    /// Add a spread argument to the [`Stack`] and reference it from the [`Call`].
+    pub fn add_spread(&mut self, stack: &mut Stack, span: Span, vals: Value) -> &mut Self {
+        self.add_argument(
+            stack,
+            Argument::Spread {
+                span,
+                vals,
+                ast: None,
+            },
+        )
+    }
+
+    /// Add a flag (no-value named) argument to the [`Stack`] and reference it from the [`Call`].
+    pub fn add_flag(
+        &mut self,
+        stack: &mut Stack,
+        name: impl AsRef<str>,
+        short: impl AsRef<str>,
+        span: Span,
+    ) -> &mut Self {
+        let (data, name, short) = data_from_name_and_short(name.as_ref(), short.as_ref());
+        self.add_argument(
+            stack,
+            Argument::Flag {
+                data,
+                name,
+                short,
+                span,
+            },
+        )
+    }
+
+    /// Add a named argument to the [`Stack`] and reference it from the [`Call`].
+    pub fn add_named(
+        &mut self,
+        stack: &mut Stack,
+        name: impl AsRef<str>,
+        short: impl AsRef<str>,
+        span: Span,
+        val: Value,
+    ) -> &mut Self {
+        let (data, name, short) = data_from_name_and_short(name.as_ref(), short.as_ref());
+        self.add_argument(
+            stack,
+            Argument::Named {
+                data,
+                name,
+                short,
+                span,
+                val,
+                ast: None,
+            },
+        )
+    }
+
+    /// Produce the finished [`Call`] from the builder.
+    ///
+    /// The call should be entered / run before any other calls are constructed, because the
+    /// argument stack will be reset when they exit.
+    pub fn finish(&self) -> Call {
+        self.inner.clone()
+    }
+
+    /// Run a closure with the [`Call`] as an [`engine::Call`] reference, and then clean up the
+    /// arguments that were added to the [`Stack`] after.
+    ///
+    /// For convenience. Calls [`Call::leave`] after the closure ends.
+    pub fn with<T>(
+        self,
+        stack: &mut Stack,
+        f: impl FnOnce(&mut Stack, &engine::Call<'_>) -> T,
+    ) -> T {
+        let call = engine::Call::from(&self.inner);
+        let result = f(stack, &call);
+        self.inner.leave(stack);
+        result
+    }
+}
+
+fn data_from_name_and_short(name: &str, short: &str) -> (Arc<[u8]>, DataSlice, DataSlice) {
+    let data: Vec<u8> = name.bytes().chain(short.bytes()).collect();
+    let data: Arc<[u8]> = data.into();
+    let name = DataSlice {
+        start: 0,
+        len: name.len().try_into().expect("flag name too big"),
+    };
+    let short = DataSlice {
+        start: name.start.checked_add(name.len).expect("flag name too big"),
+        len: short.len().try_into().expect("flag short name too big"),
+    };
+    (data, name, short)
+}

crates/nu-protocol/src/ir/display.rs

@@ -0,0 +1,452 @@
+use std::fmt;
+
+use crate::{ast::Pattern, engine::EngineState, DeclId, VarId};
+
+use super::{DataSlice, Instruction, IrBlock, Literal, RedirectMode};
+
+pub struct FmtIrBlock<'a> {
+    pub(super) engine_state: &'a EngineState,
+    pub(super) ir_block: &'a IrBlock,
+}
+
+impl<'a> fmt::Display for FmtIrBlock<'a> {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        let plural = |count| if count == 1 { "" } else { "s" };
+        writeln!(
+            f,
+            "# {} register{}, {} instruction{}, {} byte{} of data",
+            self.ir_block.register_count,
+            plural(self.ir_block.register_count as usize),
+            self.ir_block.instructions.len(),
+            plural(self.ir_block.instructions.len()),
+            self.ir_block.data.len(),
+            plural(self.ir_block.data.len()),
+        )?;
+        if self.ir_block.file_count > 0 {
+            writeln!(
+                f,
+                "# {} file{} used for redirection",
+                self.ir_block.file_count,
+                plural(self.ir_block.file_count as usize)
+            )?;
+        }
+        for (index, instruction) in self.ir_block.instructions.iter().enumerate() {
+            let formatted = format!(
+                "{:-4}: {}",
+                index,
+                FmtInstruction {
+                    engine_state: self.engine_state,
+                    instruction,
+                    data: &self.ir_block.data,
+                }
+            );
+            let comment = &self.ir_block.comments[index];
+            if comment.is_empty() {
+                writeln!(f, "{formatted}")?;
+            } else {
+                writeln!(f, "{formatted:40} # {comment}")?;
+            }
+        }
+        Ok(())
+    }
+}
+
+pub struct FmtInstruction<'a> {
+    pub(super) engine_state: &'a EngineState,
+    pub(super) instruction: &'a Instruction,
+    pub(super) data: &'a [u8],
+}
+
+impl<'a> fmt::Display for FmtInstruction<'a> {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        const WIDTH: usize = 22;
+
+        match self.instruction {
+            Instruction::Unreachable => {
+                write!(f, "{:WIDTH$}", "unreachable")
+            }
+            Instruction::LoadLiteral { dst, lit } => {
+                let lit = FmtLiteral {
+                    literal: lit,
+                    data: self.data,
+                };
+                write!(f, "{:WIDTH$} {dst}, {lit}", "load-literal")
+            }
+            Instruction::LoadValue { dst, val } => {
+                let val = val.to_debug_string();
+                write!(f, "{:WIDTH$} {dst}, {val}", "load-value")
+            }
+            Instruction::Move { dst, src } => {
+                write!(f, "{:WIDTH$} {dst}, {src}", "move")
+            }
+            Instruction::Clone { dst, src } => {
+                write!(f, "{:WIDTH$} {dst}, {src}", "clone")
+            }
+            Instruction::Collect { src_dst } => {
+                write!(f, "{:WIDTH$} {src_dst}", "collect")
+            }
+            Instruction::Span { src_dst } => {
+                write!(f, "{:WIDTH$} {src_dst}", "span")
+            }
+            Instruction::Drop { src } => {
+                write!(f, "{:WIDTH$} {src}", "drop")
+            }
+            Instruction::Drain { src } => {
+                write!(f, "{:WIDTH$} {src}", "drain")
+            }
+            Instruction::LoadVariable { dst, var_id } => {
+                let var = FmtVar::new(self.engine_state, *var_id);
+                write!(f, "{:WIDTH$} {dst}, {var}", "load-variable")
+            }
+            Instruction::StoreVariable { var_id, src } => {
+                let var = FmtVar::new(self.engine_state, *var_id);
+                write!(f, "{:WIDTH$} {var}, {src}", "store-variable")
+            }
+            Instruction::LoadEnv { dst, key } => {
+                let key = FmtData(self.data, *key);
+                write!(f, "{:WIDTH$} {dst}, {key}", "load-env")
+            }
+            Instruction::LoadEnvOpt { dst, key } => {
+                let key = FmtData(self.data, *key);
+                write!(f, "{:WIDTH$} {dst}, {key}", "load-env-opt")
+            }
+            Instruction::StoreEnv { key, src } => {
+                let key = FmtData(self.data, *key);
+                write!(f, "{:WIDTH$} {key}, {src}", "store-env")
+            }
+            Instruction::PushPositional { src } => {
+                write!(f, "{:WIDTH$} {src}", "push-positional")
+            }
+            Instruction::AppendRest { src } => {
+                write!(f, "{:WIDTH$} {src}", "append-rest")
+            }
+            Instruction::PushFlag { name } => {
+                let name = FmtData(self.data, *name);
+                write!(f, "{:WIDTH$} {name}", "push-flag")
+            }
+            Instruction::PushShortFlag { short } => {
+                let short = FmtData(self.data, *short);
+                write!(f, "{:WIDTH$} {short}", "push-short-flag")
+            }
+            Instruction::PushNamed { name, src } => {
+                let name = FmtData(self.data, *name);
+                write!(f, "{:WIDTH$} {name}, {src}", "push-named")
+            }
+            Instruction::PushShortNamed { short, src } => {
+                let short = FmtData(self.data, *short);
+                write!(f, "{:WIDTH$} {short}, {src}", "push-short-named")
+            }
+            Instruction::PushParserInfo { name, info } => {
+                let name = FmtData(self.data, *name);
+                write!(f, "{:WIDTH$} {name}, {info:?}", "push-parser-info")
+            }
+            Instruction::RedirectOut { mode } => {
+                write!(f, "{:WIDTH$} {mode}", "redirect-out")
+            }
+            Instruction::RedirectErr { mode } => {
+                write!(f, "{:WIDTH$} {mode}", "redirect-err")
+            }
+            Instruction::CheckErrRedirected { src } => {
+                write!(f, "{:WIDTH$} {src}", "check-err-redirected")
+            }
+            Instruction::OpenFile {
+                file_num,
+                path,
+                append,
+            } => {
+                write!(
+                    f,
+                    "{:WIDTH$} file({file_num}), {path}, append = {append:?}",
+                    "open-file"
+                )
+            }
+            Instruction::WriteFile { file_num, src } => {
+                write!(f, "{:WIDTH$} file({file_num}), {src}", "write-file")
+            }
+            Instruction::CloseFile { file_num } => {
+                write!(f, "{:WIDTH$} file({file_num})", "close-file")
+            }
+            Instruction::Call { decl_id, src_dst } => {
+                let decl = FmtDecl::new(self.engine_state, *decl_id);
+                write!(f, "{:WIDTH$} {decl}, {src_dst}", "call")
+            }
+            Instruction::StringAppend { src_dst, val } => {
+                write!(f, "{:WIDTH$} {src_dst}, {val}", "string-append")
+            }
+            Instruction::GlobFrom { src_dst, no_expand } => {
+                let no_expand = if *no_expand { "no-expand" } else { "expand" };
+                write!(f, "{:WIDTH$} {src_dst}, {no_expand}", "glob-from",)
+            }
+            Instruction::ListPush { src_dst, item } => {
+                write!(f, "{:WIDTH$} {src_dst}, {item}", "list-push")
+            }
+            Instruction::ListSpread { src_dst, items } => {
+                write!(f, "{:WIDTH$} {src_dst}, {items}", "list-spread")
+            }
+            Instruction::RecordInsert { src_dst, key, val } => {
+                write!(f, "{:WIDTH$} {src_dst}, {key}, {val}", "record-insert")
+            }
+            Instruction::RecordSpread { src_dst, items } => {
+                write!(f, "{:WIDTH$} {src_dst}, {items}", "record-spread")
+            }
+            Instruction::Not { src_dst } => {
+                write!(f, "{:WIDTH$} {src_dst}", "not")
+            }
+            Instruction::BinaryOp { lhs_dst, op, rhs } => {
+                write!(f, "{:WIDTH$} {lhs_dst}, {op:?}, {rhs}", "binary-op")
+            }
+            Instruction::FollowCellPath { src_dst, path } => {
+                write!(f, "{:WIDTH$} {src_dst}, {path}", "follow-cell-path")
+            }
+            Instruction::CloneCellPath { dst, src, path } => {
+                write!(f, "{:WIDTH$} {dst}, {src}, {path}", "clone-cell-path")
+            }
+            Instruction::UpsertCellPath {
+                src_dst,
+                path,
+                new_value,
+            } => {
+                write!(
+                    f,
+                    "{:WIDTH$} {src_dst}, {path}, {new_value}",
+                    "upsert-cell-path"
+                )
+            }
+            Instruction::Jump { index } => {
+                write!(f, "{:WIDTH$} {index}", "jump")
+            }
+            Instruction::BranchIf { cond, index } => {
+                write!(f, "{:WIDTH$} {cond}, {index}", "branch-if")
+            }
+            Instruction::BranchIfEmpty { src, index } => {
+                write!(f, "{:WIDTH$} {src}, {index}", "branch-if-empty")
+            }
+            Instruction::Match {
+                pattern,
+                src,
+                index,
+            } => {
+                let pattern = FmtPattern {
+                    engine_state: self.engine_state,
+                    pattern,
+                };
+                write!(f, "{:WIDTH$} ({pattern}), {src}, {index}", "match")
+            }
+            Instruction::CheckMatchGuard { src } => {
+                write!(f, "{:WIDTH$} {src}", "check-match-guard")
+            }
+            Instruction::Iterate {
+                dst,
+                stream,
+                end_index,
+            } => {
+                write!(f, "{:WIDTH$} {dst}, {stream}, end {end_index}", "iterate")
+            }
+            Instruction::OnError { index } => {
+                write!(f, "{:WIDTH$} {index}", "on-error")
+            }
+            Instruction::OnErrorInto { index, dst } => {
+                write!(f, "{:WIDTH$} {index}, {dst}", "on-error-into")
+            }
+            Instruction::PopErrorHandler => {
+                write!(f, "{:WIDTH$}", "pop-error-handler")
+            }
+            Instruction::CheckExternalFailed { dst, src } => {
+                write!(f, "{:WIDTH$} {dst}, {src}", "check-external-failed")
+            }
+            Instruction::ReturnEarly { src } => {
+                write!(f, "{:WIDTH$} {src}", "return-early")
+            }
+            Instruction::Return { src } => {
+                write!(f, "{:WIDTH$} {src}", "return")
+            }
+        }
+    }
+}
+
+struct FmtDecl<'a>(DeclId, &'a str);
+
+impl<'a> FmtDecl<'a> {
+    fn new(engine_state: &'a EngineState, decl_id: DeclId) -> Self {
+        FmtDecl(decl_id, engine_state.get_decl(decl_id).name())
+    }
+}
+
+impl fmt::Display for FmtDecl<'_> {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        write!(f, "decl {} {:?}", self.0, self.1)
+    }
+}
+
+struct FmtVar<'a>(DeclId, Option<&'a str>);
+
+impl<'a> FmtVar<'a> {
+    fn new(engine_state: &'a EngineState, var_id: VarId) -> Self {
+        // Search for the name of the variable
+        let name: Option<&str> = engine_state
+            .active_overlays(&[])
+            .flat_map(|overlay| overlay.vars.iter())
+            .find(|(_, v)| **v == var_id)
+            .map(|(k, _)| std::str::from_utf8(k).unwrap_or("<utf-8 error>"));
+        FmtVar(var_id, name)
+    }
+}
+
+impl fmt::Display for FmtVar<'_> {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        if let Some(name) = self.1 {
+            write!(f, "var {} {:?}", self.0, name)
+        } else {
+            write!(f, "var {}", self.0)
+        }
+    }
+}
+
+impl fmt::Display for RedirectMode {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        match self {
+            RedirectMode::Pipe => write!(f, "pipe"),
+            RedirectMode::Capture => write!(f, "capture"),
+            RedirectMode::Null => write!(f, "null"),
+            RedirectMode::Inherit => write!(f, "inherit"),
+            RedirectMode::File { file_num } => write!(f, "file({file_num})"),
+            RedirectMode::Caller => write!(f, "caller"),
+        }
+    }
+}
+
+struct FmtData<'a>(&'a [u8], DataSlice);
+
+impl<'a> fmt::Display for FmtData<'a> {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        if let Ok(s) = std::str::from_utf8(&self.0[self.1]) {
+            // Write as string
+            write!(f, "{s:?}")
+        } else {
+            // Write as byte array
+            write!(f, "0x{:x?}", self.0)
+        }
+    }
+}
+
+struct FmtLiteral<'a> {
+    literal: &'a Literal,
+    data: &'a [u8],
+}
+
+impl<'a> fmt::Display for FmtLiteral<'a> {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        match self.literal {
+            Literal::Bool(b) => write!(f, "bool({b:?})"),
+            Literal::Int(i) => write!(f, "int({i:?})"),
+            Literal::Float(fl) => write!(f, "float({fl:?})"),
+            Literal::Filesize(q) => write!(f, "filesize({q}b)"),
+            Literal::Duration(q) => write!(f, "duration({q}ns)"),
+            Literal::Binary(b) => write!(f, "binary({})", FmtData(self.data, *b)),
+            Literal::Block(id) => write!(f, "block({id})"),
+            Literal::Closure(id) => write!(f, "closure({id})"),
+            Literal::RowCondition(id) => write!(f, "row_condition({id})"),
+            Literal::Range {
+                start,
+                step,
+                end,
+                inclusion,
+            } => write!(f, "range({start}, {step}, {end}, {inclusion:?})"),
+            Literal::List { capacity } => write!(f, "list(capacity = {capacity})"),
+            Literal::Record { capacity } => write!(f, "record(capacity = {capacity})"),
+            Literal::Filepath { val, no_expand } => write!(
+                f,
+                "filepath({}, no_expand = {no_expand:?})",
+                FmtData(self.data, *val)
+            ),
+            Literal::Directory { val, no_expand } => write!(
+                f,
+                "directory({}, no_expand = {no_expand:?})",
+                FmtData(self.data, *val)
+            ),
+            Literal::GlobPattern { val, no_expand } => write!(
+                f,
+                "glob-pattern({}, no_expand = {no_expand:?})",
+                FmtData(self.data, *val)
+            ),
+            Literal::String(s) => write!(f, "string({})", FmtData(self.data, *s)),
+            Literal::RawString(rs) => write!(f, "raw-string({})", FmtData(self.data, *rs)),
+            Literal::CellPath(p) => write!(f, "cell-path({p})"),
+            Literal::Date(dt) => write!(f, "date({dt})"),
+            Literal::Nothing => write!(f, "nothing"),
+        }
+    }
+}
+
+struct FmtPattern<'a> {
+    engine_state: &'a EngineState,
+    pattern: &'a Pattern,
+}
+
+impl<'a> fmt::Display for FmtPattern<'a> {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        match self.pattern {
+            Pattern::Record(bindings) => {
+                f.write_str("{")?;
+                for (name, pattern) in bindings {
+                    write!(
+                        f,
+                        "{}: {}",
+                        name,
+                        FmtPattern {
+                            engine_state: self.engine_state,
+                            pattern: &pattern.pattern,
+                        }
+                    )?;
+                }
+                f.write_str("}")
+            }
+            Pattern::List(bindings) => {
+                f.write_str("[")?;
+                for pattern in bindings {
+                    write!(
+                        f,
+                        "{}",
+                        FmtPattern {
+                            engine_state: self.engine_state,
+                            pattern: &pattern.pattern
+                        }
+                    )?;
+                }
+                f.write_str("]")
+            }
+            Pattern::Value(expr) => {
+                let string =
+                    String::from_utf8_lossy(self.engine_state.get_span_contents(expr.span));
+                f.write_str(&string)
+            }
+            Pattern::Variable(var_id) => {
+                let variable = FmtVar::new(self.engine_state, *var_id);
+                write!(f, "{}", variable)
+            }
+            Pattern::Or(patterns) => {
+                for (index, pattern) in patterns.iter().enumerate() {
+                    if index > 0 {
+                        f.write_str(" | ")?;
+                    }
+                    write!(
+                        f,
+                        "{}",
+                        FmtPattern {
+                            engine_state: self.engine_state,
+                            pattern: &pattern.pattern
+                        }
+                    )?;
+                }
+                Ok(())
+            }
+            Pattern::Rest(var_id) => {
+                let variable = FmtVar::new(self.engine_state, *var_id);
+                write!(f, "..{}", variable)
+            }
+            Pattern::IgnoreRest => f.write_str(".."),
+            Pattern::IgnoreValue => f.write_str("_"),
+            Pattern::Garbage => f.write_str("<garbage>"),
+        }
+    }
+}

crates/nu-protocol/src/ir/mod.rs

@@ -0,0 +1,419 @@
+use std::{fmt, sync::Arc};
+
+use crate::{
+    ast::{CellPath, Expression, Operator, Pattern, RangeInclusion},
+    engine::EngineState,
+    BlockId, DeclId, RegId, Span, Value, VarId,
+};
+
+use chrono::{DateTime, FixedOffset};
+use serde::{Deserialize, Serialize};
+
+mod call;
+mod display;
+
+pub use call::*;
+pub use display::{FmtInstruction, FmtIrBlock};
+
+#[derive(Clone, Serialize, Deserialize)]
+pub struct IrBlock {
+    pub instructions: Vec<Instruction>,
+    pub spans: Vec<Span>,
+    #[serde(with = "serde_arc_u8_array")]
+    pub data: Arc<[u8]>,
+    pub ast: Vec<Option<IrAstRef>>,
+    /// Additional information that can be added to help with debugging
+    pub comments: Vec<Box<str>>,
+    pub register_count: u32,
+    pub file_count: u32,
+}
+
+impl fmt::Debug for IrBlock {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        // the ast field is too verbose and doesn't add much
+        f.debug_struct("IrBlock")
+            .field("instructions", &self.instructions)
+            .field("spans", &self.spans)
+            .field("data", &self.data)
+            .field("comments", &self.comments)
+            .field("register_count", &self.register_count)
+            .field("file_count", &self.register_count)
+            .finish_non_exhaustive()
+    }
+}
+
+impl IrBlock {
+    /// Returns a value that can be formatted with [`Display`](std::fmt::Display) to show a detailed
+    /// listing of the instructions contained within this [`IrBlock`].
+    pub fn display<'a>(&'a self, engine_state: &'a EngineState) -> FmtIrBlock<'a> {
+        FmtIrBlock {
+            engine_state,
+            ir_block: self,
+        }
+    }
+}
+
+/// A slice into the `data` array of a block. This is a compact and cache-friendly way to store
+/// string data that a block uses.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
+pub struct DataSlice {
+    pub start: u32,
+    pub len: u32,
+}
+
+impl DataSlice {
+    /// A data slice that contains no data. This slice is always valid.
+    pub const fn empty() -> DataSlice {
+        DataSlice { start: 0, len: 0 }
+    }
+}
+
+impl std::ops::Index<DataSlice> for [u8] {
+    type Output = [u8];
+
+    fn index(&self, index: DataSlice) -> &Self::Output {
+        &self[index.start as usize..(index.start as usize + index.len as usize)]
+    }
+}
+
+/// A possible reference into the abstract syntax tree for an instruction. This is not present for
+/// most instructions and is just added when needed.
+#[derive(Debug, Clone)]
+pub struct IrAstRef(pub Arc<Expression>);
+
+impl Serialize for IrAstRef {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: serde::Serializer,
+    {
+        self.0.as_ref().serialize(serializer)
+    }
+}
+
+impl<'de> Deserialize<'de> for IrAstRef {
+    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+    where
+        D: serde::Deserializer<'de>,
+    {
+        Expression::deserialize(deserializer).map(|expr| IrAstRef(Arc::new(expr)))
+    }
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub enum Instruction {
+    /// Unreachable code path (error)
+    Unreachable,
+    /// Load a literal value into the `dst` register
+    LoadLiteral { dst: RegId, lit: Literal },
+    /// Load a clone of a boxed value into the `dst` register (e.g. from const evaluation)
+    LoadValue { dst: RegId, val: Box<Value> },
+    /// Move a register. Value is taken from `src` (used by this instruction).
+    Move { dst: RegId, src: RegId },
+    /// Copy a register (must be a collected value). Value is still in `src` after this instruction.
+    Clone { dst: RegId, src: RegId },
+    /// Collect a stream in a register to a value
+    Collect { src_dst: RegId },
+    /// Change the span of the contents of a register to the span of this instruction.
+    Span { src_dst: RegId },
+    /// Drop the value/stream in a register, without draining
+    Drop { src: RegId },
+    /// Drain the value/stream in a register and discard (e.g. semicolon).
+    ///
+    /// If passed a stream from an external command, sets $env.LAST_EXIT_CODE to the resulting exit
+    /// code, and invokes any available error handler with Empty, or if not available, returns an
+    /// exit-code-only stream, leaving the block.
+    Drain { src: RegId },
+    /// Load the value of a variable into the `dst` register
+    LoadVariable { dst: RegId, var_id: VarId },
+    /// Store the value of a variable from the `src` register
+    StoreVariable { var_id: VarId, src: RegId },
+    /// Load the value of an environment variable into the `dst` register
+    LoadEnv { dst: RegId, key: DataSlice },
+    /// Load the value of an environment variable into the `dst` register, or `Nothing` if it
+    /// doesn't exist
+    LoadEnvOpt { dst: RegId, key: DataSlice },
+    /// Store the value of an environment variable from the `src` register
+    StoreEnv { key: DataSlice, src: RegId },
+    /// Add a positional arg to the next (internal) call.
+    PushPositional { src: RegId },
+    /// Add a list of args to the next (internal) call (spread/rest).
+    AppendRest { src: RegId },
+    /// Add a named arg with no value to the next (internal) call.
+    PushFlag { name: DataSlice },
+    /// Add a short named arg with no value to the next (internal) call.
+    PushShortFlag { short: DataSlice },
+    /// Add a named arg with a value to the next (internal) call.
+    PushNamed { name: DataSlice, src: RegId },
+    /// Add a short named arg with a value to the next (internal) call.
+    PushShortNamed { short: DataSlice, src: RegId },
+    /// Add parser info to the next (internal) call.
+    PushParserInfo {
+        name: DataSlice,
+        info: Box<Expression>,
+    },
+    /// Set the redirection for stdout for the next call (only).
+    ///
+    /// The register for a file redirection is not consumed.
+    RedirectOut { mode: RedirectMode },
+    /// Set the redirection for stderr for the next call (only).
+    ///
+    /// The register for a file redirection is not consumed.
+    RedirectErr { mode: RedirectMode },
+    /// Throw an error if stderr wasn't redirected in the given stream. `src` is preserved.
+    CheckErrRedirected { src: RegId },
+    /// Open a file for redirection, pushing it onto the file stack.
+    OpenFile {
+        file_num: u32,
+        path: RegId,
+        append: bool,
+    },
+    /// Write data from the register to a file. This is done to finish a file redirection, in case
+    /// an internal command or expression was evaluated rather than an external one.
+    WriteFile { file_num: u32, src: RegId },
+    /// Pop a file used for redirection from the file stack.
+    CloseFile { file_num: u32 },
+    /// Make a call. The input is taken from `src_dst`, and the output is placed in `src_dst`,
+    /// overwriting it. The argument stack is used implicitly and cleared when the call ends.
+    Call { decl_id: DeclId, src_dst: RegId },
+    /// Append a value onto the end of a string. Uses `to_expanded_string(", ", ...)` on the value.
+    /// Used for string interpolation literals. Not the same thing as the `++` operator.
+    StringAppend { src_dst: RegId, val: RegId },
+    /// Convert a string into a glob. Used for glob interpolation and setting glob variables. If the
+    /// value is already a glob, it won't be modified (`no_expand` will have no effect).
+    GlobFrom { src_dst: RegId, no_expand: bool },
+    /// Push a value onto the end of a list. Used to construct list literals.
+    ListPush { src_dst: RegId, item: RegId },
+    /// Spread a value onto the end of a list. Used to construct list literals.
+    ListSpread { src_dst: RegId, items: RegId },
+    /// Insert a key-value pair into a record. Used to construct record literals. Raises an error if
+    /// the key already existed in the record.
+    RecordInsert {
+        src_dst: RegId,
+        key: RegId,
+        val: RegId,
+    },
+    /// Spread a record onto a record. Used to construct record literals. Any existing value for the
+    /// key is overwritten.
+    RecordSpread { src_dst: RegId, items: RegId },
+    /// Negate a boolean.
+    Not { src_dst: RegId },
+    /// Do a binary operation on `lhs_dst` (left) and `rhs` (right) and write the result to
+    /// `lhs_dst`.
+    BinaryOp {
+        lhs_dst: RegId,
+        op: Operator,
+        rhs: RegId,
+    },
+    /// Follow a cell path on the value in `src_dst`, storing the result back to `src_dst`
+    FollowCellPath { src_dst: RegId, path: RegId },
+    /// Clone the value at a cell path in `src`, storing the result to `dst`. The original value
+    /// remains in `src`. Must be a collected value.
+    CloneCellPath { dst: RegId, src: RegId, path: RegId },
+    /// Update/insert a cell path to `new_value` on the value in `src_dst`, storing the modified
+    /// value back to `src_dst`
+    UpsertCellPath {
+        src_dst: RegId,
+        path: RegId,
+        new_value: RegId,
+    },
+    /// Jump to an offset in this block
+    Jump { index: usize },
+    /// Branch to an offset in this block if the value of the `cond` register is a true boolean,
+    /// otherwise continue execution
+    BranchIf { cond: RegId, index: usize },
+    /// Branch to an offset in this block if the value of the `src` register is Empty or Nothing,
+    /// otherwise continue execution. The original value in `src` is preserved.
+    BranchIfEmpty { src: RegId, index: usize },
+    /// Match a pattern on `src`. If the pattern matches, branch to `index` after having set any
+    /// variables captured by the pattern. If the pattern doesn't match, continue execution. The
+    /// original value is preserved in `src` through this instruction.
+    Match {
+        pattern: Box<Pattern>,
+        src: RegId,
+        index: usize,
+    },
+    /// Check that a match guard is a boolean, throwing
+    /// [`MatchGuardNotBool`](crate::ShellError::MatchGuardNotBool) if it isn't. Preserves `src`.
+    CheckMatchGuard { src: RegId },
+    /// Iterate on register `stream`, putting the next value in `dst` if present, or jumping to
+    /// `end_index` if the iterator is finished
+    Iterate {
+        dst: RegId,
+        stream: RegId,
+        end_index: usize,
+    },
+    /// Push an error handler, without capturing the error value
+    OnError { index: usize },
+    /// Push an error handler, capturing the error value into `dst`. If the error handler is not
+    /// called, the register should be freed manually.
+    OnErrorInto { index: usize, dst: RegId },
+    /// Pop an error handler. This is not necessary when control flow is directed to the error
+    /// handler due to an error.
+    PopErrorHandler,
+    /// Check if an external command failed. Boolean value into `dst`. `src` is preserved, but it
+    /// does require waiting for the command to exit.
+    CheckExternalFailed { dst: RegId, src: RegId },
+    /// Return early from the block, raising a `ShellError::Return` instead.
+    ///
+    /// Collecting the value is unavoidable.
+    ReturnEarly { src: RegId },
+    /// Return from the block with the value in the register
+    Return { src: RegId },
+}
+
+impl Instruction {
+    /// Returns a value that can be formatted with [`Display`](std::fmt::Display) to show a detailed
+    /// listing of the instruction.
+    pub fn display<'a>(
+        &'a self,
+        engine_state: &'a EngineState,
+        data: &'a [u8],
+    ) -> FmtInstruction<'a> {
+        FmtInstruction {
+            engine_state,
+            instruction: self,
+            data,
+        }
+    }
+
+    /// Returns the branch target index of the instruction if this is a branching instruction.
+    pub fn branch_target(&self) -> Option<usize> {
+        match self {
+            Instruction::Jump { index } => Some(*index),
+            Instruction::BranchIf { cond: _, index } => Some(*index),
+            Instruction::BranchIfEmpty { src: _, index } => Some(*index),
+            Instruction::Match {
+                pattern: _,
+                src: _,
+                index,
+            } => Some(*index),
+
+            Instruction::Iterate {
+                dst: _,
+                stream: _,
+                end_index,
+            } => Some(*end_index),
+            Instruction::OnError { index } => Some(*index),
+            Instruction::OnErrorInto { index, dst: _ } => Some(*index),
+            _ => None,
+        }
+    }
+
+    /// Sets the branch target of the instruction if this is a branching instruction.
+    ///
+    /// Returns `Err(target_index)` if it isn't a branching instruction.
+    pub fn set_branch_target(&mut self, target_index: usize) -> Result<(), usize> {
+        match self {
+            Instruction::Jump { index } => *index = target_index,
+            Instruction::BranchIf { cond: _, index } => *index = target_index,
+            Instruction::BranchIfEmpty { src: _, index } => *index = target_index,
+            Instruction::Match {
+                pattern: _,
+                src: _,
+                index,
+            } => *index = target_index,
+
+            Instruction::Iterate {
+                dst: _,
+                stream: _,
+                end_index,
+            } => *end_index = target_index,
+            Instruction::OnError { index } => *index = target_index,
+            Instruction::OnErrorInto { index, dst: _ } => *index = target_index,
+            _ => return Err(target_index),
+        }
+        Ok(())
+    }
+}
+
+// This is to document/enforce the size of `Instruction` in bytes.
+// We should try to avoid increasing the size of `Instruction`,
+// and PRs that do so will have to change the number below so that it's noted in review.
+const _: () = assert!(std::mem::size_of::<Instruction>() <= 24);
+
+/// A literal value that can be embedded in an instruction.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub enum Literal {
+    Bool(bool),
+    Int(i64),
+    Float(f64),
+    Filesize(i64),
+    Duration(i64),
+    Binary(DataSlice),
+    Block(BlockId),
+    Closure(BlockId),
+    RowCondition(BlockId),
+    Range {
+        start: RegId,
+        step: RegId,
+        end: RegId,
+        inclusion: RangeInclusion,
+    },
+    List {
+        capacity: usize,
+    },
+    Record {
+        capacity: usize,
+    },
+    Filepath {
+        val: DataSlice,
+        no_expand: bool,
+    },
+    Directory {
+        val: DataSlice,
+        no_expand: bool,
+    },
+    GlobPattern {
+        val: DataSlice,
+        no_expand: bool,
+    },
+    String(DataSlice),
+    RawString(DataSlice),
+    CellPath(Box<CellPath>),
+    Date(Box<DateTime<FixedOffset>>),
+    Nothing,
+}
+
+/// A redirection mode for the next call. See [`OutDest`](crate::OutDest).
+///
+/// This is generated by:
+///
+/// 1. Explicit redirection in a [`PipelineElement`](crate::ast::PipelineElement), or
+/// 2. The [`pipe_redirection()`](crate::engine::Command::pipe_redirection) of the command being
+///    piped into.
+///
+/// Not setting it uses the default, determined by [`Stack`](crate::engine::Stack).
+#[derive(Debug, Clone, Copy, Serialize, Deserialize)]
+pub enum RedirectMode {
+    Pipe,
+    Capture,
+    Null,
+    Inherit,
+    /// Use the given numbered file.
+    File {
+        file_num: u32,
+    },
+    /// Use the redirection mode requested by the caller, for a pre-return call.
+    Caller,
+}
+
+/// Just a hack to allow `Arc<[u8]>` to be serialized and deserialized
+mod serde_arc_u8_array {
+    use serde::{Deserialize, Serialize};
+    use std::sync::Arc;
+
+    pub fn serialize<S>(data: &Arc<[u8]>, ser: S) -> Result<S::Ok, S::Error>
+    where
+        S: serde::Serializer,
+    {
+        data.as_ref().serialize(ser)
+    }
+
+    pub fn deserialize<'de, D>(de: D) -> Result<Arc<[u8]>, D::Error>
+    where
+        D: serde::Deserializer<'de>,
+    {
+        let data: Vec<u8> = Deserialize::deserialize(de)?;
+        Ok(data.into())
+    }
+}

crates/nu-protocol/src/lib.rs

@@ -9,6 +9,7 @@ pub mod eval_base;
 pub mod eval_const;
 mod example;
 mod id;
+pub mod ir;
 mod lev_distance;
 mod module;
 pub mod parser_path;

crates/nu-protocol/src/pipeline/byte_stream.rs

@@ -352,6 +352,12 @@ impl ByteStream {
         self.span
     }
 
+    /// Changes the [`Span`] associated with the [`ByteStream`].
+    pub fn with_span(mut self, span: Span) -> Self {
+        self.span = span;
+        self
+    }
+
     /// Returns the [`ByteStreamType`] associated with the [`ByteStream`].
     pub fn type_(&self) -> ByteStreamType {
         self.type_

crates/nu-protocol/src/pipeline/list_stream.rs

@@ -31,11 +31,22 @@ impl ListStream {
         self.span
     }
 
+    /// Changes the [`Span`] associated with this [`ListStream`].
+    pub fn with_span(mut self, span: Span) -> Self {
+        self.span = span;
+        self
+    }
+
     /// Convert a [`ListStream`] into its inner [`Value`] `Iterator`.
     pub fn into_inner(self) -> ValueIterator {
         self.stream
     }
 
+    /// Take a single value from the inner `Iterator`, modifying the stream.
+    pub fn next_value(&mut self) -> Option<Value> {
+        self.stream.next()
+    }
+
     /// Converts each value in a [`ListStream`] into a string and then joins the strings together
     /// using the given separator.
     pub fn into_string(self, separator: &str, config: &Config) -> String {

crates/nu-protocol/src/pipeline/pipeline_data.rs

@@ -96,6 +96,24 @@ impl PipelineData {
         }
     }
 
+    /// Change the span of the [`PipelineData`].
+    ///
+    /// Returns `Value(Nothing)` with the given span if it was [`PipelineData::Empty`].
+    pub fn with_span(self, span: Span) -> Self {
+        match self {
+            PipelineData::Empty => PipelineData::Value(Value::nothing(span), None),
+            PipelineData::Value(value, metadata) => {
+                PipelineData::Value(value.with_span(span), metadata)
+            }
+            PipelineData::ListStream(stream, metadata) => {
+                PipelineData::ListStream(stream.with_span(span), metadata)
+            }
+            PipelineData::ByteStream(stream, metadata) => {
+                PipelineData::ByteStream(stream.with_span(span), metadata)
+            }
+        }
+    }
+
     /// Get a type that is representative of the `PipelineData`.
     ///
     /// The type returned here makes no effort to collect a stream, so it may be a different type
@@ -129,7 +147,8 @@ impl PipelineData {
     /// without consuming input and without writing anything.
     ///
     /// For the other [`OutDest`]s, the given `PipelineData` will be completely consumed
-    /// and `PipelineData::Empty` will be returned.
+    /// and `PipelineData::Empty` will be returned, unless the data is from an external stream,
+    /// in which case an external stream containing only that exit code will be returned.
     pub fn write_to_out_dests(
         self,
         engine_state: &EngineState,
@@ -137,7 +156,11 @@ impl PipelineData {
     ) -> Result<PipelineData, ShellError> {
         match (self, stack.stdout()) {
             (PipelineData::ByteStream(stream, ..), stdout) => {
-                stream.write_to_out_dests(stdout, stack.stderr())?;
+                if let Some(exit_status) = stream.write_to_out_dests(stdout, stack.stderr())? {
+                    return Ok(PipelineData::new_external_stream_with_only_exit_code(
+                        exit_status.code(),
+                    ));
+                }
             }
             (data, OutDest::Pipe | OutDest::Capture) => return Ok(data),
             (PipelineData::Empty, ..) => {}
@@ -570,7 +593,7 @@ impl PipelineData {
                         self.write_all_and_flush(engine_state, no_newline, to_stderr)
                     } else {
                         let call = Call::new(Span::new(0, 0));
-                        let table = command.run(engine_state, stack, &call, self)?;
+                        let table = command.run(engine_state, stack, &(&call).into(), self)?;
                         table.write_all_and_flush(engine_state, no_newline, to_stderr)
                     }
                 } else {

crates/nu-protocol/src/signature.rs

@@ -1,6 +1,5 @@
 use crate::{
-    ast::Call,
-    engine::{Command, CommandType, EngineState, Stack},
+    engine::{Call, Command, CommandType, EngineState, Stack},
     BlockId, PipelineData, ShellError, SyntaxShape, Type, Value, VarId,
 };
 use serde::{Deserialize, Serialize};

crates/nu-protocol/src/span.rs

@@ -53,6 +53,22 @@ impl<T> Spanned<T> {
     }
 }
 
+impl<T, E> Spanned<Result<T, E>> {
+    /// Move the `Result` to the outside, resulting in a spanned `Ok` or unspanned `Err`.
+    pub fn transpose(self) -> Result<Spanned<T>, E> {
+        match self {
+            Spanned {
+                item: Ok(item),
+                span,
+            } => Ok(Spanned { item, span }),
+            Spanned {
+                item: Err(err),
+                span: _,
+            } => Err(err),
+        }
+    }
+}
+
 /// Helper trait to create [`Spanned`] more ergonomically.
 pub trait IntoSpanned: Sized {
     /// Wrap items together with a span into [`Spanned`].

crates/nu-test-support/src/macros.rs

@@ -247,6 +247,7 @@ pub struct NuOpts {
     pub locale: Option<String>,
     pub envs: Option<Vec<(String, String)>>,
     pub collapse_output: Option<bool>,
+    pub use_ir: Option<bool>,
 }
 
 pub fn nu_run_test(opts: NuOpts, commands: impl AsRef<str>, with_std: bool) -> Outcome {
@@ -296,6 +297,15 @@ pub fn nu_run_test(opts: NuOpts, commands: impl AsRef<str>, with_std: bool) -> O
         .stdout(Stdio::piped())
         .stderr(Stdio::piped());
 
+    // Explicitly set NU_USE_IR
+    if let Some(use_ir) = opts.use_ir {
+        if use_ir {
+            command.env("NU_USE_IR", "1");
+        } else {
+            command.env_remove("NU_USE_IR");
+        }
+    }
+
     // Uncomment to debug the command being run:
     // println!("=== command\n{command:?}\n");
 
@@ -373,6 +383,7 @@ where
     if !executable_path.exists() {
         executable_path = crate::fs::installed_nu_path();
     }
+
     let process = match setup_command(&executable_path, &target_cwd)
         .envs(envs)
         .arg("--commands")

src/run.rs

@@ -26,6 +26,10 @@ pub(crate) fn run_commands(
     let mut stack = Stack::new();
     let start_time = std::time::Instant::now();
 
+    if stack.has_env_var(engine_state, "NU_USE_IR") {
+        stack.use_ir = true;
+    }
+
     // if the --no-config-file(-n) option is NOT passed, load the plugin file,
     // load the default env file or custom (depending on parsed_nu_cli_args.env_file),
     // and maybe a custom config file (depending on parsed_nu_cli_args.config_file)
@@ -109,6 +113,10 @@ pub(crate) fn run_file(
     trace!("run_file");
     let mut stack = Stack::new();
 
+    if stack.has_env_var(engine_state, "NU_USE_IR") {
+        stack.use_ir = true;
+    }
+
     // if the --no-config-file(-n) option is NOT passed, load the plugin file,
     // load the default env file or custom (depending on parsed_nu_cli_args.env_file),
     // and maybe a custom config file (depending on parsed_nu_cli_args.config_file)
@@ -184,6 +192,10 @@ pub(crate) fn run_repl(
     let mut stack = Stack::new();
     let start_time = std::time::Instant::now();
 
+    if stack.has_env_var(engine_state, "NU_USE_IR") {
+        stack.use_ir = true;
+    }
+
     if parsed_nu_cli_args.no_config_file.is_none() {
         setup_config(
             engine_state,

tests/eval/mod.rs

@@ -1,7 +1,8 @@
-use nu_test_support::nu;
+use nu_test_support::{nu, playground::Playground};
+use regex::Regex;
 
 #[test]
-fn source_file_relative_to_file() {
+fn record_with_redefined_key() {
     let actual = nu!("{x: 1, x: 2}");
 
     assert!(actual.err.contains("redefined"));
@@ -16,3 +17,455 @@ fn run_file_parse_error() {
 
     assert!(actual.err.contains("unknown type"));
 }
+
+enum ExpectedOut<'a> {
+    /// Equals a string exactly
+    Eq(&'a str),
+    /// Matches a regex
+    Matches(&'a str),
+    /// Produces an error (match regex)
+    Error(&'a str),
+    /// Drops a file that contains these contents
+    FileEq(&'a str, &'a str),
+}
+use self::ExpectedOut::*;
+
+fn test_eval(source: &str, expected_out: ExpectedOut) {
+    Playground::setup("test_eval_ast", |ast_dirs, _playground| {
+        Playground::setup("test_eval_ir", |ir_dirs, _playground| {
+            let actual_ast = nu!(
+                cwd: ast_dirs.test(),
+                use_ir: false,
+                source,
+            );
+            let actual_ir = nu!(
+                cwd: ir_dirs.test(),
+                use_ir: true,
+                source,
+            );
+
+            match expected_out {
+                Eq(eq) => {
+                    assert_eq!(actual_ast.out, eq);
+                    assert_eq!(actual_ir.out, eq);
+                    assert!(actual_ast.status.success());
+                    assert!(actual_ir.status.success());
+                }
+                Matches(regex) => {
+                    let compiled_regex = Regex::new(regex).expect("regex failed to compile");
+                    assert!(
+                        compiled_regex.is_match(&actual_ast.out),
+                        "AST eval out does not match: {}\n{}",
+                        regex,
+                        actual_ast.out
+                    );
+                    assert!(
+                        compiled_regex.is_match(&actual_ir.out),
+                        "IR eval out does not match: {}\n{}",
+                        regex,
+                        actual_ir.out,
+                    );
+                    assert!(actual_ast.status.success());
+                    assert!(actual_ir.status.success());
+                }
+                Error(regex) => {
+                    let compiled_regex = Regex::new(regex).expect("regex failed to compile");
+                    assert!(
+                        compiled_regex.is_match(&actual_ast.err),
+                        "AST eval err does not match: {}",
+                        regex
+                    );
+                    assert!(
+                        compiled_regex.is_match(&actual_ir.err),
+                        "IR eval err does not match: {}",
+                        regex
+                    );
+                    assert!(!actual_ast.status.success());
+                    assert!(!actual_ir.status.success());
+                }
+                FileEq(path, contents) => {
+                    let ast_contents = std::fs::read_to_string(ast_dirs.test().join(path))
+                        .expect("failed to read AST file");
+                    let ir_contents = std::fs::read_to_string(ir_dirs.test().join(path))
+                        .expect("failed to read IR file");
+                    assert_eq!(ast_contents.trim(), contents);
+                    assert_eq!(ir_contents.trim(), contents);
+                    assert!(actual_ast.status.success());
+                    assert!(actual_ir.status.success());
+                }
+            }
+            assert_eq!(actual_ast.out, actual_ir.out);
+        })
+    });
+}
+
+#[test]
+fn literal_bool() {
+    test_eval("true", Eq("true"))
+}
+
+#[test]
+fn literal_int() {
+    test_eval("1", Eq("1"))
+}
+
+#[test]
+fn literal_float() {
+    test_eval("1.5", Eq("1.5"))
+}
+
+#[test]
+fn literal_filesize() {
+    test_eval("30MiB", Eq("30.0 MiB"))
+}
+
+#[test]
+fn literal_duration() {
+    test_eval("30ms", Eq("30ms"))
+}
+
+#[test]
+fn literal_binary() {
+    test_eval("0x[1f 2f f0]", Matches("Length.*1f.*2f.*f0"))
+}
+
+#[test]
+fn literal_closure() {
+    test_eval("{||}", Matches("<Closure"))
+}
+
+#[test]
+fn literal_range() {
+    test_eval("0..2..10", Matches("10"))
+}
+
+#[test]
+fn literal_list() {
+    test_eval("[foo bar baz]", Matches("foo.*bar.*baz"))
+}
+
+#[test]
+fn literal_record() {
+    test_eval("{foo: bar, baz: quux}", Matches("foo.*bar.*baz.*quux"))
+}
+
+#[test]
+fn literal_table() {
+    test_eval("[[a b]; [1 2] [3 4]]", Matches("a.*b.*1.*2.*3.*4"))
+}
+
+#[test]
+fn literal_string() {
+    test_eval(r#""foobar""#, Eq("foobar"))
+}
+
+#[test]
+fn literal_raw_string() {
+    test_eval(r#"r#'bazquux'#"#, Eq("bazquux"))
+}
+
+#[test]
+fn literal_date() {
+    test_eval("2020-01-01T00:00:00Z", Matches("2020"))
+}
+
+#[test]
+fn literal_nothing() {
+    test_eval("null", Eq(""))
+}
+
+#[test]
+fn list_spread() {
+    test_eval("[foo bar ...[baz quux]] | length", Eq("4"))
+}
+
+#[test]
+fn record_spread() {
+    test_eval("{foo: bar ...{baz: quux}} | columns | length", Eq("2"))
+}
+
+#[test]
+fn binary_op_example() {
+    test_eval(
+        "(([1 2] ++ [3 4]) == [1 2 3 4]) and (([1 2 3] ++ 4) == ([1] ++ [2 3 4]))",
+        Eq("true"),
+    )
+}
+
+#[test]
+fn range_from_expressions() {
+    test_eval("(1 + 1)..(2 + 2)", Matches("2.*3.*4"))
+}
+
+#[test]
+fn list_from_expressions() {
+    test_eval(
+        "[('foo' | str upcase) ('BAR' | str downcase)]",
+        Matches("FOO.*bar"),
+    )
+}
+
+#[test]
+fn record_from_expressions() {
+    test_eval("{('foo' | str upcase): 42}", Matches("FOO.*42"))
+}
+
+#[test]
+fn call_spread() {
+    test_eval(
+        "echo foo bar ...[baz quux nushell]",
+        Matches("foo.*bar.*baz.*quux.*nushell"),
+    )
+}
+
+#[test]
+fn call_flag() {
+    test_eval("print -e message", Eq("")) // should not be visible on stdout
+}
+
+#[test]
+fn call_named() {
+    test_eval("10.123 | into string --decimals 1", Eq("10.1"))
+}
+
+#[test]
+fn external_call() {
+    test_eval("nu --testbin cococo foo=bar baz", Eq("foo=bar baz"))
+}
+
+#[test]
+fn external_call_redirect_pipe() {
+    test_eval(
+        "nu --testbin cococo foo=bar baz | str upcase",
+        Eq("FOO=BAR BAZ"),
+    )
+}
+
+#[test]
+fn external_call_redirect_capture() {
+    test_eval(
+        "echo (nu --testbin cococo foo=bar baz) | str upcase",
+        Eq("FOO=BAR BAZ"),
+    )
+}
+
+#[test]
+fn external_call_redirect_file() {
+    test_eval(
+        "nu --testbin cococo hello out> hello.txt",
+        FileEq("hello.txt", "hello"),
+    )
+}
+
+#[test]
+fn let_variable() {
+    test_eval("let foo = 'test'; print $foo", Eq("test"))
+}
+
+#[test]
+fn let_variable_mutate_error() {
+    test_eval(
+        "let foo = 'test'; $foo = 'bar'; print $foo",
+        Error("immutable"),
+    )
+}
+
+#[test]
+fn constant() {
+    test_eval("const foo = 1 + 2; print $foo", Eq("3"))
+}
+
+#[test]
+fn constant_assign_error() {
+    test_eval(
+        "const foo = 1 + 2; $foo = 4; print $foo",
+        Error("immutable"),
+    )
+}
+
+#[test]
+fn mut_variable() {
+    test_eval("mut foo = 'test'; $foo = 'bar'; print $foo", Eq("bar"))
+}
+
+#[test]
+fn mut_variable_append_assign() {
+    test_eval(
+        "mut foo = 'test'; $foo ++= 'bar'; print $foo",
+        Eq("testbar"),
+    )
+}
+
+#[test]
+fn bind_in_variable_to_input() {
+    test_eval("3 | (4 + $in)", Eq("7"))
+}
+
+#[test]
+fn if_true() {
+    test_eval("if true { 'foo' }", Eq("foo"))
+}
+
+#[test]
+fn if_false() {
+    test_eval("if false { 'foo' } | describe", Eq("nothing"))
+}
+
+#[test]
+fn if_else_true() {
+    test_eval("if 5 > 3 { 'foo' } else { 'bar' }", Eq("foo"))
+}
+
+#[test]
+fn if_else_false() {
+    test_eval("if 5 < 3 { 'foo' } else { 'bar' }", Eq("bar"))
+}
+
+#[test]
+fn match_empty_fallthrough() {
+    test_eval("match 42 { }; 'pass'", Eq("pass"))
+}
+
+#[test]
+fn match_value() {
+    test_eval("match 1 { 1 => 'pass', 2 => 'fail' }", Eq("pass"))
+}
+
+#[test]
+fn match_value_default() {
+    test_eval(
+        "match 3 { 1 => 'fail1', 2 => 'fail2', _ => 'pass' }",
+        Eq("pass"),
+    )
+}
+
+#[test]
+fn match_value_fallthrough() {
+    test_eval("match 3 { 1 => 'fail1', 2 => 'fail2' }", Eq(""))
+}
+
+#[test]
+fn match_variable() {
+    test_eval(
+        "match 'pass' { $s => { print $s }, _ => { print 'fail' } }",
+        Eq("pass"),
+    )
+}
+
+#[test]
+fn match_variable_in_list() {
+    test_eval("match [fail pass] { [$f, $p] => { print $p } }", Eq("pass"))
+}
+
+#[test]
+fn match_passthrough_input() {
+    test_eval(
+        "'yes' | match [pass fail] { [$p, ..] => (collect { |y| $y ++ $p }) }",
+        Eq("yespass"),
+    )
+}
+
+#[test]
+fn while_mutate_var() {
+    test_eval("mut x = 2; while $x > 0 { print $x; $x -= 1 }", Eq("21"))
+}
+
+#[test]
+fn for_list() {
+    test_eval("for v in [1 2 3] { print ($v * 2) }", Eq(r"246"))
+}
+
+#[test]
+fn for_seq() {
+    test_eval("for v in (seq 1 4) { print ($v * 2) }", Eq("2468"))
+}
+
+#[test]
+fn early_return() {
+    test_eval("do { return 'foo'; 'bar' }", Eq("foo"))
+}
+
+#[test]
+fn early_return_from_if() {
+    test_eval("do { if true { return 'pass' }; 'fail' }", Eq("pass"))
+}
+
+#[test]
+fn early_return_from_loop() {
+    test_eval("do { loop { return 'pass' } }", Eq("pass"))
+}
+
+#[test]
+fn early_return_from_while() {
+    test_eval(
+        "do { let x = true; while $x { return 'pass' } }",
+        Eq("pass"),
+    )
+}
+
+#[test]
+fn early_return_from_for() {
+    test_eval("do { for x in [pass fail] { return $x } }", Eq("pass"))
+}
+
+#[test]
+fn try_no_catch() {
+    test_eval("try { error make { msg: foo } }; 'pass'", Eq("pass"))
+}
+
+#[test]
+fn try_catch_no_var() {
+    test_eval(
+        "try { error make { msg: foo } } catch { 'pass' }",
+        Eq("pass"),
+    )
+}
+
+#[test]
+fn try_catch_var() {
+    test_eval(
+        "try { error make { msg: foo } } catch { |err| $err.msg }",
+        Eq("foo"),
+    )
+}
+
+#[test]
+fn try_catch_with_non_literal_closure_no_var() {
+    test_eval(
+        r#"
+            let error_handler = { || "pass" }
+            try { error make { msg: foobar } } catch $error_handler
+        "#,
+        Eq("pass"),
+    )
+}
+
+#[test]
+fn try_catch_with_non_literal_closure() {
+    test_eval(
+        r#"
+            let error_handler = { |err| $err.msg }
+            try { error make { msg: foobar } } catch $error_handler
+        "#,
+        Eq("foobar"),
+    )
+}
+
+#[test]
+fn row_condition() {
+    test_eval(
+        "[[a b]; [1 2] [3 4]] | where a < 3 | to nuon",
+        Eq("[[a, b]; [1, 2]]"),
+    )
+}
+
+#[test]
+fn custom_command() {
+    test_eval(
+        r#"
+            def cmd [a: int, b: string = 'fail', ...c: string, --x: int] { $"($a)($b)($c)($x)" }
+            cmd 42 pass foo --x 30
+        "#,
+        Eq("42pass[foo]30"),
+    )
+}