Extract core stuff into own crates
This commit extracts five new crates:
- nu-source, which contains the core source-code handling logic in Nu,
including Text, Span, and also the pretty.rs-based debug logic
- nu-parser, which is the parser and expander logic
- nu-protocol, which is the bulk of the types and basic conveniences
used by plugins
- nu-errors, which contains ShellError, ParseError and error handling
conveniences
- nu-textview, which is the textview plugin extracted into a crate
One of the major consequences of this refactor is that it's no longer
possible to `impl X for Spanned<Y>` outside of the `nu-source` crate, so
a lot of types became more concrete (Value became a concrete type
instead of Spanned<Value>, for example).
This also turned a number of inherent methods in the main nu crate into
plain functions (impl Value {} became a bunch of functions in the
`value` namespace in `crate::data::value`).
2019-11-26 02:30:48 +00:00
|
|
|
[package]
|
2022-03-22 20:25:38 +00:00
|
|
|
authors = ["The Nushell Project Developers"]
|
2022-04-11 18:17:06 +00:00
|
|
|
description = "Nushell's internal protocols, including its abstract syntax tree"
|
2022-08-14 12:21:20 +00:00
|
|
|
repository = "https://github.com/nushell/nushell/tree/main/crates/nu-protocol"
|
2022-03-22 20:25:38 +00:00
|
|
|
edition = "2021"
|
|
|
|
|
license = "MIT"
|
2020-07-05 20:12:44 +00:00
|
|
|
name = "nu-protocol"
|
2024-06-26 01:26:07 +00:00
|
|
|
version = "0.95.1"
|
2021-09-02 01:29:43 +00:00
|
|
|
|
|
|
|
|
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
|
|
|
|
|
|
2023-02-12 22:22:00 +00:00
|
|
|
[lib]
|
|
|
|
|
bench = false
|
|
|
|
|
|
2021-09-02 01:29:43 +00:00
|
|
|
[dependencies]
|
2024-06-26 01:26:07 +00:00
|
|
|
nu-utils = { path = "../nu-utils", version = "0.95.1" }
|
|
|
|
|
nu-path = { path = "../nu-path", version = "0.95.1" }
|
|
|
|
|
nu-system = { path = "../nu-system", version = "0.95.1" }
|
|
|
|
|
nu-derive-value = { path = "../nu-derive-value", version = "0.95.1" }
|
2022-08-04 19:51:02 +00:00
|
|
|
|
2024-04-21 12:36:26 +00:00
|
|
|
brotli = { workspace = true, optional = true }
|
2024-02-06 22:20:09 +00:00
|
|
|
byte-unit = { version = "5.1", features = [ "serde" ] }
|
2024-03-07 22:40:31 +00:00
|
|
|
chrono = { workspace = true, features = [ "serde", "std", "unstable-locales" ], default-features = false }
|
2024-03-23 23:46:02 +00:00
|
|
|
chrono-humanize = { workspace = true }
|
Add derive macros for `FromValue` and `IntoValue` to ease the use of `Value`s in Rust code (#13031)
# Description
After discussing with @sholderbach the cumbersome usage of
`nu_protocol::Value` in Rust, I created a derive macro to simplify it.
I’ve added a new crate called `nu-derive-value`, which includes two
macros, `IntoValue` and `FromValue`. These are re-exported in
`nu-protocol` and should be encouraged to be used via that re-export.
The macros ensure that all types can easily convert from and into
`Value`. For example, as a plugin author, you can define your plugin
configuration using a Rust struct and easily convert it using
`FromValue`. This makes plugin configuration less of a hassle.
I introduced the `IntoValue` trait for a standardized approach to
converting values into `Value` (and a fallible variant `TryIntoValue`).
This trait could potentially replace existing `into_value` methods.
Along with this, I've implemented `FromValue` for several standard types
and refined other implementations to use blanket implementations where
applicable.
I made these design choices with input from @devyn.
There are more improvements possible, but this is a solid start and the
PR is already quite substantial.
# User-Facing Changes
For `nu-protocol` users, these changes simplify the handling of
`Value`s. There are no changes for end-users of nushell itself.
# Tests + Formatting
Documenting the macros itself is not really possible, as they cannot
really reference any other types since they are the root of the
dependency graph. The standard library has the same problem
([std::Debug](https://doc.rust-lang.org/stable/std/fmt/derive.Debug.html)).
However I documented the `FromValue` and `IntoValue` traits completely.
For testing, I made of use `proc-macro2` in the derive macro code. This
would allow testing the generated source code. Instead I just tested
that the derived functionality is correct. This is done in
`nu_protocol::value::test_derive`, as a consumer of `nu-derive-value`
needs to do the testing of the macro usage. I think that these tests
should provide a stable baseline so that users can be sure that the impl
works.
# After Submitting
With these macros available, we can probably use them in some examples
for plugins to showcase the use of them.
2024-06-17 23:05:11 +00:00
|
|
|
convert_case = { workspace = true }
|
2024-03-07 22:40:31 +00:00
|
|
|
fancy-regex = { workspace = true }
|
2024-03-23 23:46:02 +00:00
|
|
|
indexmap = { workspace = true }
|
|
|
|
|
lru = { workspace = true }
|
2024-03-07 22:40:31 +00:00
|
|
|
miette = { workspace = true, features = ["fancy-no-backtrace"] }
|
2024-03-23 23:46:02 +00:00
|
|
|
num-format = { workspace = true }
|
2024-04-21 12:36:26 +00:00
|
|
|
rmp-serde = { workspace = true, optional = true }
|
2024-03-23 23:46:02 +00:00
|
|
|
serde = { workspace = true, default-features = false }
|
2023-05-26 15:32:48 +00:00
|
|
|
thiserror = "1.0"
|
|
|
|
|
typetag = "0.2"
|
Replace `ExternalStream` with new `ByteStream` type (#12774)
# Description
This PR introduces a `ByteStream` type which is a `Read`-able stream of
bytes. Internally, it has an enum over three different byte stream
sources:
```rust
pub enum ByteStreamSource {
Read(Box<dyn Read + Send + 'static>),
File(File),
Child(ChildProcess),
}
```
This is in comparison to the current `RawStream` type, which is an
`Iterator<Item = Vec<u8>>` and has to allocate for each read chunk.
Currently, `PipelineData::ExternalStream` serves a weird dual role where
it is either external command output or a wrapper around `RawStream`.
`ByteStream` makes this distinction more clear (via `ByteStreamSource`)
and replaces `PipelineData::ExternalStream` in this PR:
```rust
pub enum PipelineData {
Empty,
Value(Value, Option<PipelineMetadata>),
ListStream(ListStream, Option<PipelineMetadata>),
ByteStream(ByteStream, Option<PipelineMetadata>),
}
```
The PR is relatively large, but a decent amount of it is just repetitive
changes.
This PR fixes #7017, fixes #10763, and fixes #12369.
This PR also improves performance when piping external commands. Nushell
should, in most cases, have competitive pipeline throughput compared to,
e.g., bash.
| Command | Before (MB/s) | After (MB/s) | Bash (MB/s) |
| -------------------------------------------------- | -------------:|
------------:| -----------:|
| `throughput \| rg 'x'` | 3059 | 3744 | 3739 |
| `throughput \| nu --testbin relay o> /dev/null` | 3508 | 8087 | 8136 |
# User-Facing Changes
- This is a breaking change for the plugin communication protocol,
because the `ExternalStreamInfo` was replaced with `ByteStreamInfo`.
Plugins now only have to deal with a single input stream, as opposed to
the previous three streams: stdout, stderr, and exit code.
- The output of `describe` has been changed for external/byte streams.
- Temporary breaking change: `bytes starts-with` no longer works with
byte streams. This is to keep the PR smaller, and `bytes ends-with`
already does not work on byte streams.
- If a process core dumped, then instead of having a `Value::Error` in
the `exit_code` column of the output returned from `complete`, it now is
a `Value::Int` with the negation of the signal number.
# After Submitting
- Update docs and book as necessary
- Release notes (e.g., plugin protocol changes)
- Adapt/convert commands to work with byte streams (high priority is
`str length`, `bytes starts-with`, and maybe `bytes ends-with`).
- Refactor the `tee` code, Devyn has already done some work on this.
---------
Co-authored-by: Devyn Cairns <devyn.cairns@gmail.com>
2024-05-16 14:11:18 +00:00
|
|
|
os_pipe = { workspace = true, features = ["io_safety"] }
|
Internal representation (IR) compiler and evaluator (#13330)
# Description
This PR adds an internal representation language to Nushell, offering an
alternative evaluator based on simple instructions, stream-containing
registers, and indexed control flow. The number of registers required is
determined statically at compile-time, and the fixed size required is
allocated upon entering the block.
Each instruction is associated with a span, which makes going backwards
from IR instructions to source code very easy.
Motivations for IR:
1. **Performance.** By simplifying the evaluation path and making it
more cache-friendly and branch predictor-friendly, code that does a lot
of computation in Nushell itself can be sped up a decent bit. Because
the IR is fairly easy to reason about, we can also implement
optimization passes in the future to eliminate and simplify code.
2. **Correctness.** The instructions mostly have very simple and
easily-specified behavior, so hopefully engine changes are a little bit
easier to reason about, and they can be specified in a more formal way
at some point. I have made an effort to document each of the
instructions in the docs for the enum itself in a reasonably specific
way. Some of the errors that would have happened during evaluation
before are now moved to the compilation step instead, because they don't
make sense to check during evaluation.
3. **As an intermediate target.** This is a good step for us to bring
the [`new-nu-parser`](https://github.com/nushell/new-nu-parser) in at
some point, as code generated from new AST can be directly compared to
code generated from old AST. If the IR code is functionally equivalent,
it will behave the exact same way.
4. **Debugging.** With a little bit more work, we can probably give
control over advancing the virtual machine that `IrBlock`s run on to
some sort of external driver, making things like breakpoints and single
stepping possible. Tools like `view ir` and [`explore
ir`](https://github.com/devyn/nu_plugin_explore_ir) make it easier than
before to see what exactly is going on with your Nushell code.
The goal is to eventually replace the AST evaluator entirely, once we're
sure it's working just as well. You can help dogfood this by running
Nushell with `$env.NU_USE_IR` set to some value. The environment
variable is checked when Nushell starts, so config runs with IR, or it
can also be set on a line at the REPL to change it dynamically. It is
also checked when running `do` in case within a script you want to just
run a specific piece of code with or without IR.
# Example
```nushell
view ir { |data|
mut sum = 0
for n in $data {
$sum += $n
}
$sum
}
```
```gas
# 3 registers, 19 instructions, 0 bytes of data
0: load-literal %0, int(0)
1: store-variable var 904, %0 # let
2: drain %0
3: drop %0
4: load-variable %1, var 903
5: iterate %0, %1, end 15 # for, label(1), from(14:)
6: store-variable var 905, %0
7: load-variable %0, var 904
8: load-variable %2, var 905
9: binary-op %0, Math(Plus), %2
10: span %0
11: store-variable var 904, %0
12: load-literal %0, nothing
13: drain %0
14: jump 5
15: drop %0 # label(0), from(5:)
16: drain %0
17: load-variable %0, var 904
18: return %0
```
# Benchmarks
All benchmarks run on a base model Mac Mini M1.
## Iterative Fibonacci sequence
This is about as best case as possible, making use of the much faster
control flow. Most code will not experience a speed improvement nearly
this large.
```nushell
def fib [n: int] {
mut a = 0
mut b = 1
for _ in 2..=$n {
let c = $a + $b
$a = $b
$b = $c
}
$b
}
use std bench
bench { 0..50 | each { |n| fib $n } }
```
IR disabled:
```
╭───────┬─────────────────╮
│ mean │ 1ms 924µs 665ns │
│ min │ 1ms 700µs 83ns │
│ max │ 3ms 450µs 125ns │
│ std │ 395µs 759ns │
│ times │ [list 50 items] │
╰───────┴─────────────────╯
```
IR enabled:
```
╭───────┬─────────────────╮
│ mean │ 452µs 820ns │
│ min │ 427µs 417ns │
│ max │ 540µs 167ns │
│ std │ 17µs 158ns │
│ times │ [list 50 items] │
╰───────┴─────────────────╯
```
##
[gradient_benchmark_no_check.nu](https://github.com/nushell/nu_scripts/blob/main/benchmarks/gradient_benchmark_no_check.nu)
IR disabled:
```
╭───┬──────────────────╮
│ 0 │ 27ms 929µs 958ns │
│ 1 │ 21ms 153µs 459ns │
│ 2 │ 18ms 639µs 666ns │
│ 3 │ 19ms 554µs 583ns │
│ 4 │ 13ms 383µs 375ns │
│ 5 │ 11ms 328µs 208ns │
│ 6 │ 5ms 659µs 542ns │
╰───┴──────────────────╯
```
IR enabled:
```
╭───┬──────────────────╮
│ 0 │ 22ms 662µs │
│ 1 │ 17ms 221µs 792ns │
│ 2 │ 14ms 786µs 708ns │
│ 3 │ 13ms 876µs 834ns │
│ 4 │ 13ms 52µs 875ns │
│ 5 │ 11ms 269µs 666ns │
│ 6 │ 6ms 942µs 500ns │
╰───┴──────────────────╯
```
##
[random-bytes.nu](https://github.com/nushell/nu_scripts/blob/main/benchmarks/random-bytes.nu)
I got pretty random results out of this benchmark so I decided not to
include it. Not clear why.
# User-Facing Changes
- IR compilation errors may appear even if the user isn't evaluating
with IR.
- IR evaluation can be enabled by setting the `NU_USE_IR` environment
variable to any value.
- New command `view ir` pretty-prints the IR for a block, and `view ir
--json` can be piped into an external tool like [`explore
ir`](https://github.com/devyn/nu_plugin_explore_ir).
# Tests + Formatting
All tests are passing with `NU_USE_IR=1`, and I've added some more eval
tests to compare the results for some very core operations. I will
probably want to add some more so we don't have to always check
`NU_USE_IR=1 toolkit test --workspace` on a regular basis.
# After Submitting
- [ ] release notes
- [ ] further documentation of instructions?
- [ ] post-release: publish `nu_plugin_explore_ir`
2024-07-11 00:33:59 +00:00
|
|
|
log = { workspace = true }
|
Replace `ExternalStream` with new `ByteStream` type (#12774)
# Description
This PR introduces a `ByteStream` type which is a `Read`-able stream of
bytes. Internally, it has an enum over three different byte stream
sources:
```rust
pub enum ByteStreamSource {
Read(Box<dyn Read + Send + 'static>),
File(File),
Child(ChildProcess),
}
```
This is in comparison to the current `RawStream` type, which is an
`Iterator<Item = Vec<u8>>` and has to allocate for each read chunk.
Currently, `PipelineData::ExternalStream` serves a weird dual role where
it is either external command output or a wrapper around `RawStream`.
`ByteStream` makes this distinction more clear (via `ByteStreamSource`)
and replaces `PipelineData::ExternalStream` in this PR:
```rust
pub enum PipelineData {
Empty,
Value(Value, Option<PipelineMetadata>),
ListStream(ListStream, Option<PipelineMetadata>),
ByteStream(ByteStream, Option<PipelineMetadata>),
}
```
The PR is relatively large, but a decent amount of it is just repetitive
changes.
This PR fixes #7017, fixes #10763, and fixes #12369.
This PR also improves performance when piping external commands. Nushell
should, in most cases, have competitive pipeline throughput compared to,
e.g., bash.
| Command | Before (MB/s) | After (MB/s) | Bash (MB/s) |
| -------------------------------------------------- | -------------:|
------------:| -----------:|
| `throughput \| rg 'x'` | 3059 | 3744 | 3739 |
| `throughput \| nu --testbin relay o> /dev/null` | 3508 | 8087 | 8136 |
# User-Facing Changes
- This is a breaking change for the plugin communication protocol,
because the `ExternalStreamInfo` was replaced with `ByteStreamInfo`.
Plugins now only have to deal with a single input stream, as opposed to
the previous three streams: stdout, stderr, and exit code.
- The output of `describe` has been changed for external/byte streams.
- Temporary breaking change: `bytes starts-with` no longer works with
byte streams. This is to keep the PR smaller, and `bytes ends-with`
already does not work on byte streams.
- If a process core dumped, then instead of having a `Value::Error` in
the `exit_code` column of the output returned from `complete`, it now is
a `Value::Int` with the negation of the signal number.
# After Submitting
- Update docs and book as necessary
- Release notes (e.g., plugin protocol changes)
- Adapt/convert commands to work with byte streams (high priority is
`str length`, `bytes starts-with`, and maybe `bytes ends-with`).
- Refactor the `tee` code, Devyn has already done some work on this.
---------
Co-authored-by: Devyn Cairns <devyn.cairns@gmail.com>
2024-05-16 14:11:18 +00:00
|
|
|
|
|
|
|
|
[target.'cfg(unix)'.dependencies]
|
|
|
|
|
nix = { workspace = true, default-features = false, features = ["signal"] }
|
2021-11-19 02:51:42 +00:00
|
|
|
|
|
|
|
|
[features]
|
2024-04-21 12:36:26 +00:00
|
|
|
plugin = [
|
|
|
|
|
"brotli",
|
|
|
|
|
"rmp-serde",
|
|
|
|
|
]
|
2021-11-19 02:51:42 +00:00
|
|
|
|
|
|
|
|
[dev-dependencies]
|
2024-03-07 22:40:31 +00:00
|
|
|
serde_json = { workspace = true }
|
2024-03-10 19:31:54 +00:00
|
|
|
strum = "0.26"
|
Bump strum_macros from 0.25.3 to 0.26.1 (#11979)
Bumps [strum_macros](https://github.com/Peternator7/strum) from 0.25.3
to 0.26.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/Peternator7/strum/releases">strum_macros's
releases</a>.</em></p>
<blockquote>
<h2>v0.26.1</h2>
<h2>0.26.1</h2>
<ul>
<li><a
href="https://redirect.github.com/Peternator7/strum/pull/325">#325</a>:
use <code>core</code> instead of <code>std</code> in VariantArray.</li>
</ul>
<h2>0.26.0</h2>
<h3>Breaking Changes</h3>
<ul>
<li>The <code>EnumVariantNames</code> macro has been renamed
<code>VariantNames</code>. The deprecation warning should steer you in
the right direction for fixing the warning.</li>
<li>The Iterator struct generated by EnumIter now has new bounds on it.
This shouldn't break code unless you manually
added the implementation in your code.</li>
<li><code>Display</code> now supports format strings using named fields
in the enum variant. This should be a no-op for most code.
However, if you were outputting a string like <code>"Hello
{field}"</code>, this will now be interpretted as a format
string.</li>
<li>EnumDiscriminant now inherits the repr and discriminant values from
your main enum. This makes the discriminant type
closer to a mirror of the original and that's always the goal.</li>
</ul>
<h3>New features</h3>
<ul>
<li>
<p>The <code>VariantArray</code> macro has been added. This macro adds
an associated constant <code>VARIANTS</code> to your enum. The constant
is a <code>&'static [Self]</code> slice so that you can access all
the variants of your enum. This only works on enums that only
have unit variants.</p>
<pre lang="rust"><code>use strum::VariantArray;
<p>#[derive(Debug, VariantArray)]
enum Color {
Red,
Blue,
Green,
}</p>
<p>fn main() {
println!("{:?}", Color::VARIANTS); // prints:
["Red", "Blue", "Green"]
}
</code></pre></p>
</li>
<li>
<p>The <code>EnumTable</code> macro has been <em>experimentally</em>
added. This macro adds a new type that stores an item for each variant
of the enum. This is useful for storing a value for each variant of an
enum. This is an experimental feature because
I'm not convinced the current api surface area is correct.</p>
<pre lang="rust"><code>use strum::EnumTable;
<p>#[derive(Copy, Clone, Debug, EnumTable)]
enum Color {
Red,
Blue,
</code></pre></p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/Peternator7/strum/blob/master/CHANGELOG.md">strum_macros's
changelog</a>.</em></p>
<blockquote>
<h2>0.26.1</h2>
<ul>
<li><a
href="https://redirect.github.com/Peternator7/strum/pull/325">#325</a>:
use <code>core</code> instead of <code>std</code> in VariantArray.</li>
</ul>
<h2>0.26.0</h2>
<h3>Breaking Changes</h3>
<ul>
<li>The <code>EnumVariantNames</code> macro has been renamed
<code>VariantNames</code>. The deprecation warning should steer you in
the right direction for fixing the warning.</li>
<li>The Iterator struct generated by EnumIter now has new bounds on it.
This shouldn't break code unless you manually
added the implementation in your code.</li>
<li><code>Display</code> now supports format strings using named fields
in the enum variant. This should be a no-op for most code.
However, if you were outputting a string like <code>"Hello
{field}"</code>, this will now be interpretted as a format
string.</li>
<li>EnumDiscriminant now inherits the repr and discriminant values from
your main enum. This makes the discriminant type
closer to a mirror of the original and that's always the goal.</li>
</ul>
<h3>New features</h3>
<ul>
<li>
<p>The <code>VariantArray</code> macro has been added. This macro adds
an associated constant <code>VARIANTS</code> to your enum. The constant
is a <code>&'static [Self]</code> slice so that you can access all
the variants of your enum. This only works on enums that only
have unit variants.</p>
<pre lang="rust"><code>use strum::VariantArray;
<p>#[derive(Debug, VariantArray)]
enum Color {
Red,
Blue,
Green,
}</p>
<p>fn main() {
println!("{:?}", Color::VARIANTS); // prints:
["Red", "Blue", "Green"]
}
</code></pre></p>
</li>
<li>
<p>The <code>EnumTable</code> macro has been <em>experimentally</em>
added. This macro adds a new type that stores an item for each variant
of the enum. This is useful for storing a value for each variant of an
enum. This is an experimental feature because
I'm not convinced the current api surface area is correct.</p>
<pre lang="rust"><code>use strum::EnumTable;
<p>#[derive(Copy, Clone, Debug, EnumTable)]
enum Color {
Red,
Blue,
Green,
</code></pre></p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a
href="https://github.com/Peternator7/strum/commits/v0.26.1">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-26 07:37:38 +00:00
|
|
|
strum_macros = "0.26"
|
2024-06-26 01:26:07 +00:00
|
|
|
nu-test-support = { path = "../nu-test-support", version = "0.95.1" }
|
2024-04-26 11:23:16 +00:00
|
|
|
pretty_assertions = { workspace = true }
|
2024-03-07 22:40:31 +00:00
|
|
|
rstest = { workspace = true }
|
Migrate to a new PWD API (#12603)
This is the first PR towards migrating to a new `$env.PWD` API that
returns potentially un-canonicalized paths. Refer to PR #12515 for
motivations.
## New API: `EngineState::cwd()`
The goal of the new API is to cover both parse-time and runtime use
case, and avoid unintentional misuse. It takes an `Option<Stack>` as
argument, which if supplied, will search for `$env.PWD` on the stack in
additional to the engine state. I think with this design, there's less
confusion over parse-time and runtime environments. If you have access
to a stack, just supply it; otherwise supply `None`.
## Deprecation of other PWD-related APIs
Other APIs are re-implemented using `EngineState::cwd()` and properly
documented. They're marked deprecated, but their behavior is unchanged.
Unused APIs are deleted, and code that accesses `$env.PWD` directly
without using an API is rewritten.
Deprecated APIs:
* `EngineState::current_work_dir()`
* `StateWorkingSet::get_cwd()`
* `env::current_dir()`
* `env::current_dir_str()`
* `env::current_dir_const()`
* `env::current_dir_str_const()`
Other changes:
* `EngineState::get_cwd()` (deleted)
* `StateWorkingSet::list_env()` (deleted)
* `repl::do_run_cmd()` (rewritten with `env::current_dir_str()`)
## `cd` and `pwd` now use logical paths by default
This pulls the changes from PR #12515. It's currently somewhat broken
because using non-canonicalized paths exposed a bug in our path
normalization logic (Issue #12602). Once that is fixed, this should
work.
## Future plans
This PR needs some tests. Which test helpers should I use, and where
should I put those tests?
I noticed that unquoted paths are expanded within `eval_filepath()` and
`eval_directory()` before they even reach the `cd` command. This means
every paths is expanded twice. Is this intended?
Once this PR lands, the plan is to review all usages of the deprecated
APIs and migrate them to `EngineState::cwd()`. In the meantime, these
usages are annotated with `#[allow(deprecated)]` to avoid breaking CI.
---------
Co-authored-by: Jakub Žádník <kubouch@gmail.com>
2024-05-03 11:33:09 +00:00
|
|
|
tempfile = { workspace = true }
|
2024-05-13 18:48:38 +00:00
|
|
|
os_pipe = { workspace = true }
|
2023-11-29 15:17:22 +00:00
|
|
|
|
|
|
|
|
[package.metadata.docs.rs]
|
Internal representation (IR) compiler and evaluator (#13330)
# Description
This PR adds an internal representation language to Nushell, offering an
alternative evaluator based on simple instructions, stream-containing
registers, and indexed control flow. The number of registers required is
determined statically at compile-time, and the fixed size required is
allocated upon entering the block.
Each instruction is associated with a span, which makes going backwards
from IR instructions to source code very easy.
Motivations for IR:
1. **Performance.** By simplifying the evaluation path and making it
more cache-friendly and branch predictor-friendly, code that does a lot
of computation in Nushell itself can be sped up a decent bit. Because
the IR is fairly easy to reason about, we can also implement
optimization passes in the future to eliminate and simplify code.
2. **Correctness.** The instructions mostly have very simple and
easily-specified behavior, so hopefully engine changes are a little bit
easier to reason about, and they can be specified in a more formal way
at some point. I have made an effort to document each of the
instructions in the docs for the enum itself in a reasonably specific
way. Some of the errors that would have happened during evaluation
before are now moved to the compilation step instead, because they don't
make sense to check during evaluation.
3. **As an intermediate target.** This is a good step for us to bring
the [`new-nu-parser`](https://github.com/nushell/new-nu-parser) in at
some point, as code generated from new AST can be directly compared to
code generated from old AST. If the IR code is functionally equivalent,
it will behave the exact same way.
4. **Debugging.** With a little bit more work, we can probably give
control over advancing the virtual machine that `IrBlock`s run on to
some sort of external driver, making things like breakpoints and single
stepping possible. Tools like `view ir` and [`explore
ir`](https://github.com/devyn/nu_plugin_explore_ir) make it easier than
before to see what exactly is going on with your Nushell code.
The goal is to eventually replace the AST evaluator entirely, once we're
sure it's working just as well. You can help dogfood this by running
Nushell with `$env.NU_USE_IR` set to some value. The environment
variable is checked when Nushell starts, so config runs with IR, or it
can also be set on a line at the REPL to change it dynamically. It is
also checked when running `do` in case within a script you want to just
run a specific piece of code with or without IR.
# Example
```nushell
view ir { |data|
mut sum = 0
for n in $data {
$sum += $n
}
$sum
}
```
```gas
# 3 registers, 19 instructions, 0 bytes of data
0: load-literal %0, int(0)
1: store-variable var 904, %0 # let
2: drain %0
3: drop %0
4: load-variable %1, var 903
5: iterate %0, %1, end 15 # for, label(1), from(14:)
6: store-variable var 905, %0
7: load-variable %0, var 904
8: load-variable %2, var 905
9: binary-op %0, Math(Plus), %2
10: span %0
11: store-variable var 904, %0
12: load-literal %0, nothing
13: drain %0
14: jump 5
15: drop %0 # label(0), from(5:)
16: drain %0
17: load-variable %0, var 904
18: return %0
```
# Benchmarks
All benchmarks run on a base model Mac Mini M1.
## Iterative Fibonacci sequence
This is about as best case as possible, making use of the much faster
control flow. Most code will not experience a speed improvement nearly
this large.
```nushell
def fib [n: int] {
mut a = 0
mut b = 1
for _ in 2..=$n {
let c = $a + $b
$a = $b
$b = $c
}
$b
}
use std bench
bench { 0..50 | each { |n| fib $n } }
```
IR disabled:
```
╭───────┬─────────────────╮
│ mean │ 1ms 924µs 665ns │
│ min │ 1ms 700µs 83ns │
│ max │ 3ms 450µs 125ns │
│ std │ 395µs 759ns │
│ times │ [list 50 items] │
╰───────┴─────────────────╯
```
IR enabled:
```
╭───────┬─────────────────╮
│ mean │ 452µs 820ns │
│ min │ 427µs 417ns │
│ max │ 540µs 167ns │
│ std │ 17µs 158ns │
│ times │ [list 50 items] │
╰───────┴─────────────────╯
```
##
[gradient_benchmark_no_check.nu](https://github.com/nushell/nu_scripts/blob/main/benchmarks/gradient_benchmark_no_check.nu)
IR disabled:
```
╭───┬──────────────────╮
│ 0 │ 27ms 929µs 958ns │
│ 1 │ 21ms 153µs 459ns │
│ 2 │ 18ms 639µs 666ns │
│ 3 │ 19ms 554µs 583ns │
│ 4 │ 13ms 383µs 375ns │
│ 5 │ 11ms 328µs 208ns │
│ 6 │ 5ms 659µs 542ns │
╰───┴──────────────────╯
```
IR enabled:
```
╭───┬──────────────────╮
│ 0 │ 22ms 662µs │
│ 1 │ 17ms 221µs 792ns │
│ 2 │ 14ms 786µs 708ns │
│ 3 │ 13ms 876µs 834ns │
│ 4 │ 13ms 52µs 875ns │
│ 5 │ 11ms 269µs 666ns │
│ 6 │ 6ms 942µs 500ns │
╰───┴──────────────────╯
```
##
[random-bytes.nu](https://github.com/nushell/nu_scripts/blob/main/benchmarks/random-bytes.nu)
I got pretty random results out of this benchmark so I decided not to
include it. Not clear why.
# User-Facing Changes
- IR compilation errors may appear even if the user isn't evaluating
with IR.
- IR evaluation can be enabled by setting the `NU_USE_IR` environment
variable to any value.
- New command `view ir` pretty-prints the IR for a block, and `view ir
--json` can be piped into an external tool like [`explore
ir`](https://github.com/devyn/nu_plugin_explore_ir).
# Tests + Formatting
All tests are passing with `NU_USE_IR=1`, and I've added some more eval
tests to compare the results for some very core operations. I will
probably want to add some more so we don't have to always check
`NU_USE_IR=1 toolkit test --workspace` on a regular basis.
# After Submitting
- [ ] release notes
- [ ] further documentation of instructions?
- [ ] post-release: publish `nu_plugin_explore_ir`
2024-07-11 00:33:59 +00:00
|
|
|
all-features = true
|
