nushell/crates/nu-protocol/src/eval_const.rs

533 lines
16 KiB
Rust
Raw Normal View History

//! Implementation of const-evaluation
//!
//! This enables you to assign `const`-constants and execute parse-time code dependent on this.
//! e.g. `source $my_const`
use crate::{
IO and redirection overhaul (#11934) # Description The PR overhauls how IO redirection is handled, allowing more explicit and fine-grain control over `stdout` and `stderr` output as well as more efficient IO and piping. To summarize the changes in this PR: - Added a new `IoStream` type to indicate the intended destination for a pipeline element's `stdout` and `stderr`. - The `stdout` and `stderr` `IoStream`s are stored in the `Stack` and to avoid adding 6 additional arguments to every eval function and `Command::run`. The `stdout` and `stderr` streams can be temporarily overwritten through functions on `Stack` and these functions will return a guard that restores the original `stdout` and `stderr` when dropped. - In the AST, redirections are now directly part of a `PipelineElement` as a `Option<Redirection>` field instead of having multiple different `PipelineElement` enum variants for each kind of redirection. This required changes to the parser, mainly in `lite_parser.rs`. - `Command`s can also set a `IoStream` override/redirection which will apply to the previous command in the pipeline. This is used, for example, in `ignore` to allow the previous external command to have its stdout redirected to `Stdio::null()` at spawn time. In contrast, the current implementation has to create an os pipe and manually consume the output on nushell's side. File and pipe redirections (`o>`, `e>`, `e>|`, etc.) have precedence over overrides from commands. This PR improves piping and IO speed, partially addressing #10763. Using the `throughput` command from that issue, this PR gives the following speedup on my setup for the commands below: | Command | Before (MB/s) | After (MB/s) | Bash (MB/s) | | --------------------------- | -------------:| ------------:| -----------:| | `throughput o> /dev/null` | 1169 | 52938 | 54305 | | `throughput \| ignore` | 840 | 55438 | N/A | | `throughput \| null` | Error | 53617 | N/A | | `throughput \| rg 'x'` | 1165 | 3049 | 3736 | | `(throughput) \| rg 'x'` | 810 | 3085 | 3815 | (Numbers above are the median samples for throughput) This PR also paves the way to refactor our `ExternalStream` handling in the various commands. For example, this PR already fixes the following code: ```nushell ^sh -c 'echo -n "hello "; sleep 0; echo "world"' | find "hello world" ``` This returns an empty list on 0.90.1 and returns a highlighted "hello world" on this PR. Since the `stdout` and `stderr` `IoStream`s are available to commands when they are run, then this unlocks the potential for more convenient behavior. E.g., the `find` command can disable its ansi highlighting if it detects that the output `IoStream` is not the terminal. Knowing the output streams will also allow background job output to be redirected more easily and efficiently. # User-Facing Changes - External commands returned from closures will be collected (in most cases): ```nushell 1..2 | each {|_| nu -c "print a" } ``` This gives `["a", "a"]` on this PR, whereas this used to print "a\na\n" and then return an empty list. ```nushell 1..2 | each {|_| nu -c "print -e a" } ``` This gives `["", ""]` and prints "a\na\n" to stderr, whereas this used to return an empty list and print "a\na\n" to stderr. - Trailing new lines are always trimmed for external commands when piping into internal commands or collecting it as a value. (Failure to decode the output as utf-8 will keep the trailing newline for the last binary value.) In the current nushell version, the following three code snippets differ only in parenthesis placement, but they all also have different outputs: 1. `1..2 | each { ^echo a }` ``` a a ╭────────────╮ │ empty list │ ╰────────────╯ ``` 2. `1..2 | each { (^echo a) }` ``` ╭───┬───╮ │ 0 │ a │ │ 1 │ a │ ╰───┴───╯ ``` 3. `1..2 | (each { ^echo a })` ``` ╭───┬───╮ │ 0 │ a │ │ │ │ │ 1 │ a │ │ │ │ ╰───┴───╯ ``` But in this PR, the above snippets will all have the same output: ``` ╭───┬───╮ │ 0 │ a │ │ 1 │ a │ ╰───┴───╯ ``` - All existing flags on `run-external` are now deprecated. - File redirections now apply to all commands inside a code block: ```nushell (nu -c "print -e a"; nu -c "print -e b") e> test.out ``` This gives "a\nb\n" in `test.out` and prints nothing. The same result would happen when printing to stdout and using a `o>` file redirection. - External command output will (almost) never be ignored, and ignoring output must be explicit now: ```nushell (^echo a; ^echo b) ``` This prints "a\nb\n", whereas this used to print only "b\n". This only applies to external commands; values and internal commands not in return position will not print anything (e.g., `(echo a; echo b)` still only prints "b"). - `complete` now always captures stderr (`do` is not necessary). # After Submitting The language guide and other documentation will need to be updated.
2024-03-14 20:51:55 +00:00
ast::{Assignment, Block, Call, Expr, Expression, ExternalArgument},
debugger::{DebugContext, WithoutDebug},
2023-09-01 06:18:55 +00:00
engine::{EngineState, StateWorkingSet},
eval_base::Eval,
record, BlockId, Config, HistoryFileFormat, PipelineData, Record, ShellError, Span, Value,
VarId,
};
2023-09-01 06:18:55 +00:00
use nu_system::os_info::{get_kernel_version, get_os_arch, get_os_family, get_os_name};
use std::{
path::{Path, PathBuf},
sync::Arc,
};
2023-09-01 06:18:55 +00:00
Fix circular source causing Nushell to crash (#12262) # Description EngineState now tracks the script currently running, instead of the parent directory of the script. This also provides an easy way to expose the current running script to the user (Issue #12195). Similarly, StateWorkingSet now tracks scripts instead of directories. `parsed_module_files` and `currently_parsed_pwd` are merged into one variable, `scripts`, which acts like a stack for tracking the current running script (which is on the top of the stack). Circular import check is added for `source` operations, in addition to module import. A simple testcase is added for circular source. <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> It shouldn't have any user facing changes.
2024-04-19 06:38:08 +00:00
/// Create a Value for `$nu`.
pub(crate) fn create_nu_constant(engine_state: &EngineState, span: Span) -> Value {
fn canonicalize_path(engine_state: &EngineState, path: &Path) -> PathBuf {
Migrate to a new PWD API (#12603) This is the first PR towards migrating to a new `$env.PWD` API that returns potentially un-canonicalized paths. Refer to PR #12515 for motivations. ## New API: `EngineState::cwd()` The goal of the new API is to cover both parse-time and runtime use case, and avoid unintentional misuse. It takes an `Option<Stack>` as argument, which if supplied, will search for `$env.PWD` on the stack in additional to the engine state. I think with this design, there's less confusion over parse-time and runtime environments. If you have access to a stack, just supply it; otherwise supply `None`. ## Deprecation of other PWD-related APIs Other APIs are re-implemented using `EngineState::cwd()` and properly documented. They're marked deprecated, but their behavior is unchanged. Unused APIs are deleted, and code that accesses `$env.PWD` directly without using an API is rewritten. Deprecated APIs: * `EngineState::current_work_dir()` * `StateWorkingSet::get_cwd()` * `env::current_dir()` * `env::current_dir_str()` * `env::current_dir_const()` * `env::current_dir_str_const()` Other changes: * `EngineState::get_cwd()` (deleted) * `StateWorkingSet::list_env()` (deleted) * `repl::do_run_cmd()` (rewritten with `env::current_dir_str()`) ## `cd` and `pwd` now use logical paths by default This pulls the changes from PR #12515. It's currently somewhat broken because using non-canonicalized paths exposed a bug in our path normalization logic (Issue #12602). Once that is fixed, this should work. ## Future plans This PR needs some tests. Which test helpers should I use, and where should I put those tests? I noticed that unquoted paths are expanded within `eval_filepath()` and `eval_directory()` before they even reach the `cd` command. This means every paths is expanded twice. Is this intended? Once this PR lands, the plan is to review all usages of the deprecated APIs and migrate them to `EngineState::cwd()`. In the meantime, these usages are annotated with `#[allow(deprecated)]` to avoid breaking CI. --------- Co-authored-by: Jakub Žádník <kubouch@gmail.com>
2024-05-03 11:33:09 +00:00
#[allow(deprecated)]
2023-09-01 06:18:55 +00:00
let cwd = engine_state.current_work_dir();
if path.exists() {
match nu_path::canonicalize_with(path, cwd) {
Ok(canon_path) => canon_path,
Err(_) => path.to_owned(),
2023-09-01 06:18:55 +00:00
}
} else {
path.to_owned()
2023-09-01 06:18:55 +00:00
}
}
let mut record = Record::new();
let config_path = match nu_path::nu_config_dir() {
Some(path) => Ok(canonicalize_path(engine_state, path.as_ref())),
Add ConfigDirNotFound error (#11849) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Currently, there's multiple places that look for a config directory, and each of them has different error messages when it can't be found. This PR makes a `ConfigDirNotFound` error to standardize the error message for all of these cases. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Previously, the errors in `create_nu_constant()` would say which config file Nushell was trying to get when it couldn't find the config directory. Now it doesn't. However, I think that's fine, given that it doesn't matter whether it couldn't find the config directory while looking for `login.nu` or `env.nu`, it only matters that it couldn't find it. This is what the error looks like: ![image](https://github.com/nushell/nushell/assets/45539777/52298ed4-f9e9-4900-bb94-1154d389efa7) # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> --------- Co-authored-by: Antoine Stevan <44101798+amtoine@users.noreply.github.com>
2024-02-26 07:42:20 +00:00
None => Err(Value::error(
ShellError::ConfigDirNotFound { span: Some(span) },
span,
)),
};
2023-09-01 06:18:55 +00:00
record.push(
"default-config-dir",
Add ConfigDirNotFound error (#11849) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Currently, there's multiple places that look for a config directory, and each of them has different error messages when it can't be found. This PR makes a `ConfigDirNotFound` error to standardize the error message for all of these cases. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Previously, the errors in `create_nu_constant()` would say which config file Nushell was trying to get when it couldn't find the config directory. Now it doesn't. However, I think that's fine, given that it doesn't matter whether it couldn't find the config directory while looking for `login.nu` or `env.nu`, it only matters that it couldn't find it. This is what the error looks like: ![image](https://github.com/nushell/nushell/assets/45539777/52298ed4-f9e9-4900-bb94-1154d389efa7) # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> --------- Co-authored-by: Antoine Stevan <44101798+amtoine@users.noreply.github.com>
2024-02-26 07:42:20 +00:00
config_path.as_ref().map_or_else(
|e| e.clone(),
|path| Value::string(path.to_string_lossy(), span),
),
2023-09-01 06:18:55 +00:00
);
record.push(
"config-path",
if let Some(path) = engine_state.get_config_path("config-path") {
let canon_config_path = canonicalize_path(engine_state, path);
Value::string(canon_config_path.to_string_lossy(), span)
} else {
Add ConfigDirNotFound error (#11849) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Currently, there's multiple places that look for a config directory, and each of them has different error messages when it can't be found. This PR makes a `ConfigDirNotFound` error to standardize the error message for all of these cases. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Previously, the errors in `create_nu_constant()` would say which config file Nushell was trying to get when it couldn't find the config directory. Now it doesn't. However, I think that's fine, given that it doesn't matter whether it couldn't find the config directory while looking for `login.nu` or `env.nu`, it only matters that it couldn't find it. This is what the error looks like: ![image](https://github.com/nushell/nushell/assets/45539777/52298ed4-f9e9-4900-bb94-1154d389efa7) # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> --------- Co-authored-by: Antoine Stevan <44101798+amtoine@users.noreply.github.com>
2024-02-26 07:42:20 +00:00
config_path.clone().map_or_else(
|e| e,
|mut path| {
path.push("config.nu");
Canonicalize each component of config files (#12167) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Because `std::fs::canonicalize` requires the path to exist, this PR makes it so that when canonicalizing any config file, the `$nu.default-config-dir/nushell` part is canonicalized first, then `$nu.default-config-dir/nushell/foo.nu` is canonicalized. This should also fix the issue @devyn pointed out [here](https://github.com/nushell/nushell/pull/12118#issuecomment-1989546708) where a couple of the tests failed if one's `~/.config/nushell` folder was actually a symlink to a different folder. The tests previously didn't canonicalize the expected paths. I was going to make a PR that caches the config directory on startup (as suggested by fdncred and Ian in Discord), but I can make that part of this PR if we want to avoid creating unnecessary PRs. I think it probably makes more sense to separate them though. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-03-13 11:26:06 +00:00
let canon_config_path = canonicalize_path(engine_state, &path);
Value::string(canon_config_path.to_string_lossy(), span)
},
2023-09-01 06:18:55 +00:00
)
},
);
record.push(
"env-path",
if let Some(path) = engine_state.get_config_path("env-path") {
let canon_env_path = canonicalize_path(engine_state, path);
Value::string(canon_env_path.to_string_lossy(), span)
} else {
Add ConfigDirNotFound error (#11849) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Currently, there's multiple places that look for a config directory, and each of them has different error messages when it can't be found. This PR makes a `ConfigDirNotFound` error to standardize the error message for all of these cases. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Previously, the errors in `create_nu_constant()` would say which config file Nushell was trying to get when it couldn't find the config directory. Now it doesn't. However, I think that's fine, given that it doesn't matter whether it couldn't find the config directory while looking for `login.nu` or `env.nu`, it only matters that it couldn't find it. This is what the error looks like: ![image](https://github.com/nushell/nushell/assets/45539777/52298ed4-f9e9-4900-bb94-1154d389efa7) # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> --------- Co-authored-by: Antoine Stevan <44101798+amtoine@users.noreply.github.com>
2024-02-26 07:42:20 +00:00
config_path.clone().map_or_else(
|e| e,
|mut path| {
path.push("env.nu");
Canonicalize each component of config files (#12167) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Because `std::fs::canonicalize` requires the path to exist, this PR makes it so that when canonicalizing any config file, the `$nu.default-config-dir/nushell` part is canonicalized first, then `$nu.default-config-dir/nushell/foo.nu` is canonicalized. This should also fix the issue @devyn pointed out [here](https://github.com/nushell/nushell/pull/12118#issuecomment-1989546708) where a couple of the tests failed if one's `~/.config/nushell` folder was actually a symlink to a different folder. The tests previously didn't canonicalize the expected paths. I was going to make a PR that caches the config directory on startup (as suggested by fdncred and Ian in Discord), but I can make that part of this PR if we want to avoid creating unnecessary PRs. I think it probably makes more sense to separate them though. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-03-13 11:26:06 +00:00
let canon_env_path = canonicalize_path(engine_state, &path);
Value::string(canon_env_path.to_string_lossy(), span)
},
2023-09-01 06:18:55 +00:00
)
},
);
record.push(
"history-path",
Add ConfigDirNotFound error (#11849) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Currently, there's multiple places that look for a config directory, and each of them has different error messages when it can't be found. This PR makes a `ConfigDirNotFound` error to standardize the error message for all of these cases. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Previously, the errors in `create_nu_constant()` would say which config file Nushell was trying to get when it couldn't find the config directory. Now it doesn't. However, I think that's fine, given that it doesn't matter whether it couldn't find the config directory while looking for `login.nu` or `env.nu`, it only matters that it couldn't find it. This is what the error looks like: ![image](https://github.com/nushell/nushell/assets/45539777/52298ed4-f9e9-4900-bb94-1154d389efa7) # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> --------- Co-authored-by: Antoine Stevan <44101798+amtoine@users.noreply.github.com>
2024-02-26 07:42:20 +00:00
config_path.clone().map_or_else(
|e| e,
|mut path| {
match engine_state.config.history.file_format {
HistoryFileFormat::Sqlite => {
path.push("history.sqlite3");
}
HistoryFileFormat::Plaintext => {
Add ConfigDirNotFound error (#11849) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Currently, there's multiple places that look for a config directory, and each of them has different error messages when it can't be found. This PR makes a `ConfigDirNotFound` error to standardize the error message for all of these cases. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Previously, the errors in `create_nu_constant()` would say which config file Nushell was trying to get when it couldn't find the config directory. Now it doesn't. However, I think that's fine, given that it doesn't matter whether it couldn't find the config directory while looking for `login.nu` or `env.nu`, it only matters that it couldn't find it. This is what the error looks like: ![image](https://github.com/nushell/nushell/assets/45539777/52298ed4-f9e9-4900-bb94-1154d389efa7) # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> --------- Co-authored-by: Antoine Stevan <44101798+amtoine@users.noreply.github.com>
2024-02-26 07:42:20 +00:00
path.push("history.txt");
}
2023-09-01 06:18:55 +00:00
}
Add ConfigDirNotFound error (#11849) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Currently, there's multiple places that look for a config directory, and each of them has different error messages when it can't be found. This PR makes a `ConfigDirNotFound` error to standardize the error message for all of these cases. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Previously, the errors in `create_nu_constant()` would say which config file Nushell was trying to get when it couldn't find the config directory. Now it doesn't. However, I think that's fine, given that it doesn't matter whether it couldn't find the config directory while looking for `login.nu` or `env.nu`, it only matters that it couldn't find it. This is what the error looks like: ![image](https://github.com/nushell/nushell/assets/45539777/52298ed4-f9e9-4900-bb94-1154d389efa7) # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> --------- Co-authored-by: Antoine Stevan <44101798+amtoine@users.noreply.github.com>
2024-02-26 07:42:20 +00:00
let canon_hist_path = canonicalize_path(engine_state, &path);
Value::string(canon_hist_path.to_string_lossy(), span)
},
),
2023-09-01 06:18:55 +00:00
);
record.push(
"loginshell-path",
Add ConfigDirNotFound error (#11849) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Currently, there's multiple places that look for a config directory, and each of them has different error messages when it can't be found. This PR makes a `ConfigDirNotFound` error to standardize the error message for all of these cases. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Previously, the errors in `create_nu_constant()` would say which config file Nushell was trying to get when it couldn't find the config directory. Now it doesn't. However, I think that's fine, given that it doesn't matter whether it couldn't find the config directory while looking for `login.nu` or `env.nu`, it only matters that it couldn't find it. This is what the error looks like: ![image](https://github.com/nushell/nushell/assets/45539777/52298ed4-f9e9-4900-bb94-1154d389efa7) # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> --------- Co-authored-by: Antoine Stevan <44101798+amtoine@users.noreply.github.com>
2024-02-26 07:42:20 +00:00
config_path.clone().map_or_else(
|e| e,
|mut path| {
path.push("login.nu");
let canon_login_path = canonicalize_path(engine_state, &path);
Value::string(canon_login_path.to_string_lossy(), span)
},
),
2023-09-01 06:18:55 +00:00
);
#[cfg(feature = "plugin")]
{
record.push(
"plugin-path",
if let Some(path) = &engine_state.plugin_path {
2023-09-01 06:18:55 +00:00
let canon_plugin_path = canonicalize_path(engine_state, path);
Value::string(canon_plugin_path.to_string_lossy(), span)
} else {
Add ConfigDirNotFound error (#11849) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Currently, there's multiple places that look for a config directory, and each of them has different error messages when it can't be found. This PR makes a `ConfigDirNotFound` error to standardize the error message for all of these cases. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Previously, the errors in `create_nu_constant()` would say which config file Nushell was trying to get when it couldn't find the config directory. Now it doesn't. However, I think that's fine, given that it doesn't matter whether it couldn't find the config directory while looking for `login.nu` or `env.nu`, it only matters that it couldn't find it. This is what the error looks like: ![image](https://github.com/nushell/nushell/assets/45539777/52298ed4-f9e9-4900-bb94-1154d389efa7) # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> --------- Co-authored-by: Antoine Stevan <44101798+amtoine@users.noreply.github.com>
2024-02-26 07:42:20 +00:00
// If there are no signatures, we should still populate the plugin path
config_path.clone().map_or_else(
|e| e,
|mut path| {
path.push("plugin.msgpackz");
Canonicalize default-config-dir and plugin-path (#11999) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR makes sure `$nu.default-config-dir` and `$nu.plugin-path` are canonicalized. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> `$nu.default-config-dir` (and `$nu.plugin-path`) will now give canonical paths, with symlinks and whatnot resolved. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> I've added a couple of tests to check that even if the config folder and/or any of the config files within are symlinks, the `$nu.*` variables are properly canonicalized. These tests unfortunately only run on Linux and MacOS, because I couldn't figure out how to change the config directory on Windows. Also, given that they involve creating files, I'm not sure if they're excessive, so I could remove one or two of them. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-03-02 17:15:31 +00:00
let canonical_plugin_path = canonicalize_path(engine_state, &path);
Value::string(canonical_plugin_path.to_string_lossy(), span)
},
2023-09-01 06:18:55 +00:00
)
},
);
}
record.push(
"home-path",
if let Some(path) = nu_path::home_dir() {
let canon_home_path = canonicalize_path(engine_state, path.as_ref());
2023-09-01 06:18:55 +00:00
Value::string(canon_home_path.to_string_lossy(), span)
} else {
Value::error(
ShellError::IOError {
msg: "Could not get home path".into(),
},
span,
)
2023-09-01 06:18:55 +00:00
},
);
add `$nu.data-dir` for completions and `$nu.cache-dir` for other uses (#13122) # Description This PR is an attempt to add a standard location for people to put completions in. I saw this topic come up again recently and IIRC we decided to create a standard location. I used the dirs-next crate to dictate where these locations are. I know some people won't like that but at least this gets the ball rolling in a direction that has a standard directory. This is what the default NU_LIB_DIRS looks like now in the default_env.nu. It should also be like this when starting nushell with `nu -n` ```nushell $env.NU_LIB_DIRS = [ ($nu.default-config-dir | path join 'scripts') # add <nushell-config-dir>/scripts ($nu.data-dir | path join 'completions') # default home for nushell completions ] ``` I also added these default folders to the `$nu` variable so now there is `$nu.data-path` and `$nu.cache-path`. ## Data Dir Default ![image](https://github.com/nushell/nushell/assets/343840/aeeb7cd6-17b4-43e8-bb6f-986a0c7fce23) While I was in there, I also decided to add a cache dir ## Cache Dir Default ![image](https://github.com/nushell/nushell/assets/343840/87dead66-4911-4f67-bfb2-acb16f386674) ### This is what the default looks like in Ubuntu. ![image](https://github.com/nushell/nushell/assets/343840/bca8eae8-8c18-47e8-b64f-3efe34f0004f) ### This is what it looks like with XDG_CACHE_HOME and XDG_DATA_HOME overridden ```nushell XDG_DATA_HOME=/tmp/data_home XDG_CACHE_HOME=/tmp/cache_home cargo r ``` ![image](https://github.com/nushell/nushell/assets/343840/fae86d50-9821-41f1-868e-3814eca3730b) ### This is what the defaults look like in Windows (username scrubbed to protect the innocent) ![image](https://github.com/nushell/nushell/assets/343840/3ebdb5cd-0150-448c-aff5-c57053e4788a) How my NU_LIB_DIRS is set in the images above ```nushell $env.NU_LIB_DIRS = [ ($nu.default-config-dir | path join 'scripts') # add <nushell-config-dir>/scripts '/Users/fdncred/src/nu_scripts' ($nu.config-path | path dirname) ($nu.data-dir | path join 'completions') # default home for nushell completions ] ``` Let the debate begin. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-06-11 19:10:31 +00:00
record.push(
"data-dir",
if let Some(path) = nu_path::data_dir() {
let mut canon_data_path = canonicalize_path(engine_state, path.as_ref());
add `$nu.data-dir` for completions and `$nu.cache-dir` for other uses (#13122) # Description This PR is an attempt to add a standard location for people to put completions in. I saw this topic come up again recently and IIRC we decided to create a standard location. I used the dirs-next crate to dictate where these locations are. I know some people won't like that but at least this gets the ball rolling in a direction that has a standard directory. This is what the default NU_LIB_DIRS looks like now in the default_env.nu. It should also be like this when starting nushell with `nu -n` ```nushell $env.NU_LIB_DIRS = [ ($nu.default-config-dir | path join 'scripts') # add <nushell-config-dir>/scripts ($nu.data-dir | path join 'completions') # default home for nushell completions ] ``` I also added these default folders to the `$nu` variable so now there is `$nu.data-path` and `$nu.cache-path`. ## Data Dir Default ![image](https://github.com/nushell/nushell/assets/343840/aeeb7cd6-17b4-43e8-bb6f-986a0c7fce23) While I was in there, I also decided to add a cache dir ## Cache Dir Default ![image](https://github.com/nushell/nushell/assets/343840/87dead66-4911-4f67-bfb2-acb16f386674) ### This is what the default looks like in Ubuntu. ![image](https://github.com/nushell/nushell/assets/343840/bca8eae8-8c18-47e8-b64f-3efe34f0004f) ### This is what it looks like with XDG_CACHE_HOME and XDG_DATA_HOME overridden ```nushell XDG_DATA_HOME=/tmp/data_home XDG_CACHE_HOME=/tmp/cache_home cargo r ``` ![image](https://github.com/nushell/nushell/assets/343840/fae86d50-9821-41f1-868e-3814eca3730b) ### This is what the defaults look like in Windows (username scrubbed to protect the innocent) ![image](https://github.com/nushell/nushell/assets/343840/3ebdb5cd-0150-448c-aff5-c57053e4788a) How my NU_LIB_DIRS is set in the images above ```nushell $env.NU_LIB_DIRS = [ ($nu.default-config-dir | path join 'scripts') # add <nushell-config-dir>/scripts '/Users/fdncred/src/nu_scripts' ($nu.config-path | path dirname) ($nu.data-dir | path join 'completions') # default home for nushell completions ] ``` Let the debate begin. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-06-11 19:10:31 +00:00
canon_data_path.push("nushell");
Value::string(canon_data_path.to_string_lossy(), span)
} else {
Value::error(
ShellError::IOError {
msg: "Could not get data path".into(),
},
span,
)
},
);
record.push(
"cache-dir",
if let Some(path) = nu_path::cache_dir() {
let mut canon_cache_path = canonicalize_path(engine_state, path.as_ref());
add `$nu.data-dir` for completions and `$nu.cache-dir` for other uses (#13122) # Description This PR is an attempt to add a standard location for people to put completions in. I saw this topic come up again recently and IIRC we decided to create a standard location. I used the dirs-next crate to dictate where these locations are. I know some people won't like that but at least this gets the ball rolling in a direction that has a standard directory. This is what the default NU_LIB_DIRS looks like now in the default_env.nu. It should also be like this when starting nushell with `nu -n` ```nushell $env.NU_LIB_DIRS = [ ($nu.default-config-dir | path join 'scripts') # add <nushell-config-dir>/scripts ($nu.data-dir | path join 'completions') # default home for nushell completions ] ``` I also added these default folders to the `$nu` variable so now there is `$nu.data-path` and `$nu.cache-path`. ## Data Dir Default ![image](https://github.com/nushell/nushell/assets/343840/aeeb7cd6-17b4-43e8-bb6f-986a0c7fce23) While I was in there, I also decided to add a cache dir ## Cache Dir Default ![image](https://github.com/nushell/nushell/assets/343840/87dead66-4911-4f67-bfb2-acb16f386674) ### This is what the default looks like in Ubuntu. ![image](https://github.com/nushell/nushell/assets/343840/bca8eae8-8c18-47e8-b64f-3efe34f0004f) ### This is what it looks like with XDG_CACHE_HOME and XDG_DATA_HOME overridden ```nushell XDG_DATA_HOME=/tmp/data_home XDG_CACHE_HOME=/tmp/cache_home cargo r ``` ![image](https://github.com/nushell/nushell/assets/343840/fae86d50-9821-41f1-868e-3814eca3730b) ### This is what the defaults look like in Windows (username scrubbed to protect the innocent) ![image](https://github.com/nushell/nushell/assets/343840/3ebdb5cd-0150-448c-aff5-c57053e4788a) How my NU_LIB_DIRS is set in the images above ```nushell $env.NU_LIB_DIRS = [ ($nu.default-config-dir | path join 'scripts') # add <nushell-config-dir>/scripts '/Users/fdncred/src/nu_scripts' ($nu.config-path | path dirname) ($nu.data-dir | path join 'completions') # default home for nushell completions ] ``` Let the debate begin. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-06-11 19:10:31 +00:00
canon_cache_path.push("nushell");
Value::string(canon_cache_path.to_string_lossy(), span)
} else {
Value::error(
ShellError::IOError {
msg: "Could not get cache path".into(),
},
span,
)
},
);
add a system level folder for future autoloading (#13180) # Description This PR adds a directory to the `$nu` constant that shows where the system level autoload directory is located at. This folder is modifiable at compile time with environment variables. ```rust // Create a system level directory for nushell scripts, modules, completions, etc // that can be changed by setting the NU_VENDOR_AUTOLOAD_DIR env var on any platform // before nushell is compiled OR if NU_VENDOR_AUTOLOAD_DIR is not set for non-windows // systems, the PREFIX env var can be set before compile and used as PREFIX/nushell/vendor/autoload // pseudo code // if env var NU_VENDOR_AUTOLOAD_DIR is set, in any platform, use it // if not, if windows, use ALLUSERPROFILE\nushell\vendor\autoload // if not, if non-windows, if env var PREFIX is set, use PREFIX/share/nushell/vendor/autoload // if not, use the default /usr/share/nushell/vendor/autoload ``` ### Windows default ```nushell ❯ $nu.vendor-autoload-dir C:\ProgramData\nushell\vendor\autoload ``` ### Non-Windows default ```nushell ❯ $nu.vendor-autoload-dir /usr/local/share/nushell/vendor/autoload ``` ### Non-Windows with PREFIX set ```nushell ❯ PREFIX=/usr/bob cargo r ❯ $nu.vendor-autoload-dir /usr/bob/share/nushell/vendor/autoload ``` ### Non-Windows with NU_VENDOR_AUTOLOAD_DIR set ```nushell ❯ NU_VENDOR_AUTOLOAD_DIR=/some/other/path/nushell/stuff cargo r ❯ $nu.vendor-autoload-dir /some/other/path/nushell/stuff ``` > [!IMPORTANT] To be clear, this PR does not do the auto-loading, it just sets up the folder to support that functionality that can be added in a later PR. The PR also does not create the folder defined. It's just setting the $nu constant.   # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-06-20 11:30:43 +00:00
record.push(
"vendor-autoload-dirs",
Value::list(
get_vendor_autoload_dirs(engine_state)
.iter()
.map(|path| Value::string(path.to_string_lossy(), span))
.collect(),
span,
),
add a system level folder for future autoloading (#13180) # Description This PR adds a directory to the `$nu` constant that shows where the system level autoload directory is located at. This folder is modifiable at compile time with environment variables. ```rust // Create a system level directory for nushell scripts, modules, completions, etc // that can be changed by setting the NU_VENDOR_AUTOLOAD_DIR env var on any platform // before nushell is compiled OR if NU_VENDOR_AUTOLOAD_DIR is not set for non-windows // systems, the PREFIX env var can be set before compile and used as PREFIX/nushell/vendor/autoload // pseudo code // if env var NU_VENDOR_AUTOLOAD_DIR is set, in any platform, use it // if not, if windows, use ALLUSERPROFILE\nushell\vendor\autoload // if not, if non-windows, if env var PREFIX is set, use PREFIX/share/nushell/vendor/autoload // if not, use the default /usr/share/nushell/vendor/autoload ``` ### Windows default ```nushell ❯ $nu.vendor-autoload-dir C:\ProgramData\nushell\vendor\autoload ``` ### Non-Windows default ```nushell ❯ $nu.vendor-autoload-dir /usr/local/share/nushell/vendor/autoload ``` ### Non-Windows with PREFIX set ```nushell ❯ PREFIX=/usr/bob cargo r ❯ $nu.vendor-autoload-dir /usr/bob/share/nushell/vendor/autoload ``` ### Non-Windows with NU_VENDOR_AUTOLOAD_DIR set ```nushell ❯ NU_VENDOR_AUTOLOAD_DIR=/some/other/path/nushell/stuff cargo r ❯ $nu.vendor-autoload-dir /some/other/path/nushell/stuff ``` > [!IMPORTANT] To be clear, this PR does not do the auto-loading, it just sets up the folder to support that functionality that can be added in a later PR. The PR also does not create the folder defined. It's just setting the $nu constant.   # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-06-20 11:30:43 +00:00
);
2023-09-01 06:18:55 +00:00
record.push("temp-path", {
let canon_temp_path = canonicalize_path(engine_state, &std::env::temp_dir());
Value::string(canon_temp_path.to_string_lossy(), span)
});
record.push("pid", Value::int(std::process::id().into(), span));
record.push("os-info", {
let ver = get_kernel_version();
Value::record(
record! {
"name" => Value::string(get_os_name(), span),
"arch" => Value::string(get_os_arch(), span),
"family" => Value::string(get_os_family(), span),
"kernel_version" => Value::string(ver, span),
},
span,
)
});
record.push(
"startup-time",
Value::duration(engine_state.get_startup_time(), span),
);
record.push(
"is-interactive",
Value::bool(engine_state.is_interactive, span),
);
record.push("is-login", Value::bool(engine_state.is_login, span));
record.push(
"history-enabled",
Value::bool(engine_state.history_enabled, span),
);
2023-09-01 06:18:55 +00:00
record.push(
"current-exe",
if let Ok(current_exe) = std::env::current_exe() {
Value::string(current_exe.to_string_lossy(), span)
} else {
Value::error(
ShellError::IOError {
msg: "Could not get current executable path".to_string(),
},
2023-09-01 06:18:55 +00:00
span,
)
},
);
Value::record(record, span)
2023-09-01 06:18:55 +00:00
}
pub fn get_vendor_autoload_dirs(_engine_state: &EngineState) -> Vec<PathBuf> {
// load order for autoload dirs
// /Library/Application Support/nushell/vendor/autoload on macOS
// <dir>/nushell/vendor/autoload for every dir in XDG_DATA_DIRS in reverse order on platforms other than windows. If XDG_DATA_DIRS is not set, it falls back to <PREFIX>/share if PREFIX ends in local, or <PREFIX>/local/share:<PREFIX>/share otherwise. If PREFIX is not set, fall back to /usr/local/share:/usr/share.
// %ProgramData%\nushell\vendor\autoload on windows
// NU_VENDOR_AUTOLOAD_DIR from compile time, if env var is set at compile time
// if on macOS, additionally check XDG_DATA_HOME, which `dirs` is only doing on Linux
// <data_dir>/nushell/vendor/autoload of the current user according to the `dirs` crate
// NU_VENDOR_AUTOLOAD_DIR at runtime, if env var is set
let into_autoload_path_fn = |mut path: PathBuf| {
path.push("nushell");
path.push("vendor");
path.push("autoload");
path
};
let mut dirs = Vec::new();
let mut append_fn = |path: PathBuf| {
if !dirs.contains(&path) {
dirs.push(path)
}
};
#[cfg(target_os = "macos")]
std::iter::once("/Library/Application Support")
.map(PathBuf::from)
.map(into_autoload_path_fn)
.for_each(&mut append_fn);
#[cfg(unix)]
{
use std::os::unix::ffi::OsStrExt;
std::env::var_os("XDG_DATA_DIRS")
.or_else(|| {
option_env!("PREFIX").map(|prefix| {
if prefix.ends_with("local") {
std::ffi::OsString::from(format!("{prefix}/share"))
} else {
std::ffi::OsString::from(format!("{prefix}/local/share:{prefix}/share"))
}
})
implement autoloading (#13217) # Description This PR implements script or module autoloading. It does this by finding the `$nu.vendor-autoload-dir`, lists the contents and sorts them by file name. These files are evaluated in that order. To see what's going on, you can use `--log-level warn` ``` ❯ cargo r -- --log-level warn Finished dev [unoptimized + debuginfo] target(s) in 0.58s Running `target\debug\nu.exe --log-level warn` 2024-06-24 09:23:20.494 PM [WARN ] nu::config_files: set_config_path() cwd: "C:\\Users\\fdncred\\source\\repos\\nushell", default_config: config.nu, key: config-path, config_file_specified: None 2024-06-24 09:23:20.495 PM [WARN ] nu::config_files: set_config_path() cwd: "C:\\Users\\fdncred\\source\\repos\\nushell", default_config: env.nu, key: env-path, config_file_specified: None 2024-06-24 09:23:20.629 PM [WARN ] nu::config_files: setup_config() config_file_specified: None, env_file_specified: None, login: false 2024-06-24 09:23:20.660 PM [WARN ] nu::config_files: read_config_file() config_file_specified: None, is_env_config: true Hello, from env.nu 2024-06-24 09:23:20.679 PM [WARN ] nu::config_files: read_config_file() config_file_specified: None, is_env_config: false Hello, from config.nu Hello, from defs.nu Activating Microsoft Visual Studio environment. 2024-06-24 09:23:21.398 PM [WARN ] nu::config_files: read_vendor_autoload_files() src\config_files.rs:234:9 2024-06-24 09:23:21.399 PM [WARN ] nu::config_files: read_vendor_autoload_files: C:\ProgramData\nushell\vendor\autoload 2024-06-24 09:23:21.399 PM [WARN ] nu::config_files: AutoLoading: "C:\\ProgramData\\nushell\\vendor\\autoload\\01_get-weather.nu" 2024-06-24 09:23:21.675 PM [WARN ] nu::config_files: AutoLoading: "C:\\ProgramData\\nushell\\vendor\\autoload\\02_temp.nu" 2024-06-24 09:23:21.817 PM [WARN ] nu_cli::repl: Terminal doesn't support use_kitty_protocol config ``` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-06-26 01:31:54 +00:00
})
.unwrap_or_else(|| std::ffi::OsString::from("/usr/local/share/:/usr/share/"))
.as_encoded_bytes()
.split(|b| *b == b':')
.map(|split| into_autoload_path_fn(PathBuf::from(std::ffi::OsStr::from_bytes(split))))
.rev()
.for_each(&mut append_fn);
}
#[cfg(target_os = "windows")]
dirs_sys::known_folder(windows_sys::Win32::UI::Shell::FOLDERID_ProgramData)
.into_iter()
.map(into_autoload_path_fn)
.for_each(&mut append_fn);
option_env!("NU_VENDOR_AUTOLOAD_DIR")
.into_iter()
.map(PathBuf::from)
.for_each(&mut append_fn);
#[cfg(target_os = "macos")]
std::env::var("XDG_DATA_HOME")
.ok()
.map(PathBuf::from)
.or_else(|| {
dirs::home_dir().map(|mut home| {
home.push(".local");
home.push("share");
home
})
})
.map(into_autoload_path_fn)
.into_iter()
.for_each(&mut append_fn);
dirs::data_dir()
.into_iter()
.map(into_autoload_path_fn)
.for_each(&mut append_fn);
std::env::var_os("NU_VENDOR_AUTOLOAD_DIR")
.into_iter()
.map(PathBuf::from)
.for_each(&mut append_fn);
dirs
implement autoloading (#13217) # Description This PR implements script or module autoloading. It does this by finding the `$nu.vendor-autoload-dir`, lists the contents and sorts them by file name. These files are evaluated in that order. To see what's going on, you can use `--log-level warn` ``` ❯ cargo r -- --log-level warn Finished dev [unoptimized + debuginfo] target(s) in 0.58s Running `target\debug\nu.exe --log-level warn` 2024-06-24 09:23:20.494 PM [WARN ] nu::config_files: set_config_path() cwd: "C:\\Users\\fdncred\\source\\repos\\nushell", default_config: config.nu, key: config-path, config_file_specified: None 2024-06-24 09:23:20.495 PM [WARN ] nu::config_files: set_config_path() cwd: "C:\\Users\\fdncred\\source\\repos\\nushell", default_config: env.nu, key: env-path, config_file_specified: None 2024-06-24 09:23:20.629 PM [WARN ] nu::config_files: setup_config() config_file_specified: None, env_file_specified: None, login: false 2024-06-24 09:23:20.660 PM [WARN ] nu::config_files: read_config_file() config_file_specified: None, is_env_config: true Hello, from env.nu 2024-06-24 09:23:20.679 PM [WARN ] nu::config_files: read_config_file() config_file_specified: None, is_env_config: false Hello, from config.nu Hello, from defs.nu Activating Microsoft Visual Studio environment. 2024-06-24 09:23:21.398 PM [WARN ] nu::config_files: read_vendor_autoload_files() src\config_files.rs:234:9 2024-06-24 09:23:21.399 PM [WARN ] nu::config_files: read_vendor_autoload_files: C:\ProgramData\nushell\vendor\autoload 2024-06-24 09:23:21.399 PM [WARN ] nu::config_files: AutoLoading: "C:\\ProgramData\\nushell\\vendor\\autoload\\01_get-weather.nu" 2024-06-24 09:23:21.675 PM [WARN ] nu::config_files: AutoLoading: "C:\\ProgramData\\nushell\\vendor\\autoload\\02_temp.nu" 2024-06-24 09:23:21.817 PM [WARN ] nu_cli::repl: Terminal doesn't support use_kitty_protocol config ``` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-06-26 01:31:54 +00:00
}
fn eval_const_call(
working_set: &StateWorkingSet,
call: &Call,
input: PipelineData,
) -> Result<PipelineData, ShellError> {
let decl = working_set.get_decl(call.decl_id);
if !decl.is_const() {
return Err(ShellError::NotAConstCommand { span: call.head });
}
Revert "Span ID Refactor (Step 2): Make Call SpanId-friendly (#13268)" (#13292) This reverts commit 0cfd5fbece6f25b54ab9dc417a9e06af9d83f282. The original PR messed up syntax higlighting of aliases and causes panics of completion in the presence of alias. <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-07-03 21:02:13 +00:00
if !decl.is_known_external() && call.named_iter().any(|(flag, _, _)| flag.item == "help") {
// It would require re-implementing get_full_help() for const evaluation. Assuming that
// getting help messages at parse-time is rare enough, we can simply disallow it.
return Err(ShellError::NotAConstHelp { span: call.head });
}
Internal representation (IR) compiler and evaluator (#13330) # Description This PR adds an internal representation language to Nushell, offering an alternative evaluator based on simple instructions, stream-containing registers, and indexed control flow. The number of registers required is determined statically at compile-time, and the fixed size required is allocated upon entering the block. Each instruction is associated with a span, which makes going backwards from IR instructions to source code very easy. Motivations for IR: 1. **Performance.** By simplifying the evaluation path and making it more cache-friendly and branch predictor-friendly, code that does a lot of computation in Nushell itself can be sped up a decent bit. Because the IR is fairly easy to reason about, we can also implement optimization passes in the future to eliminate and simplify code. 2. **Correctness.** The instructions mostly have very simple and easily-specified behavior, so hopefully engine changes are a little bit easier to reason about, and they can be specified in a more formal way at some point. I have made an effort to document each of the instructions in the docs for the enum itself in a reasonably specific way. Some of the errors that would have happened during evaluation before are now moved to the compilation step instead, because they don't make sense to check during evaluation. 3. **As an intermediate target.** This is a good step for us to bring the [`new-nu-parser`](https://github.com/nushell/new-nu-parser) in at some point, as code generated from new AST can be directly compared to code generated from old AST. If the IR code is functionally equivalent, it will behave the exact same way. 4. **Debugging.** With a little bit more work, we can probably give control over advancing the virtual machine that `IrBlock`s run on to some sort of external driver, making things like breakpoints and single stepping possible. Tools like `view ir` and [`explore ir`](https://github.com/devyn/nu_plugin_explore_ir) make it easier than before to see what exactly is going on with your Nushell code. The goal is to eventually replace the AST evaluator entirely, once we're sure it's working just as well. You can help dogfood this by running Nushell with `$env.NU_USE_IR` set to some value. The environment variable is checked when Nushell starts, so config runs with IR, or it can also be set on a line at the REPL to change it dynamically. It is also checked when running `do` in case within a script you want to just run a specific piece of code with or without IR. # Example ```nushell view ir { |data| mut sum = 0 for n in $data { $sum += $n } $sum } ``` ```gas # 3 registers, 19 instructions, 0 bytes of data 0: load-literal %0, int(0) 1: store-variable var 904, %0 # let 2: drain %0 3: drop %0 4: load-variable %1, var 903 5: iterate %0, %1, end 15 # for, label(1), from(14:) 6: store-variable var 905, %0 7: load-variable %0, var 904 8: load-variable %2, var 905 9: binary-op %0, Math(Plus), %2 10: span %0 11: store-variable var 904, %0 12: load-literal %0, nothing 13: drain %0 14: jump 5 15: drop %0 # label(0), from(5:) 16: drain %0 17: load-variable %0, var 904 18: return %0 ``` # Benchmarks All benchmarks run on a base model Mac Mini M1. ## Iterative Fibonacci sequence This is about as best case as possible, making use of the much faster control flow. Most code will not experience a speed improvement nearly this large. ```nushell def fib [n: int] { mut a = 0 mut b = 1 for _ in 2..=$n { let c = $a + $b $a = $b $b = $c } $b } use std bench bench { 0..50 | each { |n| fib $n } } ``` IR disabled: ``` ╭───────┬─────────────────╮ │ mean │ 1ms 924µs 665ns │ │ min │ 1ms 700µs 83ns │ │ max │ 3ms 450µs 125ns │ │ std │ 395µs 759ns │ │ times │ [list 50 items] │ ╰───────┴─────────────────╯ ``` IR enabled: ``` ╭───────┬─────────────────╮ │ mean │ 452µs 820ns │ │ min │ 427µs 417ns │ │ max │ 540µs 167ns │ │ std │ 17µs 158ns │ │ times │ [list 50 items] │ ╰───────┴─────────────────╯ ``` ![explore ir view](https://github.com/nushell/nushell/assets/10729/d7bccc03-5222-461c-9200-0dce71b83b83) ## [gradient_benchmark_no_check.nu](https://github.com/nushell/nu_scripts/blob/main/benchmarks/gradient_benchmark_no_check.nu) IR disabled: ``` ╭───┬──────────────────╮ │ 0 │ 27ms 929µs 958ns │ │ 1 │ 21ms 153µs 459ns │ │ 2 │ 18ms 639µs 666ns │ │ 3 │ 19ms 554µs 583ns │ │ 4 │ 13ms 383µs 375ns │ │ 5 │ 11ms 328µs 208ns │ │ 6 │ 5ms 659µs 542ns │ ╰───┴──────────────────╯ ``` IR enabled: ``` ╭───┬──────────────────╮ │ 0 │ 22ms 662µs │ │ 1 │ 17ms 221µs 792ns │ │ 2 │ 14ms 786µs 708ns │ │ 3 │ 13ms 876µs 834ns │ │ 4 │ 13ms 52µs 875ns │ │ 5 │ 11ms 269µs 666ns │ │ 6 │ 6ms 942µs 500ns │ ╰───┴──────────────────╯ ``` ## [random-bytes.nu](https://github.com/nushell/nu_scripts/blob/main/benchmarks/random-bytes.nu) I got pretty random results out of this benchmark so I decided not to include it. Not clear why. # User-Facing Changes - IR compilation errors may appear even if the user isn't evaluating with IR. - IR evaluation can be enabled by setting the `NU_USE_IR` environment variable to any value. - New command `view ir` pretty-prints the IR for a block, and `view ir --json` can be piped into an external tool like [`explore ir`](https://github.com/devyn/nu_plugin_explore_ir). # Tests + Formatting All tests are passing with `NU_USE_IR=1`, and I've added some more eval tests to compare the results for some very core operations. I will probably want to add some more so we don't have to always check `NU_USE_IR=1 toolkit test --workspace` on a regular basis. # After Submitting - [ ] release notes - [ ] further documentation of instructions? - [ ] post-release: publish `nu_plugin_explore_ir`
2024-07-11 00:33:59 +00:00
decl.run_const(working_set, &call.into(), input)
}
pub fn eval_const_subexpression(
working_set: &StateWorkingSet,
block: &Block,
mut input: PipelineData,
span: Span,
) -> Result<PipelineData, ShellError> {
for pipeline in block.pipelines.iter() {
for element in pipeline.elements.iter() {
IO and redirection overhaul (#11934) # Description The PR overhauls how IO redirection is handled, allowing more explicit and fine-grain control over `stdout` and `stderr` output as well as more efficient IO and piping. To summarize the changes in this PR: - Added a new `IoStream` type to indicate the intended destination for a pipeline element's `stdout` and `stderr`. - The `stdout` and `stderr` `IoStream`s are stored in the `Stack` and to avoid adding 6 additional arguments to every eval function and `Command::run`. The `stdout` and `stderr` streams can be temporarily overwritten through functions on `Stack` and these functions will return a guard that restores the original `stdout` and `stderr` when dropped. - In the AST, redirections are now directly part of a `PipelineElement` as a `Option<Redirection>` field instead of having multiple different `PipelineElement` enum variants for each kind of redirection. This required changes to the parser, mainly in `lite_parser.rs`. - `Command`s can also set a `IoStream` override/redirection which will apply to the previous command in the pipeline. This is used, for example, in `ignore` to allow the previous external command to have its stdout redirected to `Stdio::null()` at spawn time. In contrast, the current implementation has to create an os pipe and manually consume the output on nushell's side. File and pipe redirections (`o>`, `e>`, `e>|`, etc.) have precedence over overrides from commands. This PR improves piping and IO speed, partially addressing #10763. Using the `throughput` command from that issue, this PR gives the following speedup on my setup for the commands below: | Command | Before (MB/s) | After (MB/s) | Bash (MB/s) | | --------------------------- | -------------:| ------------:| -----------:| | `throughput o> /dev/null` | 1169 | 52938 | 54305 | | `throughput \| ignore` | 840 | 55438 | N/A | | `throughput \| null` | Error | 53617 | N/A | | `throughput \| rg 'x'` | 1165 | 3049 | 3736 | | `(throughput) \| rg 'x'` | 810 | 3085 | 3815 | (Numbers above are the median samples for throughput) This PR also paves the way to refactor our `ExternalStream` handling in the various commands. For example, this PR already fixes the following code: ```nushell ^sh -c 'echo -n "hello "; sleep 0; echo "world"' | find "hello world" ``` This returns an empty list on 0.90.1 and returns a highlighted "hello world" on this PR. Since the `stdout` and `stderr` `IoStream`s are available to commands when they are run, then this unlocks the potential for more convenient behavior. E.g., the `find` command can disable its ansi highlighting if it detects that the output `IoStream` is not the terminal. Knowing the output streams will also allow background job output to be redirected more easily and efficiently. # User-Facing Changes - External commands returned from closures will be collected (in most cases): ```nushell 1..2 | each {|_| nu -c "print a" } ``` This gives `["a", "a"]` on this PR, whereas this used to print "a\na\n" and then return an empty list. ```nushell 1..2 | each {|_| nu -c "print -e a" } ``` This gives `["", ""]` and prints "a\na\n" to stderr, whereas this used to return an empty list and print "a\na\n" to stderr. - Trailing new lines are always trimmed for external commands when piping into internal commands or collecting it as a value. (Failure to decode the output as utf-8 will keep the trailing newline for the last binary value.) In the current nushell version, the following three code snippets differ only in parenthesis placement, but they all also have different outputs: 1. `1..2 | each { ^echo a }` ``` a a ╭────────────╮ │ empty list │ ╰────────────╯ ``` 2. `1..2 | each { (^echo a) }` ``` ╭───┬───╮ │ 0 │ a │ │ 1 │ a │ ╰───┴───╯ ``` 3. `1..2 | (each { ^echo a })` ``` ╭───┬───╮ │ 0 │ a │ │ │ │ │ 1 │ a │ │ │ │ ╰───┴───╯ ``` But in this PR, the above snippets will all have the same output: ``` ╭───┬───╮ │ 0 │ a │ │ 1 │ a │ ╰───┴───╯ ``` - All existing flags on `run-external` are now deprecated. - File redirections now apply to all commands inside a code block: ```nushell (nu -c "print -e a"; nu -c "print -e b") e> test.out ``` This gives "a\nb\n" in `test.out` and prints nothing. The same result would happen when printing to stdout and using a `o>` file redirection. - External command output will (almost) never be ignored, and ignoring output must be explicit now: ```nushell (^echo a; ^echo b) ``` This prints "a\nb\n", whereas this used to print only "b\n". This only applies to external commands; values and internal commands not in return position will not print anything (e.g., `(echo a; echo b)` still only prints "b"). - `complete` now always captures stderr (`do` is not necessary). # After Submitting The language guide and other documentation will need to be updated.
2024-03-14 20:51:55 +00:00
if element.redirection.is_some() {
return Err(ShellError::NotAConstant { span });
IO and redirection overhaul (#11934) # Description The PR overhauls how IO redirection is handled, allowing more explicit and fine-grain control over `stdout` and `stderr` output as well as more efficient IO and piping. To summarize the changes in this PR: - Added a new `IoStream` type to indicate the intended destination for a pipeline element's `stdout` and `stderr`. - The `stdout` and `stderr` `IoStream`s are stored in the `Stack` and to avoid adding 6 additional arguments to every eval function and `Command::run`. The `stdout` and `stderr` streams can be temporarily overwritten through functions on `Stack` and these functions will return a guard that restores the original `stdout` and `stderr` when dropped. - In the AST, redirections are now directly part of a `PipelineElement` as a `Option<Redirection>` field instead of having multiple different `PipelineElement` enum variants for each kind of redirection. This required changes to the parser, mainly in `lite_parser.rs`. - `Command`s can also set a `IoStream` override/redirection which will apply to the previous command in the pipeline. This is used, for example, in `ignore` to allow the previous external command to have its stdout redirected to `Stdio::null()` at spawn time. In contrast, the current implementation has to create an os pipe and manually consume the output on nushell's side. File and pipe redirections (`o>`, `e>`, `e>|`, etc.) have precedence over overrides from commands. This PR improves piping and IO speed, partially addressing #10763. Using the `throughput` command from that issue, this PR gives the following speedup on my setup for the commands below: | Command | Before (MB/s) | After (MB/s) | Bash (MB/s) | | --------------------------- | -------------:| ------------:| -----------:| | `throughput o> /dev/null` | 1169 | 52938 | 54305 | | `throughput \| ignore` | 840 | 55438 | N/A | | `throughput \| null` | Error | 53617 | N/A | | `throughput \| rg 'x'` | 1165 | 3049 | 3736 | | `(throughput) \| rg 'x'` | 810 | 3085 | 3815 | (Numbers above are the median samples for throughput) This PR also paves the way to refactor our `ExternalStream` handling in the various commands. For example, this PR already fixes the following code: ```nushell ^sh -c 'echo -n "hello "; sleep 0; echo "world"' | find "hello world" ``` This returns an empty list on 0.90.1 and returns a highlighted "hello world" on this PR. Since the `stdout` and `stderr` `IoStream`s are available to commands when they are run, then this unlocks the potential for more convenient behavior. E.g., the `find` command can disable its ansi highlighting if it detects that the output `IoStream` is not the terminal. Knowing the output streams will also allow background job output to be redirected more easily and efficiently. # User-Facing Changes - External commands returned from closures will be collected (in most cases): ```nushell 1..2 | each {|_| nu -c "print a" } ``` This gives `["a", "a"]` on this PR, whereas this used to print "a\na\n" and then return an empty list. ```nushell 1..2 | each {|_| nu -c "print -e a" } ``` This gives `["", ""]` and prints "a\na\n" to stderr, whereas this used to return an empty list and print "a\na\n" to stderr. - Trailing new lines are always trimmed for external commands when piping into internal commands or collecting it as a value. (Failure to decode the output as utf-8 will keep the trailing newline for the last binary value.) In the current nushell version, the following three code snippets differ only in parenthesis placement, but they all also have different outputs: 1. `1..2 | each { ^echo a }` ``` a a ╭────────────╮ │ empty list │ ╰────────────╯ ``` 2. `1..2 | each { (^echo a) }` ``` ╭───┬───╮ │ 0 │ a │ │ 1 │ a │ ╰───┴───╯ ``` 3. `1..2 | (each { ^echo a })` ``` ╭───┬───╮ │ 0 │ a │ │ │ │ │ 1 │ a │ │ │ │ ╰───┴───╯ ``` But in this PR, the above snippets will all have the same output: ``` ╭───┬───╮ │ 0 │ a │ │ 1 │ a │ ╰───┴───╯ ``` - All existing flags on `run-external` are now deprecated. - File redirections now apply to all commands inside a code block: ```nushell (nu -c "print -e a"; nu -c "print -e b") e> test.out ``` This gives "a\nb\n" in `test.out` and prints nothing. The same result would happen when printing to stdout and using a `o>` file redirection. - External command output will (almost) never be ignored, and ignoring output must be explicit now: ```nushell (^echo a; ^echo b) ``` This prints "a\nb\n", whereas this used to print only "b\n". This only applies to external commands; values and internal commands not in return position will not print anything (e.g., `(echo a; echo b)` still only prints "b"). - `complete` now always captures stderr (`do` is not necessary). # After Submitting The language guide and other documentation will need to be updated.
2024-03-14 20:51:55 +00:00
}
IO and redirection overhaul (#11934) # Description The PR overhauls how IO redirection is handled, allowing more explicit and fine-grain control over `stdout` and `stderr` output as well as more efficient IO and piping. To summarize the changes in this PR: - Added a new `IoStream` type to indicate the intended destination for a pipeline element's `stdout` and `stderr`. - The `stdout` and `stderr` `IoStream`s are stored in the `Stack` and to avoid adding 6 additional arguments to every eval function and `Command::run`. The `stdout` and `stderr` streams can be temporarily overwritten through functions on `Stack` and these functions will return a guard that restores the original `stdout` and `stderr` when dropped. - In the AST, redirections are now directly part of a `PipelineElement` as a `Option<Redirection>` field instead of having multiple different `PipelineElement` enum variants for each kind of redirection. This required changes to the parser, mainly in `lite_parser.rs`. - `Command`s can also set a `IoStream` override/redirection which will apply to the previous command in the pipeline. This is used, for example, in `ignore` to allow the previous external command to have its stdout redirected to `Stdio::null()` at spawn time. In contrast, the current implementation has to create an os pipe and manually consume the output on nushell's side. File and pipe redirections (`o>`, `e>`, `e>|`, etc.) have precedence over overrides from commands. This PR improves piping and IO speed, partially addressing #10763. Using the `throughput` command from that issue, this PR gives the following speedup on my setup for the commands below: | Command | Before (MB/s) | After (MB/s) | Bash (MB/s) | | --------------------------- | -------------:| ------------:| -----------:| | `throughput o> /dev/null` | 1169 | 52938 | 54305 | | `throughput \| ignore` | 840 | 55438 | N/A | | `throughput \| null` | Error | 53617 | N/A | | `throughput \| rg 'x'` | 1165 | 3049 | 3736 | | `(throughput) \| rg 'x'` | 810 | 3085 | 3815 | (Numbers above are the median samples for throughput) This PR also paves the way to refactor our `ExternalStream` handling in the various commands. For example, this PR already fixes the following code: ```nushell ^sh -c 'echo -n "hello "; sleep 0; echo "world"' | find "hello world" ``` This returns an empty list on 0.90.1 and returns a highlighted "hello world" on this PR. Since the `stdout` and `stderr` `IoStream`s are available to commands when they are run, then this unlocks the potential for more convenient behavior. E.g., the `find` command can disable its ansi highlighting if it detects that the output `IoStream` is not the terminal. Knowing the output streams will also allow background job output to be redirected more easily and efficiently. # User-Facing Changes - External commands returned from closures will be collected (in most cases): ```nushell 1..2 | each {|_| nu -c "print a" } ``` This gives `["a", "a"]` on this PR, whereas this used to print "a\na\n" and then return an empty list. ```nushell 1..2 | each {|_| nu -c "print -e a" } ``` This gives `["", ""]` and prints "a\na\n" to stderr, whereas this used to return an empty list and print "a\na\n" to stderr. - Trailing new lines are always trimmed for external commands when piping into internal commands or collecting it as a value. (Failure to decode the output as utf-8 will keep the trailing newline for the last binary value.) In the current nushell version, the following three code snippets differ only in parenthesis placement, but they all also have different outputs: 1. `1..2 | each { ^echo a }` ``` a a ╭────────────╮ │ empty list │ ╰────────────╯ ``` 2. `1..2 | each { (^echo a) }` ``` ╭───┬───╮ │ 0 │ a │ │ 1 │ a │ ╰───┴───╯ ``` 3. `1..2 | (each { ^echo a })` ``` ╭───┬───╮ │ 0 │ a │ │ │ │ │ 1 │ a │ │ │ │ ╰───┴───╯ ``` But in this PR, the above snippets will all have the same output: ``` ╭───┬───╮ │ 0 │ a │ │ 1 │ a │ ╰───┴───╯ ``` - All existing flags on `run-external` are now deprecated. - File redirections now apply to all commands inside a code block: ```nushell (nu -c "print -e a"; nu -c "print -e b") e> test.out ``` This gives "a\nb\n" in `test.out` and prints nothing. The same result would happen when printing to stdout and using a `o>` file redirection. - External command output will (almost) never be ignored, and ignoring output must be explicit now: ```nushell (^echo a; ^echo b) ``` This prints "a\nb\n", whereas this used to print only "b\n". This only applies to external commands; values and internal commands not in return position will not print anything (e.g., `(echo a; echo b)` still only prints "b"). - `complete` now always captures stderr (`do` is not necessary). # After Submitting The language guide and other documentation will need to be updated.
2024-03-14 20:51:55 +00:00
input = eval_constant_with_input(working_set, &element.expr, input)?
}
}
Ok(input)
}
pub fn eval_constant_with_input(
working_set: &StateWorkingSet,
expr: &Expression,
input: PipelineData,
) -> Result<PipelineData, ShellError> {
match &expr.expr {
Expr::Call(call) => eval_const_call(working_set, call, input),
Expr::Subexpression(block_id) => {
let block = working_set.get_block(*block_id);
Span ID Refactor (Step 2): Use SpanId of expressions in some places (#13102) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Part of https://github.com/nushell/nushell/issues/12963, step 2. This PR refactors changes the use of `expression.span` to `expression.span_id` via a new helper `Expression::span()`. A new `GetSpan` is added to abstract getting the span from both `EngineState` and `StateWorkingSet`. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> `format pattern` loses the ability to use variables in the pattern, e.g., `... | format pattern 'value of {$it.name} is {$it.value}'`. This is because the command did a custom parse-eval cycle, creating spans that are not merged into the main engine state. We could clone the engine state, add Clone trait to StateDelta and merge the cloned delta to the cloned state, but IMO there is not much value from having this ability, since we have string interpolation nowadays: `... | $"value of ($in.name) is ($in.value)"`. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-06-09 09:15:53 +00:00
eval_const_subexpression(working_set, block, input, expr.span(&working_set))
}
_ => eval_constant(working_set, expr).map(|v| PipelineData::Value(v, None)),
}
}
/// Evaluate a constant value at parse time
pub fn eval_constant(
working_set: &StateWorkingSet,
expr: &Expression,
) -> Result<Value, ShellError> {
Debugger experiments (#11441) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR adds a new evaluator path with callbacks to a mutable trait object implementing a Debugger trait. The trait object can do anything, e.g., profiling, code coverage, step debugging. Currently, entering/leaving a block and a pipeline element is marked with callbacks, but more callbacks can be added as necessary. Not all callbacks need to be used by all debuggers; unused ones are simply empty calls. A simple profiler is implemented as a proof of concept. The debugging support is implementing by making `eval_xxx()` functions generic depending on whether we're debugging or not. This has zero computational overhead, but makes the binary slightly larger (see benchmarks below). `eval_xxx()` variants called from commands (like `eval_block_with_early_return()` in `each`) are chosen with a dynamic dispatch for two reasons: to not grow the binary size due to duplicating the code of many commands, and for the fact that it isn't possible because it would make Command trait objects object-unsafe. In the future, I hope it will be possible to allow plugin callbacks such that users would be able to implement their profiler plugins instead of having to recompile Nushell. [DAP](https://microsoft.github.io/debug-adapter-protocol/) would also be interesting to explore. Try `help debug profile`. ## Screenshots Basic output: ![profiler_new](https://github.com/nushell/nushell/assets/25571562/418b9df0-b659-4dcb-b023-2d5fcef2c865) To profile with more granularity, increase the profiler depth (you'll see that repeated `is-windows` calls take a large chunk of total time, making it a good candidate for optimizing): ![profiler_new_m3](https://github.com/nushell/nushell/assets/25571562/636d756d-5d56-460c-a372-14716f65f37f) ## Benchmarks ### Binary size Binary size increase vs. main: **+40360 bytes**. _(Both built with `--release --features=extra,dataframe`.)_ ### Time ```nushell # bench_debug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'debug:' let res2 = bench { debug profile $test } --pretty print $res2 ``` ```nushell # bench_nodebug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'no debug:' let res1 = bench { do $test } --pretty print $res1 ``` `cargo run --release -- bench_debug.nu` is consistently 1--2 ms slower than `cargo run --release -- bench_nodebug.nu` due to the collection overhead + gathering the report. This is expected. When gathering more stuff, the overhead is obviously higher. `cargo run --release -- bench_nodebug.nu` vs. `nu bench_nodebug.nu` I didn't measure any difference. Both benchmarks report times between 97 and 103 ms randomly, without one being consistently higher than the other. This suggests that at least in this particular case, when not running any debugger, there is no runtime overhead. ## API changes This PR adds a generic parameter to all `eval_xxx` functions that forces you to specify whether you use the debugger. You can resolve it in two ways: * Use a provided helper that will figure it out for you. If you wanted to use `eval_block(&engine_state, ...)`, call `let eval_block = get_eval_block(&engine_state); eval_block(&engine_state, ...)` * If you know you're in an evaluation path that doesn't need debugger support, call `eval_block::<WithoutDebug>(&engine_state, ...)` (this is the case of hooks, for example). I tried to add more explanation in the docstring of `debugger_trait.rs`. ## TODO - [x] Better profiler output to reduce spam of iterative commands like `each` - [x] Resolve `TODO: DEBUG` comments - [x] Resolve unwraps - [x] Add doc comments - [x] Add usage and extra usage for `debug profile`, explaining all columns # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Hopefully none. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-03-08 18:21:35 +00:00
// TODO: Allow debugging const eval
<EvalConst as Eval>::eval::<WithoutDebug>(working_set, &mut (), expr)
}
struct EvalConst;
impl Eval for EvalConst {
type State<'a> = &'a StateWorkingSet<'a>;
type MutState = ();
fn get_config(state: Self::State<'_>, _: &mut ()) -> Arc<Config> {
state.get_config().clone()
Evaluate string interpolation at parse time (#11562) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> Closes #11561 # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR will allow string interpolation at parse time. Since the actual config hasn't been loaded at parse time, this uses the `get_config()` method on `StateWorkingSet`. So file sizes and datetimes (I think those are the only things whose string representations depend on the config) may be formatted differently from how users have configured things, which may come as a surprise to some. It does seem unlikely that anyone would be formatting file sizes or date times at parse time. Still, something to think about if/before this PR merged. Also, I changed the `ModuleNotFound` error to include the name of the module. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Users will be able to do stuff like: ```nu const x = [1 2 3] const y = $"foo($x)" // foo[1, 2, 3] ``` The main use case is `use`-ing and `source`-ing files at parse time: ```nu const file = "foo.nu" use $"($file)" ``` If the module isn't found, you'll see an error like this: ``` Error: nu::parser::module_not_found × Module not found. ╭─[entry #3:1:1] 1 │ use $"($file)" · ─────┬──── · ╰── module foo.nu not found ╰──── help: module files and their paths must be available before your script is run as parsing occurs before anything is evaluated ``` # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> Although there's user-facing changes, there's probably no need to change the docs since people probably already expect string interpolation to work at parse time. Edit: @kubouch pointed out that we'd need to document the fact that stuff like file sizes and datetimes won't get formatted according to user's runtime configs, so I'll make a PR to nushell.github.io after this one
2024-01-22 07:13:48 +00:00
}
fn eval_filepath(
_: &StateWorkingSet,
_: &mut (),
path: String,
do not attempt to glob expand if the file path is wrapped in quotes (#11569) # Description Fixes: #11455 ### For arguments which is annotated with `:path/:directory/:glob` To fix the issue, we need to have a way to know if a path is originally quoted during runtime. So the information needed to be added at several levels: * parse time (from user input to expression) We need to add quoted information into `Expr::Filepath`, `Expr::Directory`, `Expr::GlobPattern` * eval time When convert from `Expr::Filepath`, `Expr::Directory`, `Expr::GlobPattern` to `Value::String` during runtime, we won't auto expanded the path if it's quoted ### For `ls` It's really special, because it accepts a `String` as a pattern, and it generates `glob` expression inside the command itself. So the idea behind the change is introducing a special SyntaxShape to ls: `SyntaxShape::LsGlobPattern`. So we can track if the pattern is originally quoted easier, and we don't auto expand the path either. Then when constructing a glob pattern inside ls, we check if input pattern is quoted, if so: we escape the input pattern, so we can run `ls a[123]b`, because it's already escaped. Finally, to accomplish the checking process, we also need to introduce a new value type called `Value::QuotedString` to differ from `Value::String`, it's used to generate an enum called `NuPath`, which is finally used in `ls` function. `ls` learned from `NuPath` to know if user input is quoted. # User-Facing Changes Actually it contains several changes ### For arguments which is annotated with `:path/:directory/:glob` #### Before ```nushell > def foo [p: path] { echo $p }; print (foo "~/a"); print (foo '~/a') /home/windsoilder/a /home/windsoilder/a > def foo [p: directory] { echo $p }; print (foo "~/a"); print (foo '~/a') /home/windsoilder/a /home/windsoilder/a > def foo [p: glob] { echo $p }; print (foo "~/a"); print (foo '~/a') /home/windsoilder/a /home/windsoilder/a ``` #### After ```nushell > def foo [p: path] { echo $p }; print (foo "~/a"); print (foo '~/a') ~/a ~/a > def foo [p: directory] { echo $p }; print (foo "~/a"); print (foo '~/a') ~/a ~/a > def foo [p: glob] { echo $p }; print (foo "~/a"); print (foo '~/a') ~/a ~/a ``` ### For ls command `touch '[uwu]'` #### Before ``` ❯ ls -D "[uwu]" Error: × No matches found for [uwu] ╭─[entry #6:1:1] 1 │ ls -D "[uwu]" · ───┬─── · ╰── Pattern, file or folder not found ╰──── help: no matches found ``` #### After ``` ❯ ls -D "[uwu]" ╭───┬───────┬──────┬──────┬──────────╮ │ # │ name │ type │ size │ modified │ ├───┼───────┼──────┼──────┼──────────┤ │ 0 │ [uwu] │ file │ 0 B │ now │ ╰───┴───────┴──────┴──────┴──────────╯ ``` # Tests + Formatting Done # After Submitting NaN
2024-01-21 15:22:25 +00:00
_: bool,
span: Span,
) -> Result<Value, ShellError> {
Ok(Value::string(path, span))
}
fn eval_directory(
_: &StateWorkingSet,
_: &mut (),
_: String,
do not attempt to glob expand if the file path is wrapped in quotes (#11569) # Description Fixes: #11455 ### For arguments which is annotated with `:path/:directory/:glob` To fix the issue, we need to have a way to know if a path is originally quoted during runtime. So the information needed to be added at several levels: * parse time (from user input to expression) We need to add quoted information into `Expr::Filepath`, `Expr::Directory`, `Expr::GlobPattern` * eval time When convert from `Expr::Filepath`, `Expr::Directory`, `Expr::GlobPattern` to `Value::String` during runtime, we won't auto expanded the path if it's quoted ### For `ls` It's really special, because it accepts a `String` as a pattern, and it generates `glob` expression inside the command itself. So the idea behind the change is introducing a special SyntaxShape to ls: `SyntaxShape::LsGlobPattern`. So we can track if the pattern is originally quoted easier, and we don't auto expand the path either. Then when constructing a glob pattern inside ls, we check if input pattern is quoted, if so: we escape the input pattern, so we can run `ls a[123]b`, because it's already escaped. Finally, to accomplish the checking process, we also need to introduce a new value type called `Value::QuotedString` to differ from `Value::String`, it's used to generate an enum called `NuPath`, which is finally used in `ls` function. `ls` learned from `NuPath` to know if user input is quoted. # User-Facing Changes Actually it contains several changes ### For arguments which is annotated with `:path/:directory/:glob` #### Before ```nushell > def foo [p: path] { echo $p }; print (foo "~/a"); print (foo '~/a') /home/windsoilder/a /home/windsoilder/a > def foo [p: directory] { echo $p }; print (foo "~/a"); print (foo '~/a') /home/windsoilder/a /home/windsoilder/a > def foo [p: glob] { echo $p }; print (foo "~/a"); print (foo '~/a') /home/windsoilder/a /home/windsoilder/a ``` #### After ```nushell > def foo [p: path] { echo $p }; print (foo "~/a"); print (foo '~/a') ~/a ~/a > def foo [p: directory] { echo $p }; print (foo "~/a"); print (foo '~/a') ~/a ~/a > def foo [p: glob] { echo $p }; print (foo "~/a"); print (foo '~/a') ~/a ~/a ``` ### For ls command `touch '[uwu]'` #### Before ``` ❯ ls -D "[uwu]" Error: × No matches found for [uwu] ╭─[entry #6:1:1] 1 │ ls -D "[uwu]" · ───┬─── · ╰── Pattern, file or folder not found ╰──── help: no matches found ``` #### After ``` ❯ ls -D "[uwu]" ╭───┬───────┬──────┬──────┬──────────╮ │ # │ name │ type │ size │ modified │ ├───┼───────┼──────┼──────┼──────────┤ │ 0 │ [uwu] │ file │ 0 B │ now │ ╰───┴───────┴──────┴──────┴──────────╯ ``` # Tests + Formatting Done # After Submitting NaN
2024-01-21 15:22:25 +00:00
_: bool,
span: Span,
) -> Result<Value, ShellError> {
Err(ShellError::NotAConstant { span })
}
fn eval_var(
working_set: &StateWorkingSet,
_: &mut (),
var_id: VarId,
span: Span,
) -> Result<Value, ShellError> {
match working_set.get_variable(var_id).const_val.as_ref() {
Some(val) => Ok(val.clone()),
None => Err(ShellError::NotAConstant { span }),
}
}
Debugger experiments (#11441) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR adds a new evaluator path with callbacks to a mutable trait object implementing a Debugger trait. The trait object can do anything, e.g., profiling, code coverage, step debugging. Currently, entering/leaving a block and a pipeline element is marked with callbacks, but more callbacks can be added as necessary. Not all callbacks need to be used by all debuggers; unused ones are simply empty calls. A simple profiler is implemented as a proof of concept. The debugging support is implementing by making `eval_xxx()` functions generic depending on whether we're debugging or not. This has zero computational overhead, but makes the binary slightly larger (see benchmarks below). `eval_xxx()` variants called from commands (like `eval_block_with_early_return()` in `each`) are chosen with a dynamic dispatch for two reasons: to not grow the binary size due to duplicating the code of many commands, and for the fact that it isn't possible because it would make Command trait objects object-unsafe. In the future, I hope it will be possible to allow plugin callbacks such that users would be able to implement their profiler plugins instead of having to recompile Nushell. [DAP](https://microsoft.github.io/debug-adapter-protocol/) would also be interesting to explore. Try `help debug profile`. ## Screenshots Basic output: ![profiler_new](https://github.com/nushell/nushell/assets/25571562/418b9df0-b659-4dcb-b023-2d5fcef2c865) To profile with more granularity, increase the profiler depth (you'll see that repeated `is-windows` calls take a large chunk of total time, making it a good candidate for optimizing): ![profiler_new_m3](https://github.com/nushell/nushell/assets/25571562/636d756d-5d56-460c-a372-14716f65f37f) ## Benchmarks ### Binary size Binary size increase vs. main: **+40360 bytes**. _(Both built with `--release --features=extra,dataframe`.)_ ### Time ```nushell # bench_debug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'debug:' let res2 = bench { debug profile $test } --pretty print $res2 ``` ```nushell # bench_nodebug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'no debug:' let res1 = bench { do $test } --pretty print $res1 ``` `cargo run --release -- bench_debug.nu` is consistently 1--2 ms slower than `cargo run --release -- bench_nodebug.nu` due to the collection overhead + gathering the report. This is expected. When gathering more stuff, the overhead is obviously higher. `cargo run --release -- bench_nodebug.nu` vs. `nu bench_nodebug.nu` I didn't measure any difference. Both benchmarks report times between 97 and 103 ms randomly, without one being consistently higher than the other. This suggests that at least in this particular case, when not running any debugger, there is no runtime overhead. ## API changes This PR adds a generic parameter to all `eval_xxx` functions that forces you to specify whether you use the debugger. You can resolve it in two ways: * Use a provided helper that will figure it out for you. If you wanted to use `eval_block(&engine_state, ...)`, call `let eval_block = get_eval_block(&engine_state); eval_block(&engine_state, ...)` * If you know you're in an evaluation path that doesn't need debugger support, call `eval_block::<WithoutDebug>(&engine_state, ...)` (this is the case of hooks, for example). I tried to add more explanation in the docstring of `debugger_trait.rs`. ## TODO - [x] Better profiler output to reduce spam of iterative commands like `each` - [x] Resolve `TODO: DEBUG` comments - [x] Resolve unwraps - [x] Add doc comments - [x] Add usage and extra usage for `debug profile`, explaining all columns # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Hopefully none. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-03-08 18:21:35 +00:00
fn eval_call<D: DebugContext>(
working_set: &StateWorkingSet,
_: &mut (),
call: &Call,
span: Span,
) -> Result<Value, ShellError> {
Debugger experiments (#11441) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR adds a new evaluator path with callbacks to a mutable trait object implementing a Debugger trait. The trait object can do anything, e.g., profiling, code coverage, step debugging. Currently, entering/leaving a block and a pipeline element is marked with callbacks, but more callbacks can be added as necessary. Not all callbacks need to be used by all debuggers; unused ones are simply empty calls. A simple profiler is implemented as a proof of concept. The debugging support is implementing by making `eval_xxx()` functions generic depending on whether we're debugging or not. This has zero computational overhead, but makes the binary slightly larger (see benchmarks below). `eval_xxx()` variants called from commands (like `eval_block_with_early_return()` in `each`) are chosen with a dynamic dispatch for two reasons: to not grow the binary size due to duplicating the code of many commands, and for the fact that it isn't possible because it would make Command trait objects object-unsafe. In the future, I hope it will be possible to allow plugin callbacks such that users would be able to implement their profiler plugins instead of having to recompile Nushell. [DAP](https://microsoft.github.io/debug-adapter-protocol/) would also be interesting to explore. Try `help debug profile`. ## Screenshots Basic output: ![profiler_new](https://github.com/nushell/nushell/assets/25571562/418b9df0-b659-4dcb-b023-2d5fcef2c865) To profile with more granularity, increase the profiler depth (you'll see that repeated `is-windows` calls take a large chunk of total time, making it a good candidate for optimizing): ![profiler_new_m3](https://github.com/nushell/nushell/assets/25571562/636d756d-5d56-460c-a372-14716f65f37f) ## Benchmarks ### Binary size Binary size increase vs. main: **+40360 bytes**. _(Both built with `--release --features=extra,dataframe`.)_ ### Time ```nushell # bench_debug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'debug:' let res2 = bench { debug profile $test } --pretty print $res2 ``` ```nushell # bench_nodebug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'no debug:' let res1 = bench { do $test } --pretty print $res1 ``` `cargo run --release -- bench_debug.nu` is consistently 1--2 ms slower than `cargo run --release -- bench_nodebug.nu` due to the collection overhead + gathering the report. This is expected. When gathering more stuff, the overhead is obviously higher. `cargo run --release -- bench_nodebug.nu` vs. `nu bench_nodebug.nu` I didn't measure any difference. Both benchmarks report times between 97 and 103 ms randomly, without one being consistently higher than the other. This suggests that at least in this particular case, when not running any debugger, there is no runtime overhead. ## API changes This PR adds a generic parameter to all `eval_xxx` functions that forces you to specify whether you use the debugger. You can resolve it in two ways: * Use a provided helper that will figure it out for you. If you wanted to use `eval_block(&engine_state, ...)`, call `let eval_block = get_eval_block(&engine_state); eval_block(&engine_state, ...)` * If you know you're in an evaluation path that doesn't need debugger support, call `eval_block::<WithoutDebug>(&engine_state, ...)` (this is the case of hooks, for example). I tried to add more explanation in the docstring of `debugger_trait.rs`. ## TODO - [x] Better profiler output to reduce spam of iterative commands like `each` - [x] Resolve `TODO: DEBUG` comments - [x] Resolve unwraps - [x] Add doc comments - [x] Add usage and extra usage for `debug profile`, explaining all columns # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Hopefully none. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-03-08 18:21:35 +00:00
// TODO: Allow debugging const eval
// TODO: eval.rs uses call.head for the span rather than expr.span
Replace `ExternalStream` with new `ByteStream` type (#12774) # Description This PR introduces a `ByteStream` type which is a `Read`-able stream of bytes. Internally, it has an enum over three different byte stream sources: ```rust pub enum ByteStreamSource { Read(Box<dyn Read + Send + 'static>), File(File), Child(ChildProcess), } ``` This is in comparison to the current `RawStream` type, which is an `Iterator<Item = Vec<u8>>` and has to allocate for each read chunk. Currently, `PipelineData::ExternalStream` serves a weird dual role where it is either external command output or a wrapper around `RawStream`. `ByteStream` makes this distinction more clear (via `ByteStreamSource`) and replaces `PipelineData::ExternalStream` in this PR: ```rust pub enum PipelineData { Empty, Value(Value, Option<PipelineMetadata>), ListStream(ListStream, Option<PipelineMetadata>), ByteStream(ByteStream, Option<PipelineMetadata>), } ``` The PR is relatively large, but a decent amount of it is just repetitive changes. This PR fixes #7017, fixes #10763, and fixes #12369. This PR also improves performance when piping external commands. Nushell should, in most cases, have competitive pipeline throughput compared to, e.g., bash. | Command | Before (MB/s) | After (MB/s) | Bash (MB/s) | | -------------------------------------------------- | -------------:| ------------:| -----------:| | `throughput \| rg 'x'` | 3059 | 3744 | 3739 | | `throughput \| nu --testbin relay o> /dev/null` | 3508 | 8087 | 8136 | # User-Facing Changes - This is a breaking change for the plugin communication protocol, because the `ExternalStreamInfo` was replaced with `ByteStreamInfo`. Plugins now only have to deal with a single input stream, as opposed to the previous three streams: stdout, stderr, and exit code. - The output of `describe` has been changed for external/byte streams. - Temporary breaking change: `bytes starts-with` no longer works with byte streams. This is to keep the PR smaller, and `bytes ends-with` already does not work on byte streams. - If a process core dumped, then instead of having a `Value::Error` in the `exit_code` column of the output returned from `complete`, it now is a `Value::Int` with the negation of the signal number. # After Submitting - Update docs and book as necessary - Release notes (e.g., plugin protocol changes) - Adapt/convert commands to work with byte streams (high priority is `str length`, `bytes starts-with`, and maybe `bytes ends-with`). - Refactor the `tee` code, Devyn has already done some work on this. --------- Co-authored-by: Devyn Cairns <devyn.cairns@gmail.com>
2024-05-16 14:11:18 +00:00
eval_const_call(working_set, call, PipelineData::empty())?.into_value(span)
}
fn eval_external_call(
_: &StateWorkingSet,
_: &mut (),
_: &Expression,
Allow spreading arguments to commands (#11289) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> Finishes implementing https://github.com/nushell/nushell/issues/10598, which asks for a spread operator in lists, in records, and when calling commands. # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR will allow spreading arguments to commands (both internal and external). It will also deprecate spreading arguments automatically when passing to external commands. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> - Users will be able to use `...` to spread arguments to custom/builtin commands that have rest parameters or allow unknown arguments, or to any external command - If a custom command doesn't have a rest parameter and it doesn't allow unknown arguments either, the spread operator will not be allowed - Passing lists to external commands without `...` will work for now but will cause a deprecation warning saying that it'll stop working in 0.91 (is 2 versions enough time?) Here's a function to help with demonstrating some behavior: ```nushell > def foo [ a, b, c?, d?, ...rest ] { [$a $b $c $d $rest] | to nuon } ``` You can pass a list of arguments to fill in the `rest` parameter using `...`: ```nushell > foo 1 2 3 4 ...[5 6] [1, 2, 3, 4, [5, 6]] ``` If you don't use `...`, the list `[5 6]` will be treated as a single argument: ```nushell > foo 1 2 3 4 [5 6] # Note the double [[]] [1, 2, 3, 4, [[5, 6]]] ``` You can omit optional parameters before the spread arguments: ```nushell > foo 1 2 3 ...[4 5] # d is omitted here [1, 2, 3, null, [4, 5]] ``` If you have multiple lists, you can spread them all: ```nushell > foo 1 2 3 ...[4 5] 6 7 ...[8] ...[] [1, 2, 3, null, [4, 5, 6, 7, 8]] ``` Here's the kind of error you get when you try to spread arguments to a command with no rest parameter: ![image](https://github.com/nushell/nushell/assets/45539777/93faceae-00eb-4e59-ac3f-17f98436e6e4) And this is the warning you get when you pass a list to an external now (without `...`): ![image](https://github.com/nushell/nushell/assets/45539777/d368f590-201e-49fb-8b20-68476ced415e) # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> Added tests to cover the following cases: - Spreading arguments to a command that doesn't have a rest parameter (unexpected spread argument error) - Spreading arguments to a command that doesn't have a rest parameter *but* there's also a missing positional argument (missing positional error) - Spreading arguments to a command that doesn't have a rest parameter but does allow unknown arguments, such as `exec` (allowed) - Spreading a list literal containing arguments of the wrong type (parse error) - Spreading a non-list value, both to internal and external commands - Having named arguments in the middle of rest arguments - `explain`ing a command call that spreads its arguments # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> # Examples Suppose you have multiple tables: ```nushell let people = [[id name age]; [0 alice 100] [1 bob 200] [2 eve 300]] let evil_twins = [[id name age]; [0 ecila 100] [-1 bob 200] [-2 eve 300]] ``` Maybe you often find yourself needing to merge multiple tables and want a utility to do that. You could write a function like this: ```nushell def merge_all [ ...tables ] { $tables | reduce { |it, acc| $acc | merge $it } } ``` Then you can use it like this: ```nushell > merge_all ...([$people $evil_twins] | each { |$it| $it | select name age }) ╭───┬───────┬─────╮ │ # │ name │ age │ ├───┼───────┼─────┤ │ 0 │ ecila │ 100 │ │ 1 │ bob │ 200 │ │ 2 │ eve │ 300 │ ╰───┴───────┴─────╯ ``` Except they had duplicate columns, so now you first want to suffix every column with a number to tell you which table the column came from. You can make a command for that: ```nushell def select_and_merge [ --cols: list<string>, ...tables ] { let renamed_tables = $tables | enumerate | each { |it| $it.item | select $cols | rename ...($cols | each { |col| $col + ($it.index | into string) }) }; merge_all ...$renamed_tables } ``` And call it like this: ```nushell > select_and_merge --cols [name age] $people $evil_twins ╭───┬───────┬──────┬───────┬──────╮ │ # │ name0 │ age0 │ name1 │ age1 │ ├───┼───────┼──────┼───────┼──────┤ │ 0 │ alice │ 100 │ ecila │ 100 │ │ 1 │ bob │ 200 │ bob │ 200 │ │ 2 │ eve │ 300 │ eve │ 300 │ ╰───┴───────┴──────┴───────┴──────╯ ``` --- Suppose someone's made a command to search for APT packages: ```nushell # The main command def search-pkgs [ --install # Whether to install any packages it finds log_level: int # Pretend it's a good idea to make this a required positional parameter exclude?: list<string> # Packages to exclude repositories?: list<string> # Which repositories to look in (searches in all if not given) ...pkgs # Package names to search for ] { { install: $install, log_level: $log_level, exclude: ($exclude | to nuon), repositories: ($repositories | to nuon), pkgs: ($pkgs | to nuon) } } ``` It has a lot of parameters to configure it, so you might make your own helper commands to wrap around it for specific cases. Here's one example: ```nushell # Only look for packages locally def search-pkgs-local [ --install # Whether to install any packages it finds log_level: int exclude?: list<string> # Packages to exclude ...pkgs # Package names to search for ] { # All required and optional positional parameters are given search-pkgs --install=$install $log_level [] ["<local URI or something>"] ...$pkgs } ``` And you can run it like this: ```nushell > search-pkgs-local --install=false 5 ...["python2.7" "vim"] ╭──────────────┬──────────────────────────────╮ │ install │ false │ │ log_level │ 5 │ │ exclude │ [] │ │ repositories │ ["<local URI or something>"] │ │ pkgs │ ["python2.7", vim] │ ╰──────────────┴──────────────────────────────╯ ``` One thing I realized when writing this was that if we decide to not allow passing optional arguments using the spread operator, then you can (mis?)use the spread operator to skip optional parameters. Here, I didn't want to give `exclude` explicitly, so I used a spread operator to pass the packages to install. Without it, I would've needed to do `search-pkgs-local --install=false 5 [] "python2.7" "vim"` (explicitly pass `[]` (or `null`, in the general case) to `exclude`). There are probably more idiomatic ways to do this, but I just thought it was something interesting. If you're a virologist of the [xkcd](https://xkcd.com/350/) kind, another helper command you might make is this: ```nushell # Install any packages it finds def live-dangerously [ ...pkgs ] { # One optional argument was given (exclude), while another was not (repositories) search-pkgs 0 [] ...$pkgs --install # Flags can go after spread arguments } ``` Running it: ```nushell > live-dangerously "git" "*vi*" # *vi* because I don't feel like typing out vim and neovim ╭──────────────┬─────────────╮ │ install │ true │ │ log_level │ 0 │ │ exclude │ [] │ │ repositories │ null │ │ pkgs │ [git, *vi*] │ ╰──────────────┴─────────────╯ ``` Here's an example that uses the spread operator more than once within the same command call: ```nushell let extras = [ chrome firefox python java git ] def search-pkgs-curated [ ...pkgs ] { (search-pkgs 1 [emacs] ["example.com", "foo.com"] vim # A must for everyone! ...($pkgs | filter { |p| not ($p | str contains "*") }) # Remove packages with globs python # Good tool to have ...$extras --install=false python3) # I forget, did I already put Python in extras? } ``` Running it: ```nushell > search-pkgs-curated "git" "*vi*" ╭──────────────┬───────────────────────────────────────────────────────────────────╮ │ install │ false │ │ log_level │ 1 │ │ exclude │ [emacs] │ │ repositories │ [example.com, foo.com] │ │ pkgs │ [vim, git, python, chrome, firefox, python, java, git, "python3"] │ ╰──────────────┴───────────────────────────────────────────────────────────────────╯ ```
2023-12-28 07:43:20 +00:00
_: &[ExternalArgument],
span: Span,
) -> Result<Value, ShellError> {
// TODO: It may be more helpful to give not_a_const_command error
Err(ShellError::NotAConstant { span })
}
Overhaul `$in` expressions (#13357) # Description This grew quite a bit beyond its original scope, but I've tried to make `$in` a bit more consistent and easier to work with. Instead of the parser generating calls to `collect` and creating closures, this adds `Expr::Collect` which just evaluates in the same scope and doesn't require any closure. When `$in` is detected in an expression, it is replaced with a new variable (also called `$in`) and wrapped in `Expr::Collect`. During eval, this expression is evaluated directly, with the input and with that new variable set to the collected value. Other than being faster and less prone to gotchas, it also makes it possible to typecheck the output of an expression containing `$in`, which is nice. This is a breaking change though, because of the lack of the closure and because now typechecking will actually happen. Also, I haven't attempted to typecheck the input yet. The IR generated now just looks like this: ```gas collect %in clone %tmp, %in store-variable $in, %tmp # %out <- ...expression... <- %in drop-variable $in ``` (where `$in` is the local variable created for this collection, and not `IN_VARIABLE_ID`) which is a lot better than having to create a closure and call `collect --keep-env`, dealing with all of the capture gathering and allocation that entails. Ideally we can also detect whether that input is actually needed, so maybe we don't have to clone, but I haven't tried to do that yet. Theoretically now that the variable is a unique one every time, it should be possible to give it a type - I just don't know how to determine that yet. On top of that, I've also reworked how `$in` works in pipeline-initial position. Previously, it was a little bit inconsistent. For example, this worked: ```nushell > 3 | do { let x = $in; let y = $in; print $x $y } 3 3 ``` However, this causes a runtime variable not found error on the second `$in`: ```nushell > def foo [] { let x = $in; let y = $in; print $x $y }; 3 | foo Error: nu::shell::variable_not_found × Variable not found ╭─[entry #115:1:35] 1 │ def foo [] { let x = $in; let y = $in; print $x $y }; 3 | foo · ─┬─ · ╰── variable not found ╰──── ``` I've fixed this by making the first element `$in` detection *always* happen at the block level, so if you use `$in` in pipeline-initial position anywhere in a block, it will collect with an implicit subexpression around the whole thing, and you can then use that `$in` more than once. In doing this I also rewrote `parse_pipeline()` and hopefully it's a bit more straightforward and possibly more efficient too now. Finally, I've tried to make `let` and `mut` a lot more straightforward with how they handle the rest of the pipeline, and using a redirection with `let`/`mut` now does what you'd expect if you assume that they consume the whole pipeline - the redirection is just processed as normal. These both work now: ```nushell let x = ^foo err> err.txt let y = ^foo out+err>| str length ``` It was previously possible to accomplish this with a subexpression, but it just seemed like a weird gotcha that you couldn't do it. Intuitively, `let` and `mut` just seem to take the whole line. - closes #13137 # User-Facing Changes - `$in` will behave more consistently with blocks and closures, since the entire block is now just wrapped to handle it if it appears in the first pipeline element - `$in` no longer creates a closure, so what can be done within an expression containing `$in` is less restrictive - `$in` containing expressions are now type checked, rather than just resulting in `any`. However, `$in` itself is still `any`, so this isn't quite perfect yet - Redirections are now allowed in `let` and `mut` and behave pretty much how you'd expect # Tests + Formatting Added tests to cover the new behaviour. # After Submitting - [ ] release notes (definitely breaking change)
2024-07-17 21:02:42 +00:00
fn eval_collect<D: DebugContext>(
_: &StateWorkingSet,
_: &mut (),
_var_id: VarId,
expr: &Expression,
) -> Result<Value, ShellError> {
Err(ShellError::NotAConstant { span: expr.span })
}
Debugger experiments (#11441) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR adds a new evaluator path with callbacks to a mutable trait object implementing a Debugger trait. The trait object can do anything, e.g., profiling, code coverage, step debugging. Currently, entering/leaving a block and a pipeline element is marked with callbacks, but more callbacks can be added as necessary. Not all callbacks need to be used by all debuggers; unused ones are simply empty calls. A simple profiler is implemented as a proof of concept. The debugging support is implementing by making `eval_xxx()` functions generic depending on whether we're debugging or not. This has zero computational overhead, but makes the binary slightly larger (see benchmarks below). `eval_xxx()` variants called from commands (like `eval_block_with_early_return()` in `each`) are chosen with a dynamic dispatch for two reasons: to not grow the binary size due to duplicating the code of many commands, and for the fact that it isn't possible because it would make Command trait objects object-unsafe. In the future, I hope it will be possible to allow plugin callbacks such that users would be able to implement their profiler plugins instead of having to recompile Nushell. [DAP](https://microsoft.github.io/debug-adapter-protocol/) would also be interesting to explore. Try `help debug profile`. ## Screenshots Basic output: ![profiler_new](https://github.com/nushell/nushell/assets/25571562/418b9df0-b659-4dcb-b023-2d5fcef2c865) To profile with more granularity, increase the profiler depth (you'll see that repeated `is-windows` calls take a large chunk of total time, making it a good candidate for optimizing): ![profiler_new_m3](https://github.com/nushell/nushell/assets/25571562/636d756d-5d56-460c-a372-14716f65f37f) ## Benchmarks ### Binary size Binary size increase vs. main: **+40360 bytes**. _(Both built with `--release --features=extra,dataframe`.)_ ### Time ```nushell # bench_debug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'debug:' let res2 = bench { debug profile $test } --pretty print $res2 ``` ```nushell # bench_nodebug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'no debug:' let res1 = bench { do $test } --pretty print $res1 ``` `cargo run --release -- bench_debug.nu` is consistently 1--2 ms slower than `cargo run --release -- bench_nodebug.nu` due to the collection overhead + gathering the report. This is expected. When gathering more stuff, the overhead is obviously higher. `cargo run --release -- bench_nodebug.nu` vs. `nu bench_nodebug.nu` I didn't measure any difference. Both benchmarks report times between 97 and 103 ms randomly, without one being consistently higher than the other. This suggests that at least in this particular case, when not running any debugger, there is no runtime overhead. ## API changes This PR adds a generic parameter to all `eval_xxx` functions that forces you to specify whether you use the debugger. You can resolve it in two ways: * Use a provided helper that will figure it out for you. If you wanted to use `eval_block(&engine_state, ...)`, call `let eval_block = get_eval_block(&engine_state); eval_block(&engine_state, ...)` * If you know you're in an evaluation path that doesn't need debugger support, call `eval_block::<WithoutDebug>(&engine_state, ...)` (this is the case of hooks, for example). I tried to add more explanation in the docstring of `debugger_trait.rs`. ## TODO - [x] Better profiler output to reduce spam of iterative commands like `each` - [x] Resolve `TODO: DEBUG` comments - [x] Resolve unwraps - [x] Add doc comments - [x] Add usage and extra usage for `debug profile`, explaining all columns # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Hopefully none. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-03-08 18:21:35 +00:00
fn eval_subexpression<D: DebugContext>(
working_set: &StateWorkingSet,
_: &mut (),
block_id: BlockId,
span: Span,
) -> Result<Value, ShellError> {
Debugger experiments (#11441) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR adds a new evaluator path with callbacks to a mutable trait object implementing a Debugger trait. The trait object can do anything, e.g., profiling, code coverage, step debugging. Currently, entering/leaving a block and a pipeline element is marked with callbacks, but more callbacks can be added as necessary. Not all callbacks need to be used by all debuggers; unused ones are simply empty calls. A simple profiler is implemented as a proof of concept. The debugging support is implementing by making `eval_xxx()` functions generic depending on whether we're debugging or not. This has zero computational overhead, but makes the binary slightly larger (see benchmarks below). `eval_xxx()` variants called from commands (like `eval_block_with_early_return()` in `each`) are chosen with a dynamic dispatch for two reasons: to not grow the binary size due to duplicating the code of many commands, and for the fact that it isn't possible because it would make Command trait objects object-unsafe. In the future, I hope it will be possible to allow plugin callbacks such that users would be able to implement their profiler plugins instead of having to recompile Nushell. [DAP](https://microsoft.github.io/debug-adapter-protocol/) would also be interesting to explore. Try `help debug profile`. ## Screenshots Basic output: ![profiler_new](https://github.com/nushell/nushell/assets/25571562/418b9df0-b659-4dcb-b023-2d5fcef2c865) To profile with more granularity, increase the profiler depth (you'll see that repeated `is-windows` calls take a large chunk of total time, making it a good candidate for optimizing): ![profiler_new_m3](https://github.com/nushell/nushell/assets/25571562/636d756d-5d56-460c-a372-14716f65f37f) ## Benchmarks ### Binary size Binary size increase vs. main: **+40360 bytes**. _(Both built with `--release --features=extra,dataframe`.)_ ### Time ```nushell # bench_debug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'debug:' let res2 = bench { debug profile $test } --pretty print $res2 ``` ```nushell # bench_nodebug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'no debug:' let res1 = bench { do $test } --pretty print $res1 ``` `cargo run --release -- bench_debug.nu` is consistently 1--2 ms slower than `cargo run --release -- bench_nodebug.nu` due to the collection overhead + gathering the report. This is expected. When gathering more stuff, the overhead is obviously higher. `cargo run --release -- bench_nodebug.nu` vs. `nu bench_nodebug.nu` I didn't measure any difference. Both benchmarks report times between 97 and 103 ms randomly, without one being consistently higher than the other. This suggests that at least in this particular case, when not running any debugger, there is no runtime overhead. ## API changes This PR adds a generic parameter to all `eval_xxx` functions that forces you to specify whether you use the debugger. You can resolve it in two ways: * Use a provided helper that will figure it out for you. If you wanted to use `eval_block(&engine_state, ...)`, call `let eval_block = get_eval_block(&engine_state); eval_block(&engine_state, ...)` * If you know you're in an evaluation path that doesn't need debugger support, call `eval_block::<WithoutDebug>(&engine_state, ...)` (this is the case of hooks, for example). I tried to add more explanation in the docstring of `debugger_trait.rs`. ## TODO - [x] Better profiler output to reduce spam of iterative commands like `each` - [x] Resolve `TODO: DEBUG` comments - [x] Resolve unwraps - [x] Add doc comments - [x] Add usage and extra usage for `debug profile`, explaining all columns # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Hopefully none. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-03-08 18:21:35 +00:00
// TODO: Allow debugging const eval
let block = working_set.get_block(block_id);
Replace `ExternalStream` with new `ByteStream` type (#12774) # Description This PR introduces a `ByteStream` type which is a `Read`-able stream of bytes. Internally, it has an enum over three different byte stream sources: ```rust pub enum ByteStreamSource { Read(Box<dyn Read + Send + 'static>), File(File), Child(ChildProcess), } ``` This is in comparison to the current `RawStream` type, which is an `Iterator<Item = Vec<u8>>` and has to allocate for each read chunk. Currently, `PipelineData::ExternalStream` serves a weird dual role where it is either external command output or a wrapper around `RawStream`. `ByteStream` makes this distinction more clear (via `ByteStreamSource`) and replaces `PipelineData::ExternalStream` in this PR: ```rust pub enum PipelineData { Empty, Value(Value, Option<PipelineMetadata>), ListStream(ListStream, Option<PipelineMetadata>), ByteStream(ByteStream, Option<PipelineMetadata>), } ``` The PR is relatively large, but a decent amount of it is just repetitive changes. This PR fixes #7017, fixes #10763, and fixes #12369. This PR also improves performance when piping external commands. Nushell should, in most cases, have competitive pipeline throughput compared to, e.g., bash. | Command | Before (MB/s) | After (MB/s) | Bash (MB/s) | | -------------------------------------------------- | -------------:| ------------:| -----------:| | `throughput \| rg 'x'` | 3059 | 3744 | 3739 | | `throughput \| nu --testbin relay o> /dev/null` | 3508 | 8087 | 8136 | # User-Facing Changes - This is a breaking change for the plugin communication protocol, because the `ExternalStreamInfo` was replaced with `ByteStreamInfo`. Plugins now only have to deal with a single input stream, as opposed to the previous three streams: stdout, stderr, and exit code. - The output of `describe` has been changed for external/byte streams. - Temporary breaking change: `bytes starts-with` no longer works with byte streams. This is to keep the PR smaller, and `bytes ends-with` already does not work on byte streams. - If a process core dumped, then instead of having a `Value::Error` in the `exit_code` column of the output returned from `complete`, it now is a `Value::Int` with the negation of the signal number. # After Submitting - Update docs and book as necessary - Release notes (e.g., plugin protocol changes) - Adapt/convert commands to work with byte streams (high priority is `str length`, `bytes starts-with`, and maybe `bytes ends-with`). - Refactor the `tee` code, Devyn has already done some work on this. --------- Co-authored-by: Devyn Cairns <devyn.cairns@gmail.com>
2024-05-16 14:11:18 +00:00
eval_const_subexpression(working_set, block, PipelineData::empty(), span)?.into_value(span)
}
fn regex_match(
_: &StateWorkingSet,
_op_span: Span,
_: &Value,
_: &Value,
_: bool,
expr_span: Span,
) -> Result<Value, ShellError> {
Err(ShellError::NotAConstant { span: expr_span })
}
Debugger experiments (#11441) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR adds a new evaluator path with callbacks to a mutable trait object implementing a Debugger trait. The trait object can do anything, e.g., profiling, code coverage, step debugging. Currently, entering/leaving a block and a pipeline element is marked with callbacks, but more callbacks can be added as necessary. Not all callbacks need to be used by all debuggers; unused ones are simply empty calls. A simple profiler is implemented as a proof of concept. The debugging support is implementing by making `eval_xxx()` functions generic depending on whether we're debugging or not. This has zero computational overhead, but makes the binary slightly larger (see benchmarks below). `eval_xxx()` variants called from commands (like `eval_block_with_early_return()` in `each`) are chosen with a dynamic dispatch for two reasons: to not grow the binary size due to duplicating the code of many commands, and for the fact that it isn't possible because it would make Command trait objects object-unsafe. In the future, I hope it will be possible to allow plugin callbacks such that users would be able to implement their profiler plugins instead of having to recompile Nushell. [DAP](https://microsoft.github.io/debug-adapter-protocol/) would also be interesting to explore. Try `help debug profile`. ## Screenshots Basic output: ![profiler_new](https://github.com/nushell/nushell/assets/25571562/418b9df0-b659-4dcb-b023-2d5fcef2c865) To profile with more granularity, increase the profiler depth (you'll see that repeated `is-windows` calls take a large chunk of total time, making it a good candidate for optimizing): ![profiler_new_m3](https://github.com/nushell/nushell/assets/25571562/636d756d-5d56-460c-a372-14716f65f37f) ## Benchmarks ### Binary size Binary size increase vs. main: **+40360 bytes**. _(Both built with `--release --features=extra,dataframe`.)_ ### Time ```nushell # bench_debug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'debug:' let res2 = bench { debug profile $test } --pretty print $res2 ``` ```nushell # bench_nodebug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'no debug:' let res1 = bench { do $test } --pretty print $res1 ``` `cargo run --release -- bench_debug.nu` is consistently 1--2 ms slower than `cargo run --release -- bench_nodebug.nu` due to the collection overhead + gathering the report. This is expected. When gathering more stuff, the overhead is obviously higher. `cargo run --release -- bench_nodebug.nu` vs. `nu bench_nodebug.nu` I didn't measure any difference. Both benchmarks report times between 97 and 103 ms randomly, without one being consistently higher than the other. This suggests that at least in this particular case, when not running any debugger, there is no runtime overhead. ## API changes This PR adds a generic parameter to all `eval_xxx` functions that forces you to specify whether you use the debugger. You can resolve it in two ways: * Use a provided helper that will figure it out for you. If you wanted to use `eval_block(&engine_state, ...)`, call `let eval_block = get_eval_block(&engine_state); eval_block(&engine_state, ...)` * If you know you're in an evaluation path that doesn't need debugger support, call `eval_block::<WithoutDebug>(&engine_state, ...)` (this is the case of hooks, for example). I tried to add more explanation in the docstring of `debugger_trait.rs`. ## TODO - [x] Better profiler output to reduce spam of iterative commands like `each` - [x] Resolve `TODO: DEBUG` comments - [x] Resolve unwraps - [x] Add doc comments - [x] Add usage and extra usage for `debug profile`, explaining all columns # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Hopefully none. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-03-08 18:21:35 +00:00
fn eval_assignment<D: DebugContext>(
_: &StateWorkingSet,
_: &mut (),
_: &Expression,
_: &Expression,
_: Assignment,
_op_span: Span,
expr_span: Span,
) -> Result<Value, ShellError> {
Debugger experiments (#11441) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR adds a new evaluator path with callbacks to a mutable trait object implementing a Debugger trait. The trait object can do anything, e.g., profiling, code coverage, step debugging. Currently, entering/leaving a block and a pipeline element is marked with callbacks, but more callbacks can be added as necessary. Not all callbacks need to be used by all debuggers; unused ones are simply empty calls. A simple profiler is implemented as a proof of concept. The debugging support is implementing by making `eval_xxx()` functions generic depending on whether we're debugging or not. This has zero computational overhead, but makes the binary slightly larger (see benchmarks below). `eval_xxx()` variants called from commands (like `eval_block_with_early_return()` in `each`) are chosen with a dynamic dispatch for two reasons: to not grow the binary size due to duplicating the code of many commands, and for the fact that it isn't possible because it would make Command trait objects object-unsafe. In the future, I hope it will be possible to allow plugin callbacks such that users would be able to implement their profiler plugins instead of having to recompile Nushell. [DAP](https://microsoft.github.io/debug-adapter-protocol/) would also be interesting to explore. Try `help debug profile`. ## Screenshots Basic output: ![profiler_new](https://github.com/nushell/nushell/assets/25571562/418b9df0-b659-4dcb-b023-2d5fcef2c865) To profile with more granularity, increase the profiler depth (you'll see that repeated `is-windows` calls take a large chunk of total time, making it a good candidate for optimizing): ![profiler_new_m3](https://github.com/nushell/nushell/assets/25571562/636d756d-5d56-460c-a372-14716f65f37f) ## Benchmarks ### Binary size Binary size increase vs. main: **+40360 bytes**. _(Both built with `--release --features=extra,dataframe`.)_ ### Time ```nushell # bench_debug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'debug:' let res2 = bench { debug profile $test } --pretty print $res2 ``` ```nushell # bench_nodebug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'no debug:' let res1 = bench { do $test } --pretty print $res1 ``` `cargo run --release -- bench_debug.nu` is consistently 1--2 ms slower than `cargo run --release -- bench_nodebug.nu` due to the collection overhead + gathering the report. This is expected. When gathering more stuff, the overhead is obviously higher. `cargo run --release -- bench_nodebug.nu` vs. `nu bench_nodebug.nu` I didn't measure any difference. Both benchmarks report times between 97 and 103 ms randomly, without one being consistently higher than the other. This suggests that at least in this particular case, when not running any debugger, there is no runtime overhead. ## API changes This PR adds a generic parameter to all `eval_xxx` functions that forces you to specify whether you use the debugger. You can resolve it in two ways: * Use a provided helper that will figure it out for you. If you wanted to use `eval_block(&engine_state, ...)`, call `let eval_block = get_eval_block(&engine_state); eval_block(&engine_state, ...)` * If you know you're in an evaluation path that doesn't need debugger support, call `eval_block::<WithoutDebug>(&engine_state, ...)` (this is the case of hooks, for example). I tried to add more explanation in the docstring of `debugger_trait.rs`. ## TODO - [x] Better profiler output to reduce spam of iterative commands like `each` - [x] Resolve `TODO: DEBUG` comments - [x] Resolve unwraps - [x] Add doc comments - [x] Add usage and extra usage for `debug profile`, explaining all columns # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Hopefully none. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-03-08 18:21:35 +00:00
// TODO: Allow debugging const eval
Err(ShellError::NotAConstant { span: expr_span })
}
fn eval_row_condition_or_closure(
_: &StateWorkingSet,
_: &mut (),
_: BlockId,
span: Span,
) -> Result<Value, ShellError> {
Err(ShellError::NotAConstant { span })
}
fn eval_overlay(_: &StateWorkingSet, span: Span) -> Result<Value, ShellError> {
Err(ShellError::NotAConstant { span })
}
Span ID Refactor (Step 2): Use SpanId of expressions in some places (#13102) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Part of https://github.com/nushell/nushell/issues/12963, step 2. This PR refactors changes the use of `expression.span` to `expression.span_id` via a new helper `Expression::span()`. A new `GetSpan` is added to abstract getting the span from both `EngineState` and `StateWorkingSet`. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> `format pattern` loses the ability to use variables in the pattern, e.g., `... | format pattern 'value of {$it.name} is {$it.value}'`. This is because the command did a custom parse-eval cycle, creating spans that are not merged into the main engine state. We could clone the engine state, add Clone trait to StateDelta and merge the cloned delta to the cloned state, but IMO there is not much value from having this ability, since we have string interpolation nowadays: `... | $"value of ($in.name) is ($in.value)"`. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-06-09 09:15:53 +00:00
fn unreachable(working_set: &StateWorkingSet, expr: &Expression) -> Result<Value, ShellError> {
Err(ShellError::NotAConstant {
span: expr.span(&working_set),
})
}
}