nushell/crates/nu-engine/src/call_ext.rs

193 lines
5.6 KiB
Rust
Raw Normal View History

use crate::eval_expression;
2021-10-25 06:31:39 +00:00
use nu_protocol::{
Fix incorrect handling of boolean flags for builtin commands (#11492) # Description Possible fix of #11456 This PR fixes a bug where builtin commands did not respect the logic of dynamically passed boolean flags. The reason is [has_flag](https://github.com/nushell/nushell/blob/6f59abaf4310487f7a6319437be6ec61abcbc3b9/crates/nu-protocol/src/ast/call.rs#L204C5-L212C6) method did not evaluate and take into consideration expression used with flag. To address this issue a solution is proposed: 1. `has_flag` method is moved to `CallExt` and new logic to evaluate expression and check if it is a boolean value is added 2. `has_flag_const` method is added to `CallExt` which is a constant version of `has_flag` 3. `has_named` method is added to `Call` which is basically the old logic of `has_flag` 4. All usages of `has_flag` in code are updated, mostly to pass `engine_state` and `stack` to new `has_flag`. In `run_const` commands it is replaced with `has_flag_const`. And in a few select places: parser, `to nuon` and `into string` old logic via `has_named` is used. # User-Facing Changes Explicit values of boolean flags are now respected in builtin commands. Before: ![image](https://github.com/nushell/nushell/assets/17511668/f9fbabb2-3cfd-43f9-ba9e-ece76d80043c) After: ![image](https://github.com/nushell/nushell/assets/17511668/21867596-2075-437f-9c85-45563ac70083) Another example: Before: ![image](https://github.com/nushell/nushell/assets/17511668/efdbc5ca-5227-45a4-ac5b-532cdc2bbf5f) After: ![image](https://github.com/nushell/nushell/assets/17511668/2907d5c5-aa93-404d-af1c-21cdc3d44646) # Tests + Formatting Added test reproducing some variants of original issue.
2024-01-11 15:19:48 +00:00
ast::Call,
debugger::WithoutDebug,
make the `ansi` command `const` (#11682) # Description This PR changes the `ansi` command to be a `const` command. - ~~It's breaking because I found that I had to change the way `ansi` is used in scripts a little bit. https://github.com/nushell/nu_scripts/pull/751~~ - I had to change one of the examples because apparently `const` can't be tested yet. - ~~I'm not sure this is right at all https://github.com/nushell/nushell/pull/11682/files#diff-ba932369a40eb40d6e1985eac1c784af403dab4500a7f0568e593900bf6cd740R654-R655. I just didn't want to duplicate a ton of code. Maybe if I duplicated the code it wouldn't be a breaking change because it would have a run and run_const?~~ - I had to add `opt_const` to CallExt. /cc @kubouch Can you take a look at this? I'm a little iffy if I'm doing this right, or even if we should do this at all. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-01-30 22:09:43 +00:00
engine::{EngineState, Stack, StateWorkingSet},
eval_const::eval_constant,
Allow spreading arguments to commands (#11289) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> Finishes implementing https://github.com/nushell/nushell/issues/10598, which asks for a spread operator in lists, in records, and when calling commands. # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR will allow spreading arguments to commands (both internal and external). It will also deprecate spreading arguments automatically when passing to external commands. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> - Users will be able to use `...` to spread arguments to custom/builtin commands that have rest parameters or allow unknown arguments, or to any external command - If a custom command doesn't have a rest parameter and it doesn't allow unknown arguments either, the spread operator will not be allowed - Passing lists to external commands without `...` will work for now but will cause a deprecation warning saying that it'll stop working in 0.91 (is 2 versions enough time?) Here's a function to help with demonstrating some behavior: ```nushell > def foo [ a, b, c?, d?, ...rest ] { [$a $b $c $d $rest] | to nuon } ``` You can pass a list of arguments to fill in the `rest` parameter using `...`: ```nushell > foo 1 2 3 4 ...[5 6] [1, 2, 3, 4, [5, 6]] ``` If you don't use `...`, the list `[5 6]` will be treated as a single argument: ```nushell > foo 1 2 3 4 [5 6] # Note the double [[]] [1, 2, 3, 4, [[5, 6]]] ``` You can omit optional parameters before the spread arguments: ```nushell > foo 1 2 3 ...[4 5] # d is omitted here [1, 2, 3, null, [4, 5]] ``` If you have multiple lists, you can spread them all: ```nushell > foo 1 2 3 ...[4 5] 6 7 ...[8] ...[] [1, 2, 3, null, [4, 5, 6, 7, 8]] ``` Here's the kind of error you get when you try to spread arguments to a command with no rest parameter: ![image](https://github.com/nushell/nushell/assets/45539777/93faceae-00eb-4e59-ac3f-17f98436e6e4) And this is the warning you get when you pass a list to an external now (without `...`): ![image](https://github.com/nushell/nushell/assets/45539777/d368f590-201e-49fb-8b20-68476ced415e) # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> Added tests to cover the following cases: - Spreading arguments to a command that doesn't have a rest parameter (unexpected spread argument error) - Spreading arguments to a command that doesn't have a rest parameter *but* there's also a missing positional argument (missing positional error) - Spreading arguments to a command that doesn't have a rest parameter but does allow unknown arguments, such as `exec` (allowed) - Spreading a list literal containing arguments of the wrong type (parse error) - Spreading a non-list value, both to internal and external commands - Having named arguments in the middle of rest arguments - `explain`ing a command call that spreads its arguments # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> # Examples Suppose you have multiple tables: ```nushell let people = [[id name age]; [0 alice 100] [1 bob 200] [2 eve 300]] let evil_twins = [[id name age]; [0 ecila 100] [-1 bob 200] [-2 eve 300]] ``` Maybe you often find yourself needing to merge multiple tables and want a utility to do that. You could write a function like this: ```nushell def merge_all [ ...tables ] { $tables | reduce { |it, acc| $acc | merge $it } } ``` Then you can use it like this: ```nushell > merge_all ...([$people $evil_twins] | each { |$it| $it | select name age }) ╭───┬───────┬─────╮ │ # │ name │ age │ ├───┼───────┼─────┤ │ 0 │ ecila │ 100 │ │ 1 │ bob │ 200 │ │ 2 │ eve │ 300 │ ╰───┴───────┴─────╯ ``` Except they had duplicate columns, so now you first want to suffix every column with a number to tell you which table the column came from. You can make a command for that: ```nushell def select_and_merge [ --cols: list<string>, ...tables ] { let renamed_tables = $tables | enumerate | each { |it| $it.item | select $cols | rename ...($cols | each { |col| $col + ($it.index | into string) }) }; merge_all ...$renamed_tables } ``` And call it like this: ```nushell > select_and_merge --cols [name age] $people $evil_twins ╭───┬───────┬──────┬───────┬──────╮ │ # │ name0 │ age0 │ name1 │ age1 │ ├───┼───────┼──────┼───────┼──────┤ │ 0 │ alice │ 100 │ ecila │ 100 │ │ 1 │ bob │ 200 │ bob │ 200 │ │ 2 │ eve │ 300 │ eve │ 300 │ ╰───┴───────┴──────┴───────┴──────╯ ``` --- Suppose someone's made a command to search for APT packages: ```nushell # The main command def search-pkgs [ --install # Whether to install any packages it finds log_level: int # Pretend it's a good idea to make this a required positional parameter exclude?: list<string> # Packages to exclude repositories?: list<string> # Which repositories to look in (searches in all if not given) ...pkgs # Package names to search for ] { { install: $install, log_level: $log_level, exclude: ($exclude | to nuon), repositories: ($repositories | to nuon), pkgs: ($pkgs | to nuon) } } ``` It has a lot of parameters to configure it, so you might make your own helper commands to wrap around it for specific cases. Here's one example: ```nushell # Only look for packages locally def search-pkgs-local [ --install # Whether to install any packages it finds log_level: int exclude?: list<string> # Packages to exclude ...pkgs # Package names to search for ] { # All required and optional positional parameters are given search-pkgs --install=$install $log_level [] ["<local URI or something>"] ...$pkgs } ``` And you can run it like this: ```nushell > search-pkgs-local --install=false 5 ...["python2.7" "vim"] ╭──────────────┬──────────────────────────────╮ │ install │ false │ │ log_level │ 5 │ │ exclude │ [] │ │ repositories │ ["<local URI or something>"] │ │ pkgs │ ["python2.7", vim] │ ╰──────────────┴──────────────────────────────╯ ``` One thing I realized when writing this was that if we decide to not allow passing optional arguments using the spread operator, then you can (mis?)use the spread operator to skip optional parameters. Here, I didn't want to give `exclude` explicitly, so I used a spread operator to pass the packages to install. Without it, I would've needed to do `search-pkgs-local --install=false 5 [] "python2.7" "vim"` (explicitly pass `[]` (or `null`, in the general case) to `exclude`). There are probably more idiomatic ways to do this, but I just thought it was something interesting. If you're a virologist of the [xkcd](https://xkcd.com/350/) kind, another helper command you might make is this: ```nushell # Install any packages it finds def live-dangerously [ ...pkgs ] { # One optional argument was given (exclude), while another was not (repositories) search-pkgs 0 [] ...$pkgs --install # Flags can go after spread arguments } ``` Running it: ```nushell > live-dangerously "git" "*vi*" # *vi* because I don't feel like typing out vim and neovim ╭──────────────┬─────────────╮ │ install │ true │ │ log_level │ 0 │ │ exclude │ [] │ │ repositories │ null │ │ pkgs │ [git, *vi*] │ ╰──────────────┴─────────────╯ ``` Here's an example that uses the spread operator more than once within the same command call: ```nushell let extras = [ chrome firefox python java git ] def search-pkgs-curated [ ...pkgs ] { (search-pkgs 1 [emacs] ["example.com", "foo.com"] vim # A must for everyone! ...($pkgs | filter { |p| not ($p | str contains "*") }) # Remove packages with globs python # Good tool to have ...$extras --install=false python3) # I forget, did I already put Python in extras? } ``` Running it: ```nushell > search-pkgs-curated "git" "*vi*" ╭──────────────┬───────────────────────────────────────────────────────────────────╮ │ install │ false │ │ log_level │ 1 │ │ exclude │ [emacs] │ │ repositories │ [example.com, foo.com] │ │ pkgs │ [vim, git, python, chrome, firefox, python, java, git, "python3"] │ ╰──────────────┴───────────────────────────────────────────────────────────────────╯ ```
2023-12-28 07:43:20 +00:00
FromValue, ShellError, Value,
2021-10-25 06:31:39 +00:00
};
2021-10-01 21:53:13 +00:00
pub trait CallExt {
Fix incorrect handling of boolean flags for builtin commands (#11492) # Description Possible fix of #11456 This PR fixes a bug where builtin commands did not respect the logic of dynamically passed boolean flags. The reason is [has_flag](https://github.com/nushell/nushell/blob/6f59abaf4310487f7a6319437be6ec61abcbc3b9/crates/nu-protocol/src/ast/call.rs#L204C5-L212C6) method did not evaluate and take into consideration expression used with flag. To address this issue a solution is proposed: 1. `has_flag` method is moved to `CallExt` and new logic to evaluate expression and check if it is a boolean value is added 2. `has_flag_const` method is added to `CallExt` which is a constant version of `has_flag` 3. `has_named` method is added to `Call` which is basically the old logic of `has_flag` 4. All usages of `has_flag` in code are updated, mostly to pass `engine_state` and `stack` to new `has_flag`. In `run_const` commands it is replaced with `has_flag_const`. And in a few select places: parser, `to nuon` and `into string` old logic via `has_named` is used. # User-Facing Changes Explicit values of boolean flags are now respected in builtin commands. Before: ![image](https://github.com/nushell/nushell/assets/17511668/f9fbabb2-3cfd-43f9-ba9e-ece76d80043c) After: ![image](https://github.com/nushell/nushell/assets/17511668/21867596-2075-437f-9c85-45563ac70083) Another example: Before: ![image](https://github.com/nushell/nushell/assets/17511668/efdbc5ca-5227-45a4-ac5b-532cdc2bbf5f) After: ![image](https://github.com/nushell/nushell/assets/17511668/2907d5c5-aa93-404d-af1c-21cdc3d44646) # Tests + Formatting Added test reproducing some variants of original issue.
2024-01-11 15:19:48 +00:00
/// Check if a boolean flag is set (i.e. `--bool` or `--bool=true`)
fn has_flag(
2021-10-01 21:53:13 +00:00
&self,
2021-10-25 06:31:39 +00:00
engine_state: &EngineState,
stack: &mut Stack,
Fix incorrect handling of boolean flags for builtin commands (#11492) # Description Possible fix of #11456 This PR fixes a bug where builtin commands did not respect the logic of dynamically passed boolean flags. The reason is [has_flag](https://github.com/nushell/nushell/blob/6f59abaf4310487f7a6319437be6ec61abcbc3b9/crates/nu-protocol/src/ast/call.rs#L204C5-L212C6) method did not evaluate and take into consideration expression used with flag. To address this issue a solution is proposed: 1. `has_flag` method is moved to `CallExt` and new logic to evaluate expression and check if it is a boolean value is added 2. `has_flag_const` method is added to `CallExt` which is a constant version of `has_flag` 3. `has_named` method is added to `Call` which is basically the old logic of `has_flag` 4. All usages of `has_flag` in code are updated, mostly to pass `engine_state` and `stack` to new `has_flag`. In `run_const` commands it is replaced with `has_flag_const`. And in a few select places: parser, `to nuon` and `into string` old logic via `has_named` is used. # User-Facing Changes Explicit values of boolean flags are now respected in builtin commands. Before: ![image](https://github.com/nushell/nushell/assets/17511668/f9fbabb2-3cfd-43f9-ba9e-ece76d80043c) After: ![image](https://github.com/nushell/nushell/assets/17511668/21867596-2075-437f-9c85-45563ac70083) Another example: Before: ![image](https://github.com/nushell/nushell/assets/17511668/efdbc5ca-5227-45a4-ac5b-532cdc2bbf5f) After: ![image](https://github.com/nushell/nushell/assets/17511668/2907d5c5-aa93-404d-af1c-21cdc3d44646) # Tests + Formatting Added test reproducing some variants of original issue.
2024-01-11 15:19:48 +00:00
flag_name: &str,
) -> Result<bool, ShellError>;
2021-10-01 21:53:13 +00:00
Fix incorrect handling of boolean flags for builtin commands (#11492) # Description Possible fix of #11456 This PR fixes a bug where builtin commands did not respect the logic of dynamically passed boolean flags. The reason is [has_flag](https://github.com/nushell/nushell/blob/6f59abaf4310487f7a6319437be6ec61abcbc3b9/crates/nu-protocol/src/ast/call.rs#L204C5-L212C6) method did not evaluate and take into consideration expression used with flag. To address this issue a solution is proposed: 1. `has_flag` method is moved to `CallExt` and new logic to evaluate expression and check if it is a boolean value is added 2. `has_flag_const` method is added to `CallExt` which is a constant version of `has_flag` 3. `has_named` method is added to `Call` which is basically the old logic of `has_flag` 4. All usages of `has_flag` in code are updated, mostly to pass `engine_state` and `stack` to new `has_flag`. In `run_const` commands it is replaced with `has_flag_const`. And in a few select places: parser, `to nuon` and `into string` old logic via `has_named` is used. # User-Facing Changes Explicit values of boolean flags are now respected in builtin commands. Before: ![image](https://github.com/nushell/nushell/assets/17511668/f9fbabb2-3cfd-43f9-ba9e-ece76d80043c) After: ![image](https://github.com/nushell/nushell/assets/17511668/21867596-2075-437f-9c85-45563ac70083) Another example: Before: ![image](https://github.com/nushell/nushell/assets/17511668/efdbc5ca-5227-45a4-ac5b-532cdc2bbf5f) After: ![image](https://github.com/nushell/nushell/assets/17511668/2907d5c5-aa93-404d-af1c-21cdc3d44646) # Tests + Formatting Added test reproducing some variants of original issue.
2024-01-11 15:19:48 +00:00
fn get_flag<T: FromValue>(
&self,
Fix incorrect handling of boolean flags for builtin commands (#11492) # Description Possible fix of #11456 This PR fixes a bug where builtin commands did not respect the logic of dynamically passed boolean flags. The reason is [has_flag](https://github.com/nushell/nushell/blob/6f59abaf4310487f7a6319437be6ec61abcbc3b9/crates/nu-protocol/src/ast/call.rs#L204C5-L212C6) method did not evaluate and take into consideration expression used with flag. To address this issue a solution is proposed: 1. `has_flag` method is moved to `CallExt` and new logic to evaluate expression and check if it is a boolean value is added 2. `has_flag_const` method is added to `CallExt` which is a constant version of `has_flag` 3. `has_named` method is added to `Call` which is basically the old logic of `has_flag` 4. All usages of `has_flag` in code are updated, mostly to pass `engine_state` and `stack` to new `has_flag`. In `run_const` commands it is replaced with `has_flag_const`. And in a few select places: parser, `to nuon` and `into string` old logic via `has_named` is used. # User-Facing Changes Explicit values of boolean flags are now respected in builtin commands. Before: ![image](https://github.com/nushell/nushell/assets/17511668/f9fbabb2-3cfd-43f9-ba9e-ece76d80043c) After: ![image](https://github.com/nushell/nushell/assets/17511668/21867596-2075-437f-9c85-45563ac70083) Another example: Before: ![image](https://github.com/nushell/nushell/assets/17511668/efdbc5ca-5227-45a4-ac5b-532cdc2bbf5f) After: ![image](https://github.com/nushell/nushell/assets/17511668/2907d5c5-aa93-404d-af1c-21cdc3d44646) # Tests + Formatting Added test reproducing some variants of original issue.
2024-01-11 15:19:48 +00:00
engine_state: &EngineState,
stack: &mut Stack,
name: &str,
) -> Result<Option<T>, ShellError>;
2021-10-01 21:53:13 +00:00
fn rest<T: FromValue>(
&self,
2021-10-25 06:31:39 +00:00
engine_state: &EngineState,
stack: &mut Stack,
2021-10-01 21:53:13 +00:00
starting_pos: usize,
) -> Result<Vec<T>, ShellError>;
2021-10-02 02:59:11 +00:00
fn opt<T: FromValue>(
&self,
2021-10-25 06:31:39 +00:00
engine_state: &EngineState,
stack: &mut Stack,
2021-10-02 02:59:11 +00:00
pos: usize,
) -> Result<Option<T>, ShellError>;
make the `ansi` command `const` (#11682) # Description This PR changes the `ansi` command to be a `const` command. - ~~It's breaking because I found that I had to change the way `ansi` is used in scripts a little bit. https://github.com/nushell/nu_scripts/pull/751~~ - I had to change one of the examples because apparently `const` can't be tested yet. - ~~I'm not sure this is right at all https://github.com/nushell/nushell/pull/11682/files#diff-ba932369a40eb40d6e1985eac1c784af403dab4500a7f0568e593900bf6cd740R654-R655. I just didn't want to duplicate a ton of code. Maybe if I duplicated the code it wouldn't be a breaking change because it would have a run and run_const?~~ - I had to add `opt_const` to CallExt. /cc @kubouch Can you take a look at this? I'm a little iffy if I'm doing this right, or even if we should do this at all. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-01-30 22:09:43 +00:00
fn opt_const<T: FromValue>(
&self,
working_set: &StateWorkingSet,
pos: usize,
) -> Result<Option<T>, ShellError>;
2021-10-25 06:31:39 +00:00
fn req<T: FromValue>(
&self,
engine_state: &EngineState,
stack: &mut Stack,
pos: usize,
) -> Result<T, ShellError>;
fn req_parser_info<T: FromValue>(
&self,
engine_state: &EngineState,
stack: &mut Stack,
name: &str,
) -> Result<T, ShellError>;
2021-10-01 21:53:13 +00:00
}
impl CallExt for Call {
Fix incorrect handling of boolean flags for builtin commands (#11492) # Description Possible fix of #11456 This PR fixes a bug where builtin commands did not respect the logic of dynamically passed boolean flags. The reason is [has_flag](https://github.com/nushell/nushell/blob/6f59abaf4310487f7a6319437be6ec61abcbc3b9/crates/nu-protocol/src/ast/call.rs#L204C5-L212C6) method did not evaluate and take into consideration expression used with flag. To address this issue a solution is proposed: 1. `has_flag` method is moved to `CallExt` and new logic to evaluate expression and check if it is a boolean value is added 2. `has_flag_const` method is added to `CallExt` which is a constant version of `has_flag` 3. `has_named` method is added to `Call` which is basically the old logic of `has_flag` 4. All usages of `has_flag` in code are updated, mostly to pass `engine_state` and `stack` to new `has_flag`. In `run_const` commands it is replaced with `has_flag_const`. And in a few select places: parser, `to nuon` and `into string` old logic via `has_named` is used. # User-Facing Changes Explicit values of boolean flags are now respected in builtin commands. Before: ![image](https://github.com/nushell/nushell/assets/17511668/f9fbabb2-3cfd-43f9-ba9e-ece76d80043c) After: ![image](https://github.com/nushell/nushell/assets/17511668/21867596-2075-437f-9c85-45563ac70083) Another example: Before: ![image](https://github.com/nushell/nushell/assets/17511668/efdbc5ca-5227-45a4-ac5b-532cdc2bbf5f) After: ![image](https://github.com/nushell/nushell/assets/17511668/2907d5c5-aa93-404d-af1c-21cdc3d44646) # Tests + Formatting Added test reproducing some variants of original issue.
2024-01-11 15:19:48 +00:00
fn has_flag(
2021-10-01 21:53:13 +00:00
&self,
2021-10-25 06:31:39 +00:00
engine_state: &EngineState,
stack: &mut Stack,
Fix incorrect handling of boolean flags for builtin commands (#11492) # Description Possible fix of #11456 This PR fixes a bug where builtin commands did not respect the logic of dynamically passed boolean flags. The reason is [has_flag](https://github.com/nushell/nushell/blob/6f59abaf4310487f7a6319437be6ec61abcbc3b9/crates/nu-protocol/src/ast/call.rs#L204C5-L212C6) method did not evaluate and take into consideration expression used with flag. To address this issue a solution is proposed: 1. `has_flag` method is moved to `CallExt` and new logic to evaluate expression and check if it is a boolean value is added 2. `has_flag_const` method is added to `CallExt` which is a constant version of `has_flag` 3. `has_named` method is added to `Call` which is basically the old logic of `has_flag` 4. All usages of `has_flag` in code are updated, mostly to pass `engine_state` and `stack` to new `has_flag`. In `run_const` commands it is replaced with `has_flag_const`. And in a few select places: parser, `to nuon` and `into string` old logic via `has_named` is used. # User-Facing Changes Explicit values of boolean flags are now respected in builtin commands. Before: ![image](https://github.com/nushell/nushell/assets/17511668/f9fbabb2-3cfd-43f9-ba9e-ece76d80043c) After: ![image](https://github.com/nushell/nushell/assets/17511668/21867596-2075-437f-9c85-45563ac70083) Another example: Before: ![image](https://github.com/nushell/nushell/assets/17511668/efdbc5ca-5227-45a4-ac5b-532cdc2bbf5f) After: ![image](https://github.com/nushell/nushell/assets/17511668/2907d5c5-aa93-404d-af1c-21cdc3d44646) # Tests + Formatting Added test reproducing some variants of original issue.
2024-01-11 15:19:48 +00:00
flag_name: &str,
) -> Result<bool, ShellError> {
for name in self.named_iter() {
if flag_name == name.0.item {
return if let Some(expr) = &name.2 {
// Check --flag=false
let stack = &mut stack.use_call_arg_out_dest();
Debugger experiments (#11441) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR adds a new evaluator path with callbacks to a mutable trait object implementing a Debugger trait. The trait object can do anything, e.g., profiling, code coverage, step debugging. Currently, entering/leaving a block and a pipeline element is marked with callbacks, but more callbacks can be added as necessary. Not all callbacks need to be used by all debuggers; unused ones are simply empty calls. A simple profiler is implemented as a proof of concept. The debugging support is implementing by making `eval_xxx()` functions generic depending on whether we're debugging or not. This has zero computational overhead, but makes the binary slightly larger (see benchmarks below). `eval_xxx()` variants called from commands (like `eval_block_with_early_return()` in `each`) are chosen with a dynamic dispatch for two reasons: to not grow the binary size due to duplicating the code of many commands, and for the fact that it isn't possible because it would make Command trait objects object-unsafe. In the future, I hope it will be possible to allow plugin callbacks such that users would be able to implement their profiler plugins instead of having to recompile Nushell. [DAP](https://microsoft.github.io/debug-adapter-protocol/) would also be interesting to explore. Try `help debug profile`. ## Screenshots Basic output: ![profiler_new](https://github.com/nushell/nushell/assets/25571562/418b9df0-b659-4dcb-b023-2d5fcef2c865) To profile with more granularity, increase the profiler depth (you'll see that repeated `is-windows` calls take a large chunk of total time, making it a good candidate for optimizing): ![profiler_new_m3](https://github.com/nushell/nushell/assets/25571562/636d756d-5d56-460c-a372-14716f65f37f) ## Benchmarks ### Binary size Binary size increase vs. main: **+40360 bytes**. _(Both built with `--release --features=extra,dataframe`.)_ ### Time ```nushell # bench_debug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'debug:' let res2 = bench { debug profile $test } --pretty print $res2 ``` ```nushell # bench_nodebug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'no debug:' let res1 = bench { do $test } --pretty print $res1 ``` `cargo run --release -- bench_debug.nu` is consistently 1--2 ms slower than `cargo run --release -- bench_nodebug.nu` due to the collection overhead + gathering the report. This is expected. When gathering more stuff, the overhead is obviously higher. `cargo run --release -- bench_nodebug.nu` vs. `nu bench_nodebug.nu` I didn't measure any difference. Both benchmarks report times between 97 and 103 ms randomly, without one being consistently higher than the other. This suggests that at least in this particular case, when not running any debugger, there is no runtime overhead. ## API changes This PR adds a generic parameter to all `eval_xxx` functions that forces you to specify whether you use the debugger. You can resolve it in two ways: * Use a provided helper that will figure it out for you. If you wanted to use `eval_block(&engine_state, ...)`, call `let eval_block = get_eval_block(&engine_state); eval_block(&engine_state, ...)` * If you know you're in an evaluation path that doesn't need debugger support, call `eval_block::<WithoutDebug>(&engine_state, ...)` (this is the case of hooks, for example). I tried to add more explanation in the docstring of `debugger_trait.rs`. ## TODO - [x] Better profiler output to reduce spam of iterative commands like `each` - [x] Resolve `TODO: DEBUG` comments - [x] Resolve unwraps - [x] Add doc comments - [x] Add usage and extra usage for `debug profile`, explaining all columns # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Hopefully none. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-03-08 18:21:35 +00:00
let result = eval_expression::<WithoutDebug>(engine_state, stack, expr)?;
Fix incorrect handling of boolean flags for builtin commands (#11492) # Description Possible fix of #11456 This PR fixes a bug where builtin commands did not respect the logic of dynamically passed boolean flags. The reason is [has_flag](https://github.com/nushell/nushell/blob/6f59abaf4310487f7a6319437be6ec61abcbc3b9/crates/nu-protocol/src/ast/call.rs#L204C5-L212C6) method did not evaluate and take into consideration expression used with flag. To address this issue a solution is proposed: 1. `has_flag` method is moved to `CallExt` and new logic to evaluate expression and check if it is a boolean value is added 2. `has_flag_const` method is added to `CallExt` which is a constant version of `has_flag` 3. `has_named` method is added to `Call` which is basically the old logic of `has_flag` 4. All usages of `has_flag` in code are updated, mostly to pass `engine_state` and `stack` to new `has_flag`. In `run_const` commands it is replaced with `has_flag_const`. And in a few select places: parser, `to nuon` and `into string` old logic via `has_named` is used. # User-Facing Changes Explicit values of boolean flags are now respected in builtin commands. Before: ![image](https://github.com/nushell/nushell/assets/17511668/f9fbabb2-3cfd-43f9-ba9e-ece76d80043c) After: ![image](https://github.com/nushell/nushell/assets/17511668/21867596-2075-437f-9c85-45563ac70083) Another example: Before: ![image](https://github.com/nushell/nushell/assets/17511668/efdbc5ca-5227-45a4-ac5b-532cdc2bbf5f) After: ![image](https://github.com/nushell/nushell/assets/17511668/2907d5c5-aa93-404d-af1c-21cdc3d44646) # Tests + Formatting Added test reproducing some variants of original issue.
2024-01-11 15:19:48 +00:00
match result {
Value::Bool { val, .. } => Ok(val),
_ => Err(ShellError::CantConvert {
to_type: "bool".into(),
from_type: result.get_type().to_string(),
span: result.span(),
help: Some("".into()),
}),
}
} else {
Ok(true)
};
}
2021-10-01 21:53:13 +00:00
}
Fix incorrect handling of boolean flags for builtin commands (#11492) # Description Possible fix of #11456 This PR fixes a bug where builtin commands did not respect the logic of dynamically passed boolean flags. The reason is [has_flag](https://github.com/nushell/nushell/blob/6f59abaf4310487f7a6319437be6ec61abcbc3b9/crates/nu-protocol/src/ast/call.rs#L204C5-L212C6) method did not evaluate and take into consideration expression used with flag. To address this issue a solution is proposed: 1. `has_flag` method is moved to `CallExt` and new logic to evaluate expression and check if it is a boolean value is added 2. `has_flag_const` method is added to `CallExt` which is a constant version of `has_flag` 3. `has_named` method is added to `Call` which is basically the old logic of `has_flag` 4. All usages of `has_flag` in code are updated, mostly to pass `engine_state` and `stack` to new `has_flag`. In `run_const` commands it is replaced with `has_flag_const`. And in a few select places: parser, `to nuon` and `into string` old logic via `has_named` is used. # User-Facing Changes Explicit values of boolean flags are now respected in builtin commands. Before: ![image](https://github.com/nushell/nushell/assets/17511668/f9fbabb2-3cfd-43f9-ba9e-ece76d80043c) After: ![image](https://github.com/nushell/nushell/assets/17511668/21867596-2075-437f-9c85-45563ac70083) Another example: Before: ![image](https://github.com/nushell/nushell/assets/17511668/efdbc5ca-5227-45a4-ac5b-532cdc2bbf5f) After: ![image](https://github.com/nushell/nushell/assets/17511668/2907d5c5-aa93-404d-af1c-21cdc3d44646) # Tests + Formatting Added test reproducing some variants of original issue.
2024-01-11 15:19:48 +00:00
Ok(false)
2021-10-01 21:53:13 +00:00
}
Fix incorrect handling of boolean flags for builtin commands (#11492) # Description Possible fix of #11456 This PR fixes a bug where builtin commands did not respect the logic of dynamically passed boolean flags. The reason is [has_flag](https://github.com/nushell/nushell/blob/6f59abaf4310487f7a6319437be6ec61abcbc3b9/crates/nu-protocol/src/ast/call.rs#L204C5-L212C6) method did not evaluate and take into consideration expression used with flag. To address this issue a solution is proposed: 1. `has_flag` method is moved to `CallExt` and new logic to evaluate expression and check if it is a boolean value is added 2. `has_flag_const` method is added to `CallExt` which is a constant version of `has_flag` 3. `has_named` method is added to `Call` which is basically the old logic of `has_flag` 4. All usages of `has_flag` in code are updated, mostly to pass `engine_state` and `stack` to new `has_flag`. In `run_const` commands it is replaced with `has_flag_const`. And in a few select places: parser, `to nuon` and `into string` old logic via `has_named` is used. # User-Facing Changes Explicit values of boolean flags are now respected in builtin commands. Before: ![image](https://github.com/nushell/nushell/assets/17511668/f9fbabb2-3cfd-43f9-ba9e-ece76d80043c) After: ![image](https://github.com/nushell/nushell/assets/17511668/21867596-2075-437f-9c85-45563ac70083) Another example: Before: ![image](https://github.com/nushell/nushell/assets/17511668/efdbc5ca-5227-45a4-ac5b-532cdc2bbf5f) After: ![image](https://github.com/nushell/nushell/assets/17511668/2907d5c5-aa93-404d-af1c-21cdc3d44646) # Tests + Formatting Added test reproducing some variants of original issue.
2024-01-11 15:19:48 +00:00
fn get_flag<T: FromValue>(
&self,
Fix incorrect handling of boolean flags for builtin commands (#11492) # Description Possible fix of #11456 This PR fixes a bug where builtin commands did not respect the logic of dynamically passed boolean flags. The reason is [has_flag](https://github.com/nushell/nushell/blob/6f59abaf4310487f7a6319437be6ec61abcbc3b9/crates/nu-protocol/src/ast/call.rs#L204C5-L212C6) method did not evaluate and take into consideration expression used with flag. To address this issue a solution is proposed: 1. `has_flag` method is moved to `CallExt` and new logic to evaluate expression and check if it is a boolean value is added 2. `has_flag_const` method is added to `CallExt` which is a constant version of `has_flag` 3. `has_named` method is added to `Call` which is basically the old logic of `has_flag` 4. All usages of `has_flag` in code are updated, mostly to pass `engine_state` and `stack` to new `has_flag`. In `run_const` commands it is replaced with `has_flag_const`. And in a few select places: parser, `to nuon` and `into string` old logic via `has_named` is used. # User-Facing Changes Explicit values of boolean flags are now respected in builtin commands. Before: ![image](https://github.com/nushell/nushell/assets/17511668/f9fbabb2-3cfd-43f9-ba9e-ece76d80043c) After: ![image](https://github.com/nushell/nushell/assets/17511668/21867596-2075-437f-9c85-45563ac70083) Another example: Before: ![image](https://github.com/nushell/nushell/assets/17511668/efdbc5ca-5227-45a4-ac5b-532cdc2bbf5f) After: ![image](https://github.com/nushell/nushell/assets/17511668/2907d5c5-aa93-404d-af1c-21cdc3d44646) # Tests + Formatting Added test reproducing some variants of original issue.
2024-01-11 15:19:48 +00:00
engine_state: &EngineState,
stack: &mut Stack,
name: &str,
) -> Result<Option<T>, ShellError> {
if let Some(expr) = self.get_flag_expr(name) {
let stack = &mut stack.use_call_arg_out_dest();
Debugger experiments (#11441) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR adds a new evaluator path with callbacks to a mutable trait object implementing a Debugger trait. The trait object can do anything, e.g., profiling, code coverage, step debugging. Currently, entering/leaving a block and a pipeline element is marked with callbacks, but more callbacks can be added as necessary. Not all callbacks need to be used by all debuggers; unused ones are simply empty calls. A simple profiler is implemented as a proof of concept. The debugging support is implementing by making `eval_xxx()` functions generic depending on whether we're debugging or not. This has zero computational overhead, but makes the binary slightly larger (see benchmarks below). `eval_xxx()` variants called from commands (like `eval_block_with_early_return()` in `each`) are chosen with a dynamic dispatch for two reasons: to not grow the binary size due to duplicating the code of many commands, and for the fact that it isn't possible because it would make Command trait objects object-unsafe. In the future, I hope it will be possible to allow plugin callbacks such that users would be able to implement their profiler plugins instead of having to recompile Nushell. [DAP](https://microsoft.github.io/debug-adapter-protocol/) would also be interesting to explore. Try `help debug profile`. ## Screenshots Basic output: ![profiler_new](https://github.com/nushell/nushell/assets/25571562/418b9df0-b659-4dcb-b023-2d5fcef2c865) To profile with more granularity, increase the profiler depth (you'll see that repeated `is-windows` calls take a large chunk of total time, making it a good candidate for optimizing): ![profiler_new_m3](https://github.com/nushell/nushell/assets/25571562/636d756d-5d56-460c-a372-14716f65f37f) ## Benchmarks ### Binary size Binary size increase vs. main: **+40360 bytes**. _(Both built with `--release --features=extra,dataframe`.)_ ### Time ```nushell # bench_debug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'debug:' let res2 = bench { debug profile $test } --pretty print $res2 ``` ```nushell # bench_nodebug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'no debug:' let res1 = bench { do $test } --pretty print $res1 ``` `cargo run --release -- bench_debug.nu` is consistently 1--2 ms slower than `cargo run --release -- bench_nodebug.nu` due to the collection overhead + gathering the report. This is expected. When gathering more stuff, the overhead is obviously higher. `cargo run --release -- bench_nodebug.nu` vs. `nu bench_nodebug.nu` I didn't measure any difference. Both benchmarks report times between 97 and 103 ms randomly, without one being consistently higher than the other. This suggests that at least in this particular case, when not running any debugger, there is no runtime overhead. ## API changes This PR adds a generic parameter to all `eval_xxx` functions that forces you to specify whether you use the debugger. You can resolve it in two ways: * Use a provided helper that will figure it out for you. If you wanted to use `eval_block(&engine_state, ...)`, call `let eval_block = get_eval_block(&engine_state); eval_block(&engine_state, ...)` * If you know you're in an evaluation path that doesn't need debugger support, call `eval_block::<WithoutDebug>(&engine_state, ...)` (this is the case of hooks, for example). I tried to add more explanation in the docstring of `debugger_trait.rs`. ## TODO - [x] Better profiler output to reduce spam of iterative commands like `each` - [x] Resolve `TODO: DEBUG` comments - [x] Resolve unwraps - [x] Add doc comments - [x] Add usage and extra usage for `debug profile`, explaining all columns # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Hopefully none. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-03-08 18:21:35 +00:00
let result = eval_expression::<WithoutDebug>(engine_state, stack, expr)?;
FromValue::from_value(result).map(Some)
} else {
Ok(None)
}
}
2021-10-01 21:53:13 +00:00
fn rest<T: FromValue>(
&self,
2021-10-25 06:31:39 +00:00
engine_state: &EngineState,
stack: &mut Stack,
2021-10-01 21:53:13 +00:00
starting_pos: usize,
) -> Result<Vec<T>, ShellError> {
let stack = &mut stack.use_call_arg_out_dest();
2021-10-01 21:53:13 +00:00
let mut output = vec![];
Allow spreading arguments to commands (#11289) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> Finishes implementing https://github.com/nushell/nushell/issues/10598, which asks for a spread operator in lists, in records, and when calling commands. # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR will allow spreading arguments to commands (both internal and external). It will also deprecate spreading arguments automatically when passing to external commands. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> - Users will be able to use `...` to spread arguments to custom/builtin commands that have rest parameters or allow unknown arguments, or to any external command - If a custom command doesn't have a rest parameter and it doesn't allow unknown arguments either, the spread operator will not be allowed - Passing lists to external commands without `...` will work for now but will cause a deprecation warning saying that it'll stop working in 0.91 (is 2 versions enough time?) Here's a function to help with demonstrating some behavior: ```nushell > def foo [ a, b, c?, d?, ...rest ] { [$a $b $c $d $rest] | to nuon } ``` You can pass a list of arguments to fill in the `rest` parameter using `...`: ```nushell > foo 1 2 3 4 ...[5 6] [1, 2, 3, 4, [5, 6]] ``` If you don't use `...`, the list `[5 6]` will be treated as a single argument: ```nushell > foo 1 2 3 4 [5 6] # Note the double [[]] [1, 2, 3, 4, [[5, 6]]] ``` You can omit optional parameters before the spread arguments: ```nushell > foo 1 2 3 ...[4 5] # d is omitted here [1, 2, 3, null, [4, 5]] ``` If you have multiple lists, you can spread them all: ```nushell > foo 1 2 3 ...[4 5] 6 7 ...[8] ...[] [1, 2, 3, null, [4, 5, 6, 7, 8]] ``` Here's the kind of error you get when you try to spread arguments to a command with no rest parameter: ![image](https://github.com/nushell/nushell/assets/45539777/93faceae-00eb-4e59-ac3f-17f98436e6e4) And this is the warning you get when you pass a list to an external now (without `...`): ![image](https://github.com/nushell/nushell/assets/45539777/d368f590-201e-49fb-8b20-68476ced415e) # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> Added tests to cover the following cases: - Spreading arguments to a command that doesn't have a rest parameter (unexpected spread argument error) - Spreading arguments to a command that doesn't have a rest parameter *but* there's also a missing positional argument (missing positional error) - Spreading arguments to a command that doesn't have a rest parameter but does allow unknown arguments, such as `exec` (allowed) - Spreading a list literal containing arguments of the wrong type (parse error) - Spreading a non-list value, both to internal and external commands - Having named arguments in the middle of rest arguments - `explain`ing a command call that spreads its arguments # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> # Examples Suppose you have multiple tables: ```nushell let people = [[id name age]; [0 alice 100] [1 bob 200] [2 eve 300]] let evil_twins = [[id name age]; [0 ecila 100] [-1 bob 200] [-2 eve 300]] ``` Maybe you often find yourself needing to merge multiple tables and want a utility to do that. You could write a function like this: ```nushell def merge_all [ ...tables ] { $tables | reduce { |it, acc| $acc | merge $it } } ``` Then you can use it like this: ```nushell > merge_all ...([$people $evil_twins] | each { |$it| $it | select name age }) ╭───┬───────┬─────╮ │ # │ name │ age │ ├───┼───────┼─────┤ │ 0 │ ecila │ 100 │ │ 1 │ bob │ 200 │ │ 2 │ eve │ 300 │ ╰───┴───────┴─────╯ ``` Except they had duplicate columns, so now you first want to suffix every column with a number to tell you which table the column came from. You can make a command for that: ```nushell def select_and_merge [ --cols: list<string>, ...tables ] { let renamed_tables = $tables | enumerate | each { |it| $it.item | select $cols | rename ...($cols | each { |col| $col + ($it.index | into string) }) }; merge_all ...$renamed_tables } ``` And call it like this: ```nushell > select_and_merge --cols [name age] $people $evil_twins ╭───┬───────┬──────┬───────┬──────╮ │ # │ name0 │ age0 │ name1 │ age1 │ ├───┼───────┼──────┼───────┼──────┤ │ 0 │ alice │ 100 │ ecila │ 100 │ │ 1 │ bob │ 200 │ bob │ 200 │ │ 2 │ eve │ 300 │ eve │ 300 │ ╰───┴───────┴──────┴───────┴──────╯ ``` --- Suppose someone's made a command to search for APT packages: ```nushell # The main command def search-pkgs [ --install # Whether to install any packages it finds log_level: int # Pretend it's a good idea to make this a required positional parameter exclude?: list<string> # Packages to exclude repositories?: list<string> # Which repositories to look in (searches in all if not given) ...pkgs # Package names to search for ] { { install: $install, log_level: $log_level, exclude: ($exclude | to nuon), repositories: ($repositories | to nuon), pkgs: ($pkgs | to nuon) } } ``` It has a lot of parameters to configure it, so you might make your own helper commands to wrap around it for specific cases. Here's one example: ```nushell # Only look for packages locally def search-pkgs-local [ --install # Whether to install any packages it finds log_level: int exclude?: list<string> # Packages to exclude ...pkgs # Package names to search for ] { # All required and optional positional parameters are given search-pkgs --install=$install $log_level [] ["<local URI or something>"] ...$pkgs } ``` And you can run it like this: ```nushell > search-pkgs-local --install=false 5 ...["python2.7" "vim"] ╭──────────────┬──────────────────────────────╮ │ install │ false │ │ log_level │ 5 │ │ exclude │ [] │ │ repositories │ ["<local URI or something>"] │ │ pkgs │ ["python2.7", vim] │ ╰──────────────┴──────────────────────────────╯ ``` One thing I realized when writing this was that if we decide to not allow passing optional arguments using the spread operator, then you can (mis?)use the spread operator to skip optional parameters. Here, I didn't want to give `exclude` explicitly, so I used a spread operator to pass the packages to install. Without it, I would've needed to do `search-pkgs-local --install=false 5 [] "python2.7" "vim"` (explicitly pass `[]` (or `null`, in the general case) to `exclude`). There are probably more idiomatic ways to do this, but I just thought it was something interesting. If you're a virologist of the [xkcd](https://xkcd.com/350/) kind, another helper command you might make is this: ```nushell # Install any packages it finds def live-dangerously [ ...pkgs ] { # One optional argument was given (exclude), while another was not (repositories) search-pkgs 0 [] ...$pkgs --install # Flags can go after spread arguments } ``` Running it: ```nushell > live-dangerously "git" "*vi*" # *vi* because I don't feel like typing out vim and neovim ╭──────────────┬─────────────╮ │ install │ true │ │ log_level │ 0 │ │ exclude │ [] │ │ repositories │ null │ │ pkgs │ [git, *vi*] │ ╰──────────────┴─────────────╯ ``` Here's an example that uses the spread operator more than once within the same command call: ```nushell let extras = [ chrome firefox python java git ] def search-pkgs-curated [ ...pkgs ] { (search-pkgs 1 [emacs] ["example.com", "foo.com"] vim # A must for everyone! ...($pkgs | filter { |p| not ($p | str contains "*") }) # Remove packages with globs python # Good tool to have ...$extras --install=false python3) # I forget, did I already put Python in extras? } ``` Running it: ```nushell > search-pkgs-curated "git" "*vi*" ╭──────────────┬───────────────────────────────────────────────────────────────────╮ │ install │ false │ │ log_level │ 1 │ │ exclude │ [emacs] │ │ repositories │ [example.com, foo.com] │ │ pkgs │ [vim, git, python, chrome, firefox, python, java, git, "python3"] │ ╰──────────────┴───────────────────────────────────────────────────────────────────╯ ```
2023-12-28 07:43:20 +00:00
for result in self.rest_iter_flattened(starting_pos, |expr| {
Debugger experiments (#11441) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR adds a new evaluator path with callbacks to a mutable trait object implementing a Debugger trait. The trait object can do anything, e.g., profiling, code coverage, step debugging. Currently, entering/leaving a block and a pipeline element is marked with callbacks, but more callbacks can be added as necessary. Not all callbacks need to be used by all debuggers; unused ones are simply empty calls. A simple profiler is implemented as a proof of concept. The debugging support is implementing by making `eval_xxx()` functions generic depending on whether we're debugging or not. This has zero computational overhead, but makes the binary slightly larger (see benchmarks below). `eval_xxx()` variants called from commands (like `eval_block_with_early_return()` in `each`) are chosen with a dynamic dispatch for two reasons: to not grow the binary size due to duplicating the code of many commands, and for the fact that it isn't possible because it would make Command trait objects object-unsafe. In the future, I hope it will be possible to allow plugin callbacks such that users would be able to implement their profiler plugins instead of having to recompile Nushell. [DAP](https://microsoft.github.io/debug-adapter-protocol/) would also be interesting to explore. Try `help debug profile`. ## Screenshots Basic output: ![profiler_new](https://github.com/nushell/nushell/assets/25571562/418b9df0-b659-4dcb-b023-2d5fcef2c865) To profile with more granularity, increase the profiler depth (you'll see that repeated `is-windows` calls take a large chunk of total time, making it a good candidate for optimizing): ![profiler_new_m3](https://github.com/nushell/nushell/assets/25571562/636d756d-5d56-460c-a372-14716f65f37f) ## Benchmarks ### Binary size Binary size increase vs. main: **+40360 bytes**. _(Both built with `--release --features=extra,dataframe`.)_ ### Time ```nushell # bench_debug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'debug:' let res2 = bench { debug profile $test } --pretty print $res2 ``` ```nushell # bench_nodebug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'no debug:' let res1 = bench { do $test } --pretty print $res1 ``` `cargo run --release -- bench_debug.nu` is consistently 1--2 ms slower than `cargo run --release -- bench_nodebug.nu` due to the collection overhead + gathering the report. This is expected. When gathering more stuff, the overhead is obviously higher. `cargo run --release -- bench_nodebug.nu` vs. `nu bench_nodebug.nu` I didn't measure any difference. Both benchmarks report times between 97 and 103 ms randomly, without one being consistently higher than the other. This suggests that at least in this particular case, when not running any debugger, there is no runtime overhead. ## API changes This PR adds a generic parameter to all `eval_xxx` functions that forces you to specify whether you use the debugger. You can resolve it in two ways: * Use a provided helper that will figure it out for you. If you wanted to use `eval_block(&engine_state, ...)`, call `let eval_block = get_eval_block(&engine_state); eval_block(&engine_state, ...)` * If you know you're in an evaluation path that doesn't need debugger support, call `eval_block::<WithoutDebug>(&engine_state, ...)` (this is the case of hooks, for example). I tried to add more explanation in the docstring of `debugger_trait.rs`. ## TODO - [x] Better profiler output to reduce spam of iterative commands like `each` - [x] Resolve `TODO: DEBUG` comments - [x] Resolve unwraps - [x] Add doc comments - [x] Add usage and extra usage for `debug profile`, explaining all columns # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Hopefully none. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-03-08 18:21:35 +00:00
eval_expression::<WithoutDebug>(engine_state, stack, expr)
Allow spreading arguments to commands (#11289) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> Finishes implementing https://github.com/nushell/nushell/issues/10598, which asks for a spread operator in lists, in records, and when calling commands. # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR will allow spreading arguments to commands (both internal and external). It will also deprecate spreading arguments automatically when passing to external commands. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> - Users will be able to use `...` to spread arguments to custom/builtin commands that have rest parameters or allow unknown arguments, or to any external command - If a custom command doesn't have a rest parameter and it doesn't allow unknown arguments either, the spread operator will not be allowed - Passing lists to external commands without `...` will work for now but will cause a deprecation warning saying that it'll stop working in 0.91 (is 2 versions enough time?) Here's a function to help with demonstrating some behavior: ```nushell > def foo [ a, b, c?, d?, ...rest ] { [$a $b $c $d $rest] | to nuon } ``` You can pass a list of arguments to fill in the `rest` parameter using `...`: ```nushell > foo 1 2 3 4 ...[5 6] [1, 2, 3, 4, [5, 6]] ``` If you don't use `...`, the list `[5 6]` will be treated as a single argument: ```nushell > foo 1 2 3 4 [5 6] # Note the double [[]] [1, 2, 3, 4, [[5, 6]]] ``` You can omit optional parameters before the spread arguments: ```nushell > foo 1 2 3 ...[4 5] # d is omitted here [1, 2, 3, null, [4, 5]] ``` If you have multiple lists, you can spread them all: ```nushell > foo 1 2 3 ...[4 5] 6 7 ...[8] ...[] [1, 2, 3, null, [4, 5, 6, 7, 8]] ``` Here's the kind of error you get when you try to spread arguments to a command with no rest parameter: ![image](https://github.com/nushell/nushell/assets/45539777/93faceae-00eb-4e59-ac3f-17f98436e6e4) And this is the warning you get when you pass a list to an external now (without `...`): ![image](https://github.com/nushell/nushell/assets/45539777/d368f590-201e-49fb-8b20-68476ced415e) # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> Added tests to cover the following cases: - Spreading arguments to a command that doesn't have a rest parameter (unexpected spread argument error) - Spreading arguments to a command that doesn't have a rest parameter *but* there's also a missing positional argument (missing positional error) - Spreading arguments to a command that doesn't have a rest parameter but does allow unknown arguments, such as `exec` (allowed) - Spreading a list literal containing arguments of the wrong type (parse error) - Spreading a non-list value, both to internal and external commands - Having named arguments in the middle of rest arguments - `explain`ing a command call that spreads its arguments # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> # Examples Suppose you have multiple tables: ```nushell let people = [[id name age]; [0 alice 100] [1 bob 200] [2 eve 300]] let evil_twins = [[id name age]; [0 ecila 100] [-1 bob 200] [-2 eve 300]] ``` Maybe you often find yourself needing to merge multiple tables and want a utility to do that. You could write a function like this: ```nushell def merge_all [ ...tables ] { $tables | reduce { |it, acc| $acc | merge $it } } ``` Then you can use it like this: ```nushell > merge_all ...([$people $evil_twins] | each { |$it| $it | select name age }) ╭───┬───────┬─────╮ │ # │ name │ age │ ├───┼───────┼─────┤ │ 0 │ ecila │ 100 │ │ 1 │ bob │ 200 │ │ 2 │ eve │ 300 │ ╰───┴───────┴─────╯ ``` Except they had duplicate columns, so now you first want to suffix every column with a number to tell you which table the column came from. You can make a command for that: ```nushell def select_and_merge [ --cols: list<string>, ...tables ] { let renamed_tables = $tables | enumerate | each { |it| $it.item | select $cols | rename ...($cols | each { |col| $col + ($it.index | into string) }) }; merge_all ...$renamed_tables } ``` And call it like this: ```nushell > select_and_merge --cols [name age] $people $evil_twins ╭───┬───────┬──────┬───────┬──────╮ │ # │ name0 │ age0 │ name1 │ age1 │ ├───┼───────┼──────┼───────┼──────┤ │ 0 │ alice │ 100 │ ecila │ 100 │ │ 1 │ bob │ 200 │ bob │ 200 │ │ 2 │ eve │ 300 │ eve │ 300 │ ╰───┴───────┴──────┴───────┴──────╯ ``` --- Suppose someone's made a command to search for APT packages: ```nushell # The main command def search-pkgs [ --install # Whether to install any packages it finds log_level: int # Pretend it's a good idea to make this a required positional parameter exclude?: list<string> # Packages to exclude repositories?: list<string> # Which repositories to look in (searches in all if not given) ...pkgs # Package names to search for ] { { install: $install, log_level: $log_level, exclude: ($exclude | to nuon), repositories: ($repositories | to nuon), pkgs: ($pkgs | to nuon) } } ``` It has a lot of parameters to configure it, so you might make your own helper commands to wrap around it for specific cases. Here's one example: ```nushell # Only look for packages locally def search-pkgs-local [ --install # Whether to install any packages it finds log_level: int exclude?: list<string> # Packages to exclude ...pkgs # Package names to search for ] { # All required and optional positional parameters are given search-pkgs --install=$install $log_level [] ["<local URI or something>"] ...$pkgs } ``` And you can run it like this: ```nushell > search-pkgs-local --install=false 5 ...["python2.7" "vim"] ╭──────────────┬──────────────────────────────╮ │ install │ false │ │ log_level │ 5 │ │ exclude │ [] │ │ repositories │ ["<local URI or something>"] │ │ pkgs │ ["python2.7", vim] │ ╰──────────────┴──────────────────────────────╯ ``` One thing I realized when writing this was that if we decide to not allow passing optional arguments using the spread operator, then you can (mis?)use the spread operator to skip optional parameters. Here, I didn't want to give `exclude` explicitly, so I used a spread operator to pass the packages to install. Without it, I would've needed to do `search-pkgs-local --install=false 5 [] "python2.7" "vim"` (explicitly pass `[]` (or `null`, in the general case) to `exclude`). There are probably more idiomatic ways to do this, but I just thought it was something interesting. If you're a virologist of the [xkcd](https://xkcd.com/350/) kind, another helper command you might make is this: ```nushell # Install any packages it finds def live-dangerously [ ...pkgs ] { # One optional argument was given (exclude), while another was not (repositories) search-pkgs 0 [] ...$pkgs --install # Flags can go after spread arguments } ``` Running it: ```nushell > live-dangerously "git" "*vi*" # *vi* because I don't feel like typing out vim and neovim ╭──────────────┬─────────────╮ │ install │ true │ │ log_level │ 0 │ │ exclude │ [] │ │ repositories │ null │ │ pkgs │ [git, *vi*] │ ╰──────────────┴─────────────╯ ``` Here's an example that uses the spread operator more than once within the same command call: ```nushell let extras = [ chrome firefox python java git ] def search-pkgs-curated [ ...pkgs ] { (search-pkgs 1 [emacs] ["example.com", "foo.com"] vim # A must for everyone! ...($pkgs | filter { |p| not ($p | str contains "*") }) # Remove packages with globs python # Good tool to have ...$extras --install=false python3) # I forget, did I already put Python in extras? } ``` Running it: ```nushell > search-pkgs-curated "git" "*vi*" ╭──────────────┬───────────────────────────────────────────────────────────────────╮ │ install │ false │ │ log_level │ 1 │ │ exclude │ [emacs] │ │ repositories │ [example.com, foo.com] │ │ pkgs │ [vim, git, python, chrome, firefox, python, java, git, "python3"] │ ╰──────────────┴───────────────────────────────────────────────────────────────────╯ ```
2023-12-28 07:43:20 +00:00
})? {
output.push(FromValue::from_value(result)?);
2021-10-01 21:53:13 +00:00
}
Ok(output)
}
2021-10-02 02:59:11 +00:00
fn opt<T: FromValue>(
&self,
2021-10-25 06:31:39 +00:00
engine_state: &EngineState,
stack: &mut Stack,
2021-10-02 02:59:11 +00:00
pos: usize,
) -> Result<Option<T>, ShellError> {
if let Some(expr) = self.positional_nth(pos) {
let stack = &mut stack.use_call_arg_out_dest();
Debugger experiments (#11441) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR adds a new evaluator path with callbacks to a mutable trait object implementing a Debugger trait. The trait object can do anything, e.g., profiling, code coverage, step debugging. Currently, entering/leaving a block and a pipeline element is marked with callbacks, but more callbacks can be added as necessary. Not all callbacks need to be used by all debuggers; unused ones are simply empty calls. A simple profiler is implemented as a proof of concept. The debugging support is implementing by making `eval_xxx()` functions generic depending on whether we're debugging or not. This has zero computational overhead, but makes the binary slightly larger (see benchmarks below). `eval_xxx()` variants called from commands (like `eval_block_with_early_return()` in `each`) are chosen with a dynamic dispatch for two reasons: to not grow the binary size due to duplicating the code of many commands, and for the fact that it isn't possible because it would make Command trait objects object-unsafe. In the future, I hope it will be possible to allow plugin callbacks such that users would be able to implement their profiler plugins instead of having to recompile Nushell. [DAP](https://microsoft.github.io/debug-adapter-protocol/) would also be interesting to explore. Try `help debug profile`. ## Screenshots Basic output: ![profiler_new](https://github.com/nushell/nushell/assets/25571562/418b9df0-b659-4dcb-b023-2d5fcef2c865) To profile with more granularity, increase the profiler depth (you'll see that repeated `is-windows` calls take a large chunk of total time, making it a good candidate for optimizing): ![profiler_new_m3](https://github.com/nushell/nushell/assets/25571562/636d756d-5d56-460c-a372-14716f65f37f) ## Benchmarks ### Binary size Binary size increase vs. main: **+40360 bytes**. _(Both built with `--release --features=extra,dataframe`.)_ ### Time ```nushell # bench_debug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'debug:' let res2 = bench { debug profile $test } --pretty print $res2 ``` ```nushell # bench_nodebug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'no debug:' let res1 = bench { do $test } --pretty print $res1 ``` `cargo run --release -- bench_debug.nu` is consistently 1--2 ms slower than `cargo run --release -- bench_nodebug.nu` due to the collection overhead + gathering the report. This is expected. When gathering more stuff, the overhead is obviously higher. `cargo run --release -- bench_nodebug.nu` vs. `nu bench_nodebug.nu` I didn't measure any difference. Both benchmarks report times between 97 and 103 ms randomly, without one being consistently higher than the other. This suggests that at least in this particular case, when not running any debugger, there is no runtime overhead. ## API changes This PR adds a generic parameter to all `eval_xxx` functions that forces you to specify whether you use the debugger. You can resolve it in two ways: * Use a provided helper that will figure it out for you. If you wanted to use `eval_block(&engine_state, ...)`, call `let eval_block = get_eval_block(&engine_state); eval_block(&engine_state, ...)` * If you know you're in an evaluation path that doesn't need debugger support, call `eval_block::<WithoutDebug>(&engine_state, ...)` (this is the case of hooks, for example). I tried to add more explanation in the docstring of `debugger_trait.rs`. ## TODO - [x] Better profiler output to reduce spam of iterative commands like `each` - [x] Resolve `TODO: DEBUG` comments - [x] Resolve unwraps - [x] Add doc comments - [x] Add usage and extra usage for `debug profile`, explaining all columns # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Hopefully none. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-03-08 18:21:35 +00:00
let result = eval_expression::<WithoutDebug>(engine_state, stack, expr)?;
FromValue::from_value(result).map(Some)
2021-10-02 02:59:11 +00:00
} else {
Ok(None)
}
}
make the `ansi` command `const` (#11682) # Description This PR changes the `ansi` command to be a `const` command. - ~~It's breaking because I found that I had to change the way `ansi` is used in scripts a little bit. https://github.com/nushell/nu_scripts/pull/751~~ - I had to change one of the examples because apparently `const` can't be tested yet. - ~~I'm not sure this is right at all https://github.com/nushell/nushell/pull/11682/files#diff-ba932369a40eb40d6e1985eac1c784af403dab4500a7f0568e593900bf6cd740R654-R655. I just didn't want to duplicate a ton of code. Maybe if I duplicated the code it wouldn't be a breaking change because it would have a run and run_const?~~ - I had to add `opt_const` to CallExt. /cc @kubouch Can you take a look at this? I'm a little iffy if I'm doing this right, or even if we should do this at all. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-01-30 22:09:43 +00:00
fn opt_const<T: FromValue>(
&self,
working_set: &StateWorkingSet,
pos: usize,
) -> Result<Option<T>, ShellError> {
if let Some(expr) = self.positional_nth(pos) {
let result = eval_constant(working_set, expr)?;
FromValue::from_value(result).map(Some)
} else {
Ok(None)
}
}
2021-10-25 06:31:39 +00:00
fn req<T: FromValue>(
&self,
engine_state: &EngineState,
stack: &mut Stack,
pos: usize,
) -> Result<T, ShellError> {
if let Some(expr) = self.positional_nth(pos) {
let stack = &mut stack.use_call_arg_out_dest();
Debugger experiments (#11441) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR adds a new evaluator path with callbacks to a mutable trait object implementing a Debugger trait. The trait object can do anything, e.g., profiling, code coverage, step debugging. Currently, entering/leaving a block and a pipeline element is marked with callbacks, but more callbacks can be added as necessary. Not all callbacks need to be used by all debuggers; unused ones are simply empty calls. A simple profiler is implemented as a proof of concept. The debugging support is implementing by making `eval_xxx()` functions generic depending on whether we're debugging or not. This has zero computational overhead, but makes the binary slightly larger (see benchmarks below). `eval_xxx()` variants called from commands (like `eval_block_with_early_return()` in `each`) are chosen with a dynamic dispatch for two reasons: to not grow the binary size due to duplicating the code of many commands, and for the fact that it isn't possible because it would make Command trait objects object-unsafe. In the future, I hope it will be possible to allow plugin callbacks such that users would be able to implement their profiler plugins instead of having to recompile Nushell. [DAP](https://microsoft.github.io/debug-adapter-protocol/) would also be interesting to explore. Try `help debug profile`. ## Screenshots Basic output: ![profiler_new](https://github.com/nushell/nushell/assets/25571562/418b9df0-b659-4dcb-b023-2d5fcef2c865) To profile with more granularity, increase the profiler depth (you'll see that repeated `is-windows` calls take a large chunk of total time, making it a good candidate for optimizing): ![profiler_new_m3](https://github.com/nushell/nushell/assets/25571562/636d756d-5d56-460c-a372-14716f65f37f) ## Benchmarks ### Binary size Binary size increase vs. main: **+40360 bytes**. _(Both built with `--release --features=extra,dataframe`.)_ ### Time ```nushell # bench_debug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'debug:' let res2 = bench { debug profile $test } --pretty print $res2 ``` ```nushell # bench_nodebug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'no debug:' let res1 = bench { do $test } --pretty print $res1 ``` `cargo run --release -- bench_debug.nu` is consistently 1--2 ms slower than `cargo run --release -- bench_nodebug.nu` due to the collection overhead + gathering the report. This is expected. When gathering more stuff, the overhead is obviously higher. `cargo run --release -- bench_nodebug.nu` vs. `nu bench_nodebug.nu` I didn't measure any difference. Both benchmarks report times between 97 and 103 ms randomly, without one being consistently higher than the other. This suggests that at least in this particular case, when not running any debugger, there is no runtime overhead. ## API changes This PR adds a generic parameter to all `eval_xxx` functions that forces you to specify whether you use the debugger. You can resolve it in two ways: * Use a provided helper that will figure it out for you. If you wanted to use `eval_block(&engine_state, ...)`, call `let eval_block = get_eval_block(&engine_state); eval_block(&engine_state, ...)` * If you know you're in an evaluation path that doesn't need debugger support, call `eval_block::<WithoutDebug>(&engine_state, ...)` (this is the case of hooks, for example). I tried to add more explanation in the docstring of `debugger_trait.rs`. ## TODO - [x] Better profiler output to reduce spam of iterative commands like `each` - [x] Resolve `TODO: DEBUG` comments - [x] Resolve unwraps - [x] Add doc comments - [x] Add usage and extra usage for `debug profile`, explaining all columns # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Hopefully none. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-03-08 18:21:35 +00:00
let result = eval_expression::<WithoutDebug>(engine_state, stack, expr)?;
FromValue::from_value(result)
} else if self.positional_len() == 0 {
2023-03-06 17:33:09 +00:00
Err(ShellError::AccessEmptyContent { span: self.head })
2021-10-02 02:59:11 +00:00
} else {
2023-03-06 17:33:09 +00:00
Err(ShellError::AccessBeyondEnd {
max_idx: self.positional_len() - 1,
span: self.head,
})
2021-10-02 02:59:11 +00:00
}
}
fn req_parser_info<T: FromValue>(
&self,
engine_state: &EngineState,
stack: &mut Stack,
name: &str,
) -> Result<T, ShellError> {
if let Some(expr) = self.get_parser_info(name) {
let stack = &mut stack.use_call_arg_out_dest();
Debugger experiments (#11441) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR adds a new evaluator path with callbacks to a mutable trait object implementing a Debugger trait. The trait object can do anything, e.g., profiling, code coverage, step debugging. Currently, entering/leaving a block and a pipeline element is marked with callbacks, but more callbacks can be added as necessary. Not all callbacks need to be used by all debuggers; unused ones are simply empty calls. A simple profiler is implemented as a proof of concept. The debugging support is implementing by making `eval_xxx()` functions generic depending on whether we're debugging or not. This has zero computational overhead, but makes the binary slightly larger (see benchmarks below). `eval_xxx()` variants called from commands (like `eval_block_with_early_return()` in `each`) are chosen with a dynamic dispatch for two reasons: to not grow the binary size due to duplicating the code of many commands, and for the fact that it isn't possible because it would make Command trait objects object-unsafe. In the future, I hope it will be possible to allow plugin callbacks such that users would be able to implement their profiler plugins instead of having to recompile Nushell. [DAP](https://microsoft.github.io/debug-adapter-protocol/) would also be interesting to explore. Try `help debug profile`. ## Screenshots Basic output: ![profiler_new](https://github.com/nushell/nushell/assets/25571562/418b9df0-b659-4dcb-b023-2d5fcef2c865) To profile with more granularity, increase the profiler depth (you'll see that repeated `is-windows` calls take a large chunk of total time, making it a good candidate for optimizing): ![profiler_new_m3](https://github.com/nushell/nushell/assets/25571562/636d756d-5d56-460c-a372-14716f65f37f) ## Benchmarks ### Binary size Binary size increase vs. main: **+40360 bytes**. _(Both built with `--release --features=extra,dataframe`.)_ ### Time ```nushell # bench_debug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'debug:' let res2 = bench { debug profile $test } --pretty print $res2 ``` ```nushell # bench_nodebug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'no debug:' let res1 = bench { do $test } --pretty print $res1 ``` `cargo run --release -- bench_debug.nu` is consistently 1--2 ms slower than `cargo run --release -- bench_nodebug.nu` due to the collection overhead + gathering the report. This is expected. When gathering more stuff, the overhead is obviously higher. `cargo run --release -- bench_nodebug.nu` vs. `nu bench_nodebug.nu` I didn't measure any difference. Both benchmarks report times between 97 and 103 ms randomly, without one being consistently higher than the other. This suggests that at least in this particular case, when not running any debugger, there is no runtime overhead. ## API changes This PR adds a generic parameter to all `eval_xxx` functions that forces you to specify whether you use the debugger. You can resolve it in two ways: * Use a provided helper that will figure it out for you. If you wanted to use `eval_block(&engine_state, ...)`, call `let eval_block = get_eval_block(&engine_state); eval_block(&engine_state, ...)` * If you know you're in an evaluation path that doesn't need debugger support, call `eval_block::<WithoutDebug>(&engine_state, ...)` (this is the case of hooks, for example). I tried to add more explanation in the docstring of `debugger_trait.rs`. ## TODO - [x] Better profiler output to reduce spam of iterative commands like `each` - [x] Resolve `TODO: DEBUG` comments - [x] Resolve unwraps - [x] Add doc comments - [x] Add usage and extra usage for `debug profile`, explaining all columns # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Hopefully none. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-03-08 18:21:35 +00:00
let result = eval_expression::<WithoutDebug>(engine_state, stack, expr)?;
FromValue::from_value(result)
} else if self.parser_info.is_empty() {
2023-03-06 17:33:09 +00:00
Err(ShellError::AccessEmptyContent { span: self.head })
} else {
2023-03-06 17:33:09 +00:00
Err(ShellError::AccessBeyondEnd {
max_idx: self.parser_info.len() - 1,
span: self.head,
})
}
}
2021-10-01 21:53:13 +00:00
}