nushell/crates/nu-plugin-engine/src/declaration.rs

127 lines
3.6 KiB
Rust
Raw Normal View History

use nu_engine::{command_prelude::*, get_eval_expression};
Split the plugin crate (#12563) # Description This breaks `nu-plugin` up into four crates: - `nu-plugin-protocol`: just the type definitions for the protocol, no I/O. If someone wanted to wire up something more bare metal, maybe for async I/O, they could use this. - `nu-plugin-core`: the shared stuff between engine/plugin. Less stable interface. - `nu-plugin-engine`: everything required for the engine to talk to plugins. Less stable interface. - `nu-plugin`: everything required for the plugin to talk to the engine, what plugin developers use. Should be the most stable interface. No changes are made to the interface exposed by `nu-plugin` - it should all still be there. Re-exports from `nu-plugin-protocol` or `nu-plugin-core` are used as required. Plugins shouldn't ever have to use those crates directly. This should be somewhat faster to compile as `nu-plugin-engine` and `nu-plugin` can compile in parallel, and the engine doesn't need `nu-plugin` and plugins don't need `nu-plugin-engine` (except for test support), so that should reduce what needs to be compiled too. The only significant change here other than splitting stuff up was to break the `source` out of `PluginCustomValue` and create a new `PluginCustomValueWithSource` type that contains that instead. One bonus of that is we get rid of the option and it's now more type-safe, but it also means that the logic for that stuff (actually running the plugin for custom value ops) can live entirely within the `nu-plugin-engine` crate. # User-Facing Changes - New crates. - Added `local-socket` feature for `nu` to try to make it possible to compile without that support if needed. # Tests + Formatting - :green_circle: `toolkit fmt` - :green_circle: `toolkit clippy` - :green_circle: `toolkit test` - :green_circle: `toolkit test stdlib`
2024-04-27 17:08:12 +00:00
use nu_plugin_protocol::{CallInfo, EvaluatedCall};
use nu_protocol::{engine::CommandType, PluginIdentity, PluginSignature};
use std::sync::Arc;
Split the plugin crate (#12563) # Description This breaks `nu-plugin` up into four crates: - `nu-plugin-protocol`: just the type definitions for the protocol, no I/O. If someone wanted to wire up something more bare metal, maybe for async I/O, they could use this. - `nu-plugin-core`: the shared stuff between engine/plugin. Less stable interface. - `nu-plugin-engine`: everything required for the engine to talk to plugins. Less stable interface. - `nu-plugin`: everything required for the plugin to talk to the engine, what plugin developers use. Should be the most stable interface. No changes are made to the interface exposed by `nu-plugin` - it should all still be there. Re-exports from `nu-plugin-protocol` or `nu-plugin-core` are used as required. Plugins shouldn't ever have to use those crates directly. This should be somewhat faster to compile as `nu-plugin-engine` and `nu-plugin` can compile in parallel, and the engine doesn't need `nu-plugin` and plugins don't need `nu-plugin-engine` (except for test support), so that should reduce what needs to be compiled too. The only significant change here other than splitting stuff up was to break the `source` out of `PluginCustomValue` and create a new `PluginCustomValueWithSource` type that contains that instead. One bonus of that is we get rid of the option and it's now more type-safe, but it also means that the logic for that stuff (actually running the plugin for custom value ops) can live entirely within the `nu-plugin-engine` crate. # User-Facing Changes - New crates. - Added `local-socket` feature for `nu` to try to make it possible to compile without that support if needed. # Tests + Formatting - :green_circle: `toolkit fmt` - :green_circle: `toolkit clippy` - :green_circle: `toolkit test` - :green_circle: `toolkit test stdlib`
2024-04-27 17:08:12 +00:00
use crate::{GetPlugin, PluginExecutionCommandContext, PluginSource};
/// The command declaration proxy used within the engine for all plugin commands.
#[derive(Clone)]
pub struct PluginDeclaration {
name: String,
Make plugin commands support examples. (#7984) # Description As title, we can't provide examples for plugin commands, this pr would make it possible # User-Facing Changes Take plugin `nu-example-1` as example: ``` ❯ nu-example-1 -h PluginSignature test 1 for plugin. Returns Value::Nothing Usage: > nu-example-1 {flags} <a> <b> (opt) ...(rest) Flags: -h, --help - Display the help message for this command -f, --flag - a flag for the signature -n, --named <String> - named string Parameters: a <int>: required integer value b <string>: required string value (optional) opt <int>: Optional number ...rest <string>: rest value string Examples: running example with an int value and string value > nu-example-1 3 bb ``` The examples session is newly added. ## Basic idea behind these changes when nushell query plugin signatures, plugin just returns it's signature without any examples, so nushell have no idea about the examples of plugin commands. To adding the feature, we just making plugin returns it's signature with examples. Before: ``` 1. get signature ----------------> Nushell ------------------ Plugin <----------------- 2. returns Vec<Signature> ``` After: ``` 1. get signature ----------------> Nushell ------------------ Plugin <----------------- 2. returns Vec<PluginSignature> ``` When writing plugin signature to $nu.plugin-path: Serialize `<PluginSignature>` rather than `<Signature>`, which would enable us to serialize examples to `$nu.plugin-path` ## Shortcoming It's a breaking changes because `Plugin::signature` is changed, and it requires plugin authors to change their code for new signatures. Fortunally it should be easy to change, for rust based plugin, we just need to make a global replace from word `Signature` to word `PluginSignature` in their plugin project. Our content of plugin-path is really large, if one plugin have many examples, it'd results to larger body of $nu.plugin-path, which is not really scale. A solution would be save register information in other binary formats rather than `json`. But I think it'd be another story. # Tests + Formatting Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A clippy::needless_collect` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass # After Submitting If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date.
2023-02-08 22:14:18 +00:00
signature: PluginSignature,
Keep plugins persistently running in the background (#12064) # Description This PR uses the new plugin protocol to intelligently keep plugin processes running in the background for further plugin calls. Running plugins can be seen by running the new `plugin list` command, and stopped by running the new `plugin stop` command. This is an enhancement for the performance of plugins, as starting new plugin processes has overhead, especially for plugins in languages that take a significant amount of time on startup. It also enables plugins that have persistent state between commands, making the migration of features like dataframes and `stor` to plugins possible. Plugins are automatically stopped by the new plugin garbage collector, configurable with `$env.config.plugin_gc`: ```nushell $env.config.plugin_gc = { # Configuration for plugin garbage collection default: { enabled: true # true to enable stopping of inactive plugins stop_after: 10sec # how long to wait after a plugin is inactive to stop it } plugins: { # alternate configuration for specific plugins, by name, for example: # # gstat: { # enabled: false # } } } ``` If garbage collection is enabled, plugins will be stopped after `stop_after` passes after they were last active. Plugins are counted as inactive if they have no running plugin calls. Reading the stream from the response of a plugin call is still considered to be activity, but if a plugin holds on to a stream but the call ends without an active streaming response, it is not counted as active even if it is reading it. Plugins can explicitly disable the GC as appropriate with `engine.set_gc_disabled(true)`. The `version` command now lists plugin names rather than plugin commands. The list of plugin commands is accessible via `plugin list`. Recommend doing this together with #12029, because it will likely force plugin developers to do the right thing with mutability and lead to less unexpected behavior when running plugins nested / in parallel. # User-Facing Changes - new command: `plugin list` - new command: `plugin stop` - changed command: `version` (now lists plugin names, rather than commands) - new config: `$env.config.plugin_gc` - Plugins will keep running and be reused, at least for the configured GC period - Plugins that used mutable state in weird ways like `inc` did might misbehave until fixed - Plugins can disable GC if they need to - Had to change plugin signature to accept `&EngineInterface` so that the GC disable feature works. #12029 does this anyway, and I'm expecting (resolvable) conflicts with that # Tests + Formatting - :green_circle: `toolkit fmt` - :green_circle: `toolkit clippy` - :green_circle: `toolkit test` - :green_circle: `toolkit test stdlib` Because there is some specific OS behavior required for plugins to not respond to Ctrl-C directly, I've developed against and tested on both Linux and Windows to ensure that works properly. # After Submitting I think this probably needs to be in the book somewhere
2024-03-09 23:10:22 +00:00
source: PluginSource,
}
impl PluginDeclaration {
Add test support crate for plugin developers (#12259) # Description Adds a `nu-plugin-test-support` crate with an interface that supports testing plugins. Unlike in reality, these plugins run in the same process on separate threads. This will allow testing aspects of the plugin internal state and handling serialized plugin custom values easily. We still serialize their custom values and all of the engine to plugin logic is still in play, so from a logical perspective this should still expose any bugs that would have been caused by that. The only difference is that it doesn't run in a different process, and doesn't try to serialize everything to the final wire format for stdin/stdout. TODO still: - [x] Clean up warnings about private types exposed in trait definition - [x] Automatically deserialize plugin custom values in the result so they can be inspected - [x] Automatic plugin examples test function - [x] Write a bit more documentation - [x] More tests - [x] Add MIT License file to new crate # User-Facing Changes Plugin developers get a nice way to test their plugins. # Tests + Formatting Run the tests with `cargo test -p nu-plugin-test-support -- --show-output` to see some examples of what the failing test output for examples can look like. I used the `difference` crate (MIT licensed) to make it look nice. - :green_circle: `toolkit fmt` - :green_circle: `toolkit clippy` - :green_circle: `toolkit test` - :green_circle: `toolkit test stdlib` # After Submitting - [ ] Add a section to the book about testing - [ ] Test some of the example plugins this way - [ ] Add example tests to nu_plugin_template so plugin developers have something to start with
2024-03-23 18:29:54 +00:00
pub fn new(plugin: Arc<dyn GetPlugin>, signature: PluginSignature) -> Self {
Self {
Make plugin commands support examples. (#7984) # Description As title, we can't provide examples for plugin commands, this pr would make it possible # User-Facing Changes Take plugin `nu-example-1` as example: ``` ❯ nu-example-1 -h PluginSignature test 1 for plugin. Returns Value::Nothing Usage: > nu-example-1 {flags} <a> <b> (opt) ...(rest) Flags: -h, --help - Display the help message for this command -f, --flag - a flag for the signature -n, --named <String> - named string Parameters: a <int>: required integer value b <string>: required string value (optional) opt <int>: Optional number ...rest <string>: rest value string Examples: running example with an int value and string value > nu-example-1 3 bb ``` The examples session is newly added. ## Basic idea behind these changes when nushell query plugin signatures, plugin just returns it's signature without any examples, so nushell have no idea about the examples of plugin commands. To adding the feature, we just making plugin returns it's signature with examples. Before: ``` 1. get signature ----------------> Nushell ------------------ Plugin <----------------- 2. returns Vec<Signature> ``` After: ``` 1. get signature ----------------> Nushell ------------------ Plugin <----------------- 2. returns Vec<PluginSignature> ``` When writing plugin signature to $nu.plugin-path: Serialize `<PluginSignature>` rather than `<Signature>`, which would enable us to serialize examples to `$nu.plugin-path` ## Shortcoming It's a breaking changes because `Plugin::signature` is changed, and it requires plugin authors to change their code for new signatures. Fortunally it should be easy to change, for rust based plugin, we just need to make a global replace from word `Signature` to word `PluginSignature` in their plugin project. Our content of plugin-path is really large, if one plugin have many examples, it'd results to larger body of $nu.plugin-path, which is not really scale. A solution would be save register information in other binary formats rather than `json`. But I think it'd be another story. # Tests + Formatting Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A clippy::needless_collect` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass # After Submitting If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date.
2023-02-08 22:14:18 +00:00
name: signature.sig.name.clone(),
signature,
Keep plugins persistently running in the background (#12064) # Description This PR uses the new plugin protocol to intelligently keep plugin processes running in the background for further plugin calls. Running plugins can be seen by running the new `plugin list` command, and stopped by running the new `plugin stop` command. This is an enhancement for the performance of plugins, as starting new plugin processes has overhead, especially for plugins in languages that take a significant amount of time on startup. It also enables plugins that have persistent state between commands, making the migration of features like dataframes and `stor` to plugins possible. Plugins are automatically stopped by the new plugin garbage collector, configurable with `$env.config.plugin_gc`: ```nushell $env.config.plugin_gc = { # Configuration for plugin garbage collection default: { enabled: true # true to enable stopping of inactive plugins stop_after: 10sec # how long to wait after a plugin is inactive to stop it } plugins: { # alternate configuration for specific plugins, by name, for example: # # gstat: { # enabled: false # } } } ``` If garbage collection is enabled, plugins will be stopped after `stop_after` passes after they were last active. Plugins are counted as inactive if they have no running plugin calls. Reading the stream from the response of a plugin call is still considered to be activity, but if a plugin holds on to a stream but the call ends without an active streaming response, it is not counted as active even if it is reading it. Plugins can explicitly disable the GC as appropriate with `engine.set_gc_disabled(true)`. The `version` command now lists plugin names rather than plugin commands. The list of plugin commands is accessible via `plugin list`. Recommend doing this together with #12029, because it will likely force plugin developers to do the right thing with mutability and lead to less unexpected behavior when running plugins nested / in parallel. # User-Facing Changes - new command: `plugin list` - new command: `plugin stop` - changed command: `version` (now lists plugin names, rather than commands) - new config: `$env.config.plugin_gc` - Plugins will keep running and be reused, at least for the configured GC period - Plugins that used mutable state in weird ways like `inc` did might misbehave until fixed - Plugins can disable GC if they need to - Had to change plugin signature to accept `&EngineInterface` so that the GC disable feature works. #12029 does this anyway, and I'm expecting (resolvable) conflicts with that # Tests + Formatting - :green_circle: `toolkit fmt` - :green_circle: `toolkit clippy` - :green_circle: `toolkit test` - :green_circle: `toolkit test stdlib` Because there is some specific OS behavior required for plugins to not respond to Ctrl-C directly, I've developed against and tested on both Linux and Windows to ensure that works properly. # After Submitting I think this probably needs to be in the book somewhere
2024-03-09 23:10:22 +00:00
source: PluginSource::new(plugin),
}
}
}
impl Command for PluginDeclaration {
fn name(&self) -> &str {
&self.name
}
fn signature(&self) -> Signature {
Make plugin commands support examples. (#7984) # Description As title, we can't provide examples for plugin commands, this pr would make it possible # User-Facing Changes Take plugin `nu-example-1` as example: ``` ❯ nu-example-1 -h PluginSignature test 1 for plugin. Returns Value::Nothing Usage: > nu-example-1 {flags} <a> <b> (opt) ...(rest) Flags: -h, --help - Display the help message for this command -f, --flag - a flag for the signature -n, --named <String> - named string Parameters: a <int>: required integer value b <string>: required string value (optional) opt <int>: Optional number ...rest <string>: rest value string Examples: running example with an int value and string value > nu-example-1 3 bb ``` The examples session is newly added. ## Basic idea behind these changes when nushell query plugin signatures, plugin just returns it's signature without any examples, so nushell have no idea about the examples of plugin commands. To adding the feature, we just making plugin returns it's signature with examples. Before: ``` 1. get signature ----------------> Nushell ------------------ Plugin <----------------- 2. returns Vec<Signature> ``` After: ``` 1. get signature ----------------> Nushell ------------------ Plugin <----------------- 2. returns Vec<PluginSignature> ``` When writing plugin signature to $nu.plugin-path: Serialize `<PluginSignature>` rather than `<Signature>`, which would enable us to serialize examples to `$nu.plugin-path` ## Shortcoming It's a breaking changes because `Plugin::signature` is changed, and it requires plugin authors to change their code for new signatures. Fortunally it should be easy to change, for rust based plugin, we just need to make a global replace from word `Signature` to word `PluginSignature` in their plugin project. Our content of plugin-path is really large, if one plugin have many examples, it'd results to larger body of $nu.plugin-path, which is not really scale. A solution would be save register information in other binary formats rather than `json`. But I think it'd be another story. # Tests + Formatting Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A clippy::needless_collect` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass # After Submitting If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date.
2023-02-08 22:14:18 +00:00
self.signature.sig.clone()
}
fn usage(&self) -> &str {
Make plugin commands support examples. (#7984) # Description As title, we can't provide examples for plugin commands, this pr would make it possible # User-Facing Changes Take plugin `nu-example-1` as example: ``` ❯ nu-example-1 -h PluginSignature test 1 for plugin. Returns Value::Nothing Usage: > nu-example-1 {flags} <a> <b> (opt) ...(rest) Flags: -h, --help - Display the help message for this command -f, --flag - a flag for the signature -n, --named <String> - named string Parameters: a <int>: required integer value b <string>: required string value (optional) opt <int>: Optional number ...rest <string>: rest value string Examples: running example with an int value and string value > nu-example-1 3 bb ``` The examples session is newly added. ## Basic idea behind these changes when nushell query plugin signatures, plugin just returns it's signature without any examples, so nushell have no idea about the examples of plugin commands. To adding the feature, we just making plugin returns it's signature with examples. Before: ``` 1. get signature ----------------> Nushell ------------------ Plugin <----------------- 2. returns Vec<Signature> ``` After: ``` 1. get signature ----------------> Nushell ------------------ Plugin <----------------- 2. returns Vec<PluginSignature> ``` When writing plugin signature to $nu.plugin-path: Serialize `<PluginSignature>` rather than `<Signature>`, which would enable us to serialize examples to `$nu.plugin-path` ## Shortcoming It's a breaking changes because `Plugin::signature` is changed, and it requires plugin authors to change their code for new signatures. Fortunally it should be easy to change, for rust based plugin, we just need to make a global replace from word `Signature` to word `PluginSignature` in their plugin project. Our content of plugin-path is really large, if one plugin have many examples, it'd results to larger body of $nu.plugin-path, which is not really scale. A solution would be save register information in other binary formats rather than `json`. But I think it'd be another story. # Tests + Formatting Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A clippy::needless_collect` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass # After Submitting If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date.
2023-02-08 22:14:18 +00:00
self.signature.sig.usage.as_str()
}
Show plugin extra usage and search terms (#10952) # Description The `PluginSignature` type supports extra usage but this was not available in `plugin_name --help`. It also supports search terms but these did not appear in `help commands` New behavior show below is the "Extra usage for nu-example-1" line and the "Search terms:" line ``` ❯ nu-example-1 --help PluginSignature test 1 for plugin. Returns Value::Nothing Extra usage for nu-example-1 Search terms: example Usage: > nu-example-1 {flags} <a> <b> (opt) ...(rest) Flags: -h, --help - Display the help message for this command -f, --flag - a flag for the signature -n, --named <String> - named string Parameters: a <int>: required integer value b <string>: required string value opt <int>: Optional number (optional) ...rest <string>: rest value string Examples: running example with an int value and string value > nu-example-1 3 bb ``` Search terms are also available in `help commands`: ``` ❯ help commands | where name == "nu-example-1" | select name search_terms ╭──────────────┬──────────────╮ │ name │ search_terms │ ├──────────────┼──────────────┤ │ nu-example-1 │ example │ ╰──────────────┴──────────────╯ ``` # User-Facing Changes Users can now see plugin extra usage and search terms # Tests + Formatting - :green_circle: `toolkit fmt` - :green_circle: `toolkit clippy` - :green_circle: `toolkit test` - :green_circle: `toolkit test stdlib` # After Submitting N/A
2023-11-04 20:12:58 +00:00
fn extra_usage(&self) -> &str {
self.signature.sig.extra_usage.as_str()
}
fn search_terms(&self) -> Vec<&str> {
self.signature
.sig
.search_terms
.iter()
.map(|term| term.as_str())
.collect()
}
Make plugin commands support examples. (#7984) # Description As title, we can't provide examples for plugin commands, this pr would make it possible # User-Facing Changes Take plugin `nu-example-1` as example: ``` ❯ nu-example-1 -h PluginSignature test 1 for plugin. Returns Value::Nothing Usage: > nu-example-1 {flags} <a> <b> (opt) ...(rest) Flags: -h, --help - Display the help message for this command -f, --flag - a flag for the signature -n, --named <String> - named string Parameters: a <int>: required integer value b <string>: required string value (optional) opt <int>: Optional number ...rest <string>: rest value string Examples: running example with an int value and string value > nu-example-1 3 bb ``` The examples session is newly added. ## Basic idea behind these changes when nushell query plugin signatures, plugin just returns it's signature without any examples, so nushell have no idea about the examples of plugin commands. To adding the feature, we just making plugin returns it's signature with examples. Before: ``` 1. get signature ----------------> Nushell ------------------ Plugin <----------------- 2. returns Vec<Signature> ``` After: ``` 1. get signature ----------------> Nushell ------------------ Plugin <----------------- 2. returns Vec<PluginSignature> ``` When writing plugin signature to $nu.plugin-path: Serialize `<PluginSignature>` rather than `<Signature>`, which would enable us to serialize examples to `$nu.plugin-path` ## Shortcoming It's a breaking changes because `Plugin::signature` is changed, and it requires plugin authors to change their code for new signatures. Fortunally it should be easy to change, for rust based plugin, we just need to make a global replace from word `Signature` to word `PluginSignature` in their plugin project. Our content of plugin-path is really large, if one plugin have many examples, it'd results to larger body of $nu.plugin-path, which is not really scale. A solution would be save register information in other binary formats rather than `json`. But I think it'd be another story. # Tests + Formatting Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A clippy::needless_collect` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass # After Submitting If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date.
2023-02-08 22:14:18 +00:00
fn examples(&self) -> Vec<Example> {
let mut res = vec![];
for e in self.signature.examples.iter() {
res.push(Example {
example: &e.example,
description: &e.description,
result: e.result.clone(),
})
}
res
}
fn run(
&self,
engine_state: &EngineState,
stack: &mut Stack,
call: &Call,
input: PipelineData,
) -> Result<PipelineData, ShellError> {
Debugger experiments (#11441) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR adds a new evaluator path with callbacks to a mutable trait object implementing a Debugger trait. The trait object can do anything, e.g., profiling, code coverage, step debugging. Currently, entering/leaving a block and a pipeline element is marked with callbacks, but more callbacks can be added as necessary. Not all callbacks need to be used by all debuggers; unused ones are simply empty calls. A simple profiler is implemented as a proof of concept. The debugging support is implementing by making `eval_xxx()` functions generic depending on whether we're debugging or not. This has zero computational overhead, but makes the binary slightly larger (see benchmarks below). `eval_xxx()` variants called from commands (like `eval_block_with_early_return()` in `each`) are chosen with a dynamic dispatch for two reasons: to not grow the binary size due to duplicating the code of many commands, and for the fact that it isn't possible because it would make Command trait objects object-unsafe. In the future, I hope it will be possible to allow plugin callbacks such that users would be able to implement their profiler plugins instead of having to recompile Nushell. [DAP](https://microsoft.github.io/debug-adapter-protocol/) would also be interesting to explore. Try `help debug profile`. ## Screenshots Basic output: ![profiler_new](https://github.com/nushell/nushell/assets/25571562/418b9df0-b659-4dcb-b023-2d5fcef2c865) To profile with more granularity, increase the profiler depth (you'll see that repeated `is-windows` calls take a large chunk of total time, making it a good candidate for optimizing): ![profiler_new_m3](https://github.com/nushell/nushell/assets/25571562/636d756d-5d56-460c-a372-14716f65f37f) ## Benchmarks ### Binary size Binary size increase vs. main: **+40360 bytes**. _(Both built with `--release --features=extra,dataframe`.)_ ### Time ```nushell # bench_debug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'debug:' let res2 = bench { debug profile $test } --pretty print $res2 ``` ```nushell # bench_nodebug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'no debug:' let res1 = bench { do $test } --pretty print $res1 ``` `cargo run --release -- bench_debug.nu` is consistently 1--2 ms slower than `cargo run --release -- bench_nodebug.nu` due to the collection overhead + gathering the report. This is expected. When gathering more stuff, the overhead is obviously higher. `cargo run --release -- bench_nodebug.nu` vs. `nu bench_nodebug.nu` I didn't measure any difference. Both benchmarks report times between 97 and 103 ms randomly, without one being consistently higher than the other. This suggests that at least in this particular case, when not running any debugger, there is no runtime overhead. ## API changes This PR adds a generic parameter to all `eval_xxx` functions that forces you to specify whether you use the debugger. You can resolve it in two ways: * Use a provided helper that will figure it out for you. If you wanted to use `eval_block(&engine_state, ...)`, call `let eval_block = get_eval_block(&engine_state); eval_block(&engine_state, ...)` * If you know you're in an evaluation path that doesn't need debugger support, call `eval_block::<WithoutDebug>(&engine_state, ...)` (this is the case of hooks, for example). I tried to add more explanation in the docstring of `debugger_trait.rs`. ## TODO - [x] Better profiler output to reduce spam of iterative commands like `each` - [x] Resolve `TODO: DEBUG` comments - [x] Resolve unwraps - [x] Add doc comments - [x] Add usage and extra usage for `debug profile`, explaining all columns # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Hopefully none. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-03-08 18:21:35 +00:00
let eval_expression = get_eval_expression(engine_state);
// Create the EvaluatedCall to send to the plugin first - it's best for this to fail early,
// before we actually try to run the plugin command
Debugger experiments (#11441) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR adds a new evaluator path with callbacks to a mutable trait object implementing a Debugger trait. The trait object can do anything, e.g., profiling, code coverage, step debugging. Currently, entering/leaving a block and a pipeline element is marked with callbacks, but more callbacks can be added as necessary. Not all callbacks need to be used by all debuggers; unused ones are simply empty calls. A simple profiler is implemented as a proof of concept. The debugging support is implementing by making `eval_xxx()` functions generic depending on whether we're debugging or not. This has zero computational overhead, but makes the binary slightly larger (see benchmarks below). `eval_xxx()` variants called from commands (like `eval_block_with_early_return()` in `each`) are chosen with a dynamic dispatch for two reasons: to not grow the binary size due to duplicating the code of many commands, and for the fact that it isn't possible because it would make Command trait objects object-unsafe. In the future, I hope it will be possible to allow plugin callbacks such that users would be able to implement their profiler plugins instead of having to recompile Nushell. [DAP](https://microsoft.github.io/debug-adapter-protocol/) would also be interesting to explore. Try `help debug profile`. ## Screenshots Basic output: ![profiler_new](https://github.com/nushell/nushell/assets/25571562/418b9df0-b659-4dcb-b023-2d5fcef2c865) To profile with more granularity, increase the profiler depth (you'll see that repeated `is-windows` calls take a large chunk of total time, making it a good candidate for optimizing): ![profiler_new_m3](https://github.com/nushell/nushell/assets/25571562/636d756d-5d56-460c-a372-14716f65f37f) ## Benchmarks ### Binary size Binary size increase vs. main: **+40360 bytes**. _(Both built with `--release --features=extra,dataframe`.)_ ### Time ```nushell # bench_debug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'debug:' let res2 = bench { debug profile $test } --pretty print $res2 ``` ```nushell # bench_nodebug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'no debug:' let res1 = bench { do $test } --pretty print $res1 ``` `cargo run --release -- bench_debug.nu` is consistently 1--2 ms slower than `cargo run --release -- bench_nodebug.nu` due to the collection overhead + gathering the report. This is expected. When gathering more stuff, the overhead is obviously higher. `cargo run --release -- bench_nodebug.nu` vs. `nu bench_nodebug.nu` I didn't measure any difference. Both benchmarks report times between 97 and 103 ms randomly, without one being consistently higher than the other. This suggests that at least in this particular case, when not running any debugger, there is no runtime overhead. ## API changes This PR adds a generic parameter to all `eval_xxx` functions that forces you to specify whether you use the debugger. You can resolve it in two ways: * Use a provided helper that will figure it out for you. If you wanted to use `eval_block(&engine_state, ...)`, call `let eval_block = get_eval_block(&engine_state); eval_block(&engine_state, ...)` * If you know you're in an evaluation path that doesn't need debugger support, call `eval_block::<WithoutDebug>(&engine_state, ...)` (this is the case of hooks, for example). I tried to add more explanation in the docstring of `debugger_trait.rs`. ## TODO - [x] Better profiler output to reduce spam of iterative commands like `each` - [x] Resolve `TODO: DEBUG` comments - [x] Resolve unwraps - [x] Add doc comments - [x] Add usage and extra usage for `debug profile`, explaining all columns # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Hopefully none. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. -->
2024-03-08 18:21:35 +00:00
let evaluated_call =
EvaluatedCall::try_from_call(call, engine_state, stack, eval_expression)?;
Keep plugins persistently running in the background (#12064) # Description This PR uses the new plugin protocol to intelligently keep plugin processes running in the background for further plugin calls. Running plugins can be seen by running the new `plugin list` command, and stopped by running the new `plugin stop` command. This is an enhancement for the performance of plugins, as starting new plugin processes has overhead, especially for plugins in languages that take a significant amount of time on startup. It also enables plugins that have persistent state between commands, making the migration of features like dataframes and `stor` to plugins possible. Plugins are automatically stopped by the new plugin garbage collector, configurable with `$env.config.plugin_gc`: ```nushell $env.config.plugin_gc = { # Configuration for plugin garbage collection default: { enabled: true # true to enable stopping of inactive plugins stop_after: 10sec # how long to wait after a plugin is inactive to stop it } plugins: { # alternate configuration for specific plugins, by name, for example: # # gstat: { # enabled: false # } } } ``` If garbage collection is enabled, plugins will be stopped after `stop_after` passes after they were last active. Plugins are counted as inactive if they have no running plugin calls. Reading the stream from the response of a plugin call is still considered to be activity, but if a plugin holds on to a stream but the call ends without an active streaming response, it is not counted as active even if it is reading it. Plugins can explicitly disable the GC as appropriate with `engine.set_gc_disabled(true)`. The `version` command now lists plugin names rather than plugin commands. The list of plugin commands is accessible via `plugin list`. Recommend doing this together with #12029, because it will likely force plugin developers to do the right thing with mutability and lead to less unexpected behavior when running plugins nested / in parallel. # User-Facing Changes - new command: `plugin list` - new command: `plugin stop` - changed command: `version` (now lists plugin names, rather than commands) - new config: `$env.config.plugin_gc` - Plugins will keep running and be reused, at least for the configured GC period - Plugins that used mutable state in weird ways like `inc` did might misbehave until fixed - Plugins can disable GC if they need to - Had to change plugin signature to accept `&EngineInterface` so that the GC disable feature works. #12029 does this anyway, and I'm expecting (resolvable) conflicts with that # Tests + Formatting - :green_circle: `toolkit fmt` - :green_circle: `toolkit clippy` - :green_circle: `toolkit test` - :green_circle: `toolkit test stdlib` Because there is some specific OS behavior required for plugins to not respond to Ctrl-C directly, I've developed against and tested on both Linux and Windows to ensure that works properly. # After Submitting I think this probably needs to be in the book somewhere
2024-03-09 23:10:22 +00:00
// Get the engine config
let engine_config = nu_engine::get_config(engine_state, stack);
// Get, or start, the plugin.
let plugin = self
.source
.persistent(None)
.and_then(|p| {
// Set the garbage collector config from the local config before running
p.set_gc_config(engine_config.plugin_gc.get(p.identity().name()));
Add test support crate for plugin developers (#12259) # Description Adds a `nu-plugin-test-support` crate with an interface that supports testing plugins. Unlike in reality, these plugins run in the same process on separate threads. This will allow testing aspects of the plugin internal state and handling serialized plugin custom values easily. We still serialize their custom values and all of the engine to plugin logic is still in play, so from a logical perspective this should still expose any bugs that would have been caused by that. The only difference is that it doesn't run in a different process, and doesn't try to serialize everything to the final wire format for stdin/stdout. TODO still: - [x] Clean up warnings about private types exposed in trait definition - [x] Automatically deserialize plugin custom values in the result so they can be inspected - [x] Automatic plugin examples test function - [x] Write a bit more documentation - [x] More tests - [x] Add MIT License file to new crate # User-Facing Changes Plugin developers get a nice way to test their plugins. # Tests + Formatting Run the tests with `cargo test -p nu-plugin-test-support -- --show-output` to see some examples of what the failing test output for examples can look like. I used the `difference` crate (MIT licensed) to make it look nice. - :green_circle: `toolkit fmt` - :green_circle: `toolkit clippy` - :green_circle: `toolkit test` - :green_circle: `toolkit test stdlib` # After Submitting - [ ] Add a section to the book about testing - [ ] Test some of the example plugins this way - [ ] Add example tests to nu_plugin_template so plugin developers have something to start with
2024-03-23 18:29:54 +00:00
p.get_plugin(Some((engine_state, stack)))
Keep plugins persistently running in the background (#12064) # Description This PR uses the new plugin protocol to intelligently keep plugin processes running in the background for further plugin calls. Running plugins can be seen by running the new `plugin list` command, and stopped by running the new `plugin stop` command. This is an enhancement for the performance of plugins, as starting new plugin processes has overhead, especially for plugins in languages that take a significant amount of time on startup. It also enables plugins that have persistent state between commands, making the migration of features like dataframes and `stor` to plugins possible. Plugins are automatically stopped by the new plugin garbage collector, configurable with `$env.config.plugin_gc`: ```nushell $env.config.plugin_gc = { # Configuration for plugin garbage collection default: { enabled: true # true to enable stopping of inactive plugins stop_after: 10sec # how long to wait after a plugin is inactive to stop it } plugins: { # alternate configuration for specific plugins, by name, for example: # # gstat: { # enabled: false # } } } ``` If garbage collection is enabled, plugins will be stopped after `stop_after` passes after they were last active. Plugins are counted as inactive if they have no running plugin calls. Reading the stream from the response of a plugin call is still considered to be activity, but if a plugin holds on to a stream but the call ends without an active streaming response, it is not counted as active even if it is reading it. Plugins can explicitly disable the GC as appropriate with `engine.set_gc_disabled(true)`. The `version` command now lists plugin names rather than plugin commands. The list of plugin commands is accessible via `plugin list`. Recommend doing this together with #12029, because it will likely force plugin developers to do the right thing with mutability and lead to less unexpected behavior when running plugins nested / in parallel. # User-Facing Changes - new command: `plugin list` - new command: `plugin stop` - changed command: `version` (now lists plugin names, rather than commands) - new config: `$env.config.plugin_gc` - Plugins will keep running and be reused, at least for the configured GC period - Plugins that used mutable state in weird ways like `inc` did might misbehave until fixed - Plugins can disable GC if they need to - Had to change plugin signature to accept `&EngineInterface` so that the GC disable feature works. #12029 does this anyway, and I'm expecting (resolvable) conflicts with that # Tests + Formatting - :green_circle: `toolkit fmt` - :green_circle: `toolkit clippy` - :green_circle: `toolkit test` - :green_circle: `toolkit test stdlib` Because there is some specific OS behavior required for plugins to not respond to Ctrl-C directly, I've developed against and tested on both Linux and Windows to ensure that works properly. # After Submitting I think this probably needs to be in the book somewhere
2024-03-09 23:10:22 +00:00
})
.map_err(|err| {
let decl = engine_state.get_decl(call.decl_id);
ShellError::GenericError {
error: format!("Unable to spawn plugin for `{}`", decl.name()),
msg: err.to_string(),
span: Some(call.head),
help: None,
inner: vec![],
}
})?;
Add support for engine calls from plugins (#12029) # Description This allows plugins to make calls back to the engine to get config, evaluate closures, and do other things that must be done within the engine process. Engine calls can both produce and consume streams as necessary. Closures passed to plugins can both accept stream input and produce stream output sent back to the plugin. Engine calls referring to a plugin call's context can be processed as long either the response hasn't been received, or the response created streams that haven't ended yet. This is a breaking API change for plugins. There are some pretty major changes to the interface that plugins must implement, including: 1. Plugins now run with `&self` and must be `Sync`. Executing multiple plugin calls in parallel is supported, and there's a chance that a closure passed to a plugin could invoke the same plugin. Supporting state across plugin invocations is left up to the plugin author to do in whichever way they feel best, but the plugin object itself is still shared. Even though the engine doesn't run multiple plugin calls through the same process yet, I still considered it important to break the API in this way at this stage. We might want to consider an optional threadpool feature for performance. 2. Plugins take a reference to `EngineInterface`, which can be cloned. This interface allows plugins to make calls back to the engine, including for getting config and running closures. 3. Plugins no longer take the `config` parameter. This can be accessed from the interface via the `.get_plugin_config()` engine call. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Not only does this have plugin protocol changes, it will require plugins to make some code changes before they will work again. But on the plus side, the engine call feature is extensible, and we can add more things to it as needed. Plugin maintainers will have to change the trait signature at the very least. If they were using `config`, they will have to call `engine.get_plugin_config()` instead. If they were using the mutable reference to the plugin, they will have to come up with some strategy to work around it (for example, for `Inc` I just cloned it). This shouldn't be such a big deal at the moment as it's not like plugins have ever run as daemons with persistent state in the past, and they don't in this PR either. But I thought it was important to make the change before we support plugins as daemons, as an exclusive mutable reference is not compatible with parallel plugin calls. I suggest this gets merged sometime *after* the current pending release, so that we have some time to adjust to the previous plugin protocol changes that don't require code changes before making ones that do. # Tests + Formatting - :green_circle: `toolkit fmt` - :green_circle: `toolkit clippy` - :green_circle: `toolkit test` - :green_circle: `toolkit test stdlib` # After Submitting I will document the additional protocol features (`EngineCall`, `EngineCallResponse`), and constraints on plugin call processing if engine calls are used - basically, to be aware that an engine call could result in a nested plugin call, so the plugin should be able to handle that.
2024-03-09 17:26:30 +00:00
// Create the context to execute in - this supports engine calls and custom values
let mut context = PluginExecutionCommandContext::new(
Keep plugins persistently running in the background (#12064) # Description This PR uses the new plugin protocol to intelligently keep plugin processes running in the background for further plugin calls. Running plugins can be seen by running the new `plugin list` command, and stopped by running the new `plugin stop` command. This is an enhancement for the performance of plugins, as starting new plugin processes has overhead, especially for plugins in languages that take a significant amount of time on startup. It also enables plugins that have persistent state between commands, making the migration of features like dataframes and `stor` to plugins possible. Plugins are automatically stopped by the new plugin garbage collector, configurable with `$env.config.plugin_gc`: ```nushell $env.config.plugin_gc = { # Configuration for plugin garbage collection default: { enabled: true # true to enable stopping of inactive plugins stop_after: 10sec # how long to wait after a plugin is inactive to stop it } plugins: { # alternate configuration for specific plugins, by name, for example: # # gstat: { # enabled: false # } } } ``` If garbage collection is enabled, plugins will be stopped after `stop_after` passes after they were last active. Plugins are counted as inactive if they have no running plugin calls. Reading the stream from the response of a plugin call is still considered to be activity, but if a plugin holds on to a stream but the call ends without an active streaming response, it is not counted as active even if it is reading it. Plugins can explicitly disable the GC as appropriate with `engine.set_gc_disabled(true)`. The `version` command now lists plugin names rather than plugin commands. The list of plugin commands is accessible via `plugin list`. Recommend doing this together with #12029, because it will likely force plugin developers to do the right thing with mutability and lead to less unexpected behavior when running plugins nested / in parallel. # User-Facing Changes - new command: `plugin list` - new command: `plugin stop` - changed command: `version` (now lists plugin names, rather than commands) - new config: `$env.config.plugin_gc` - Plugins will keep running and be reused, at least for the configured GC period - Plugins that used mutable state in weird ways like `inc` did might misbehave until fixed - Plugins can disable GC if they need to - Had to change plugin signature to accept `&EngineInterface` so that the GC disable feature works. #12029 does this anyway, and I'm expecting (resolvable) conflicts with that # Tests + Formatting - :green_circle: `toolkit fmt` - :green_circle: `toolkit clippy` - :green_circle: `toolkit test` - :green_circle: `toolkit test stdlib` Because there is some specific OS behavior required for plugins to not respond to Ctrl-C directly, I've developed against and tested on both Linux and Windows to ensure that works properly. # After Submitting I think this probably needs to be in the book somewhere
2024-03-09 23:10:22 +00:00
self.source.identity.clone(),
engine_state,
stack,
call,
);
plugin.run(
CallInfo {
name: self.name.clone(),
call: evaluated_call,
input,
},
&mut context,
)
}
fn command_type(&self) -> CommandType {
CommandType::Plugin
Keep plugins persistently running in the background (#12064) # Description This PR uses the new plugin protocol to intelligently keep plugin processes running in the background for further plugin calls. Running plugins can be seen by running the new `plugin list` command, and stopped by running the new `plugin stop` command. This is an enhancement for the performance of plugins, as starting new plugin processes has overhead, especially for plugins in languages that take a significant amount of time on startup. It also enables plugins that have persistent state between commands, making the migration of features like dataframes and `stor` to plugins possible. Plugins are automatically stopped by the new plugin garbage collector, configurable with `$env.config.plugin_gc`: ```nushell $env.config.plugin_gc = { # Configuration for plugin garbage collection default: { enabled: true # true to enable stopping of inactive plugins stop_after: 10sec # how long to wait after a plugin is inactive to stop it } plugins: { # alternate configuration for specific plugins, by name, for example: # # gstat: { # enabled: false # } } } ``` If garbage collection is enabled, plugins will be stopped after `stop_after` passes after they were last active. Plugins are counted as inactive if they have no running plugin calls. Reading the stream from the response of a plugin call is still considered to be activity, but if a plugin holds on to a stream but the call ends without an active streaming response, it is not counted as active even if it is reading it. Plugins can explicitly disable the GC as appropriate with `engine.set_gc_disabled(true)`. The `version` command now lists plugin names rather than plugin commands. The list of plugin commands is accessible via `plugin list`. Recommend doing this together with #12029, because it will likely force plugin developers to do the right thing with mutability and lead to less unexpected behavior when running plugins nested / in parallel. # User-Facing Changes - new command: `plugin list` - new command: `plugin stop` - changed command: `version` (now lists plugin names, rather than commands) - new config: `$env.config.plugin_gc` - Plugins will keep running and be reused, at least for the configured GC period - Plugins that used mutable state in weird ways like `inc` did might misbehave until fixed - Plugins can disable GC if they need to - Had to change plugin signature to accept `&EngineInterface` so that the GC disable feature works. #12029 does this anyway, and I'm expecting (resolvable) conflicts with that # Tests + Formatting - :green_circle: `toolkit fmt` - :green_circle: `toolkit clippy` - :green_circle: `toolkit test` - :green_circle: `toolkit test stdlib` Because there is some specific OS behavior required for plugins to not respond to Ctrl-C directly, I've developed against and tested on both Linux and Windows to ensure that works properly. # After Submitting I think this probably needs to be in the book somewhere
2024-03-09 23:10:22 +00:00
}
fn plugin_identity(&self) -> Option<&PluginIdentity> {
Some(&self.source.identity)
}
}