Closes#13920
# User-Facing Changes
`random binary` and `random chars` now support filesize arguments:
```nushell
random binary 1kb
random chars --length 1kb
```
# Description
Fixes#13991. This was done by more clearly separating the case when a
pipeline is drained vs when it is being written (to a file).
I also added an `OutDest::Print` case which might not be strictly
necessary, but is a helpful addition.
# User-Facing Changes
Bug fix.
# Tests + Formatting
Added a test.
# After Submitting
There are still a few redirection bugs that I found, but they require
larger code changes, so I'll leave them until after the release.
# Description
Contributors to this projects will have a test failure if their `umask`
is not set to `0022`.
Apparently on Debian (at least on my install), it is set to `0002` which
makes my test fail. While `0022` is safer than the value I have, I want
to reduce the amount if issue new contributors could have.
I am making this test not assuming anything and instead, reading the
user umask.
# Related discussion
I see that the `umask` command implementation has been discussed in
#12256 . We could use this and enforce a umask for tests who rely on
this. I believe however (let me know what you think) that hard coded
values are harder to read in the test.
# User-Facing Changes
N/A
# Tests + Formatting
All green on my side after this MR 👍
# After Submitting
Documentation is not impacted
---------
Co-authored-by: Stefan Holderbach <sholderbach@users.noreply.github.com>
# Description
Removes the `group` command that was deprecated back in 0.96.0 with
#13377.
# User-Facing Changes
Breaking change, removed `group` command.
# Description
Fixes: #13425
Similar to `source-env`, `use` command should also remove `FILE_PWD` and
`CURRENT_FILE` after evaluating code block in the module file.
And user input can be a directory, in this case, we need to use the
return value of `find_in_dirs_env` carefully, so in case, I renamed
`maybe_file_path` to `maybe_file_path_or_dir` to emphasize it.
# User-Facing Changes
`$env.FILE_PWD` and `$env.CURRENT_FILE` will be more reliable to use.
# Tests + Formatting
Added 2 test cases.
# After Submitting
NaN
# Description
Closes#12535
Implements sort-by functionality of #8322
Fixes sort-by part of #8667
This PR does two main things: add a new cell path and closure parameter
to `sort-by`, and attempt to make Nushell's sorting behavior
well-defined.
## `sort-by` features
The `columns` parameter is replaced with a `comparator` parameter, which
can be a cell path or a closure. Examples are from docs PR.
1. Cell paths
The basic interactive usage of `sort-by` is the same. For example, `ls |
sort-by modified` still works the same as before. It is not quite a
drop-in replacement, see [behavior changes](#behavior-changes).
Here's an example of how the cell path comparator might be useful:
```nu
> let cities = [
{name: 'New York', info: { established: 1624, population: 18_819_000 } }
{name: 'Kyoto', info: { established: 794, population: 37_468_000 } }
{name: 'São Paulo', info: { established: 1554, population: 21_650_000 }
}
]
> $cities | sort-by info.established
╭───┬───────────┬────────────────────────────╮
│ # │ name │ info │
├───┼───────────┼────────────────────────────┤
│ 0 │ Kyoto │ ╭─────────────┬──────────╮ │
│ │ │ │ established │ 794 │ │
│ │ │ │ population │ 37468000 │ │
│ │ │ ╰─────────────┴──────────╯ │
│ 1 │ São Paulo │ ╭─────────────┬──────────╮ │
│ │ │ │ established │ 1554 │ │
│ │ │ │ population │ 21650000 │ │
│ │ │ ╰─────────────┴──────────╯ │
│ 2 │ New York │ ╭─────────────┬──────────╮ │
│ │ │ │ established │ 1624 │ │
│ │ │ │ population │ 18819000 │ │
│ │ │ ╰─────────────┴──────────╯ │
╰───┴───────────┴────────────────────────────╯
```
2. Key closures
You can supply a closure which will transform each value into a sorting
key (without changing the underlying data). Here's an example of a key
closure, where we want to sort a list of assignments by their average
grade:
```nu
> let assignments = [
{name: 'Homework 1', grades: [97 89 86 92 89] }
{name: 'Homework 2', grades: [91 100 60 82 91] }
{name: 'Exam 1', grades: [78 88 78 53 90] }
{name: 'Project', grades: [92 81 82 84 83] }
]
> $assignments | sort-by { get grades | math avg }
╭───┬────────────┬───────────────────────╮
│ # │ name │ grades │
├───┼────────────┼───────────────────────┤
│ 0 │ Exam 1 │ [78, 88, 78, 53, 90] │
│ 1 │ Project │ [92, 81, 82, 84, 83] │
│ 2 │ Homework 2 │ [91, 100, 60, 82, 91] │
│ 3 │ Homework 1 │ [97, 89, 86, 92, 89] │
╰───┴────────────┴───────────────────────╯
```
3. Custom sort closure
The `--custom`, or `-c`, flag will tell `sort-by` to interpret closures
as custom sort closures. A custom sort closure has two parameters, and
returns a boolean. The closure should return `true` if the first
parameter comes _before_ the second parameter in the sort order.
For a simple example, we could rewrite a cell path sort as a custom sort
(see
[here](https://github.com/nushell/nushell.github.io/pull/1568/files#diff-a7a233e66a361d8665caf3887eb71d4288000001f401670c72b95cc23a948e86R231)
for a more complex example):
```nu
> ls | sort-by -c {|a, b| $a.size < $b.size }
╭───┬─────────────────────┬──────┬──────────┬────────────────╮
│ # │ name │ type │ size │ modified │
├───┼─────────────────────┼──────┼──────────┼────────────────┤
│ 0 │ my-secret-plans.txt │ file │ 100 B │ 10 minutes ago │
│ 1 │ shopping_list.txt │ file │ 100 B │ 2 months ago │
│ 2 │ myscript.nu │ file │ 1.1 KiB │ 2 weeks ago │
│ 3 │ bigfile.img │ file │ 10.0 MiB │ 3 weeks ago │
╰───┴─────────────────────┴──────┴──────────┴────────────────╯
```
## Making sort more consistent
I think it's important for something as essential as `sort` to have
well-defined semantics. This PR contains some changes to try to make the
behavior of `sort` and `sort-by` consistent. In addition, after working
with the internals of sorting code, I have a much deeper understanding
of all of the edge cases. Here is my attempt to try to better define
some of the semantics of sorting (if you are just interested in changes,
skip to "User-Facing changes")
- `sort`, `sort -v`, and `sort-by` now all work the same. Each
individual sort implementation has been refactored into two functions in
`sort_utils.rs`: `sort`, and `sort_by`. These can also be used in other
parts of Nushell where values need to be sorted.
- `sort` and `sort-by` used to handle `-i` and `-n` differently.
- `sort -n` would consider all values which can't be coerced into a
string to be equal
- `sort-by -i` and `sort-by -n` would only work if all values were
strings
- In this PR, insensitive sort only affects comparison between strings,
and natural sort only applies to numbers and strings (see below).
- (not a change) Before and after this PR, `sort` and `sort-by` support
sorting mixed types. There was a lot of discussion about potentially
making `sort` and `sort-by` only work on lists of homogeneous types, but
the general consensus was that `sort` should not error just because its
input contains incompatible types.
- In order to try to make working with data containing `null` values
easier, I changed the PartialOrd order to sort `Nothing` values to the
end of a list, regardless of what other types the list contains. Before,
`null` would be sorted before `Binary`, `CellPath`, and `Custom` values.
- (not a change) When sorted, lists of mixed types will contain sorted
values of each type in order, for the most part
- (not a change) For example, `[0x[1] (date now) "a" ("yesterday" | into
datetime) "b" 0x[0]]` will be sorted as `["a", "b", a day ago, now, [0],
[1]]`, where sorted strings appear first, then sorted datetimes, etc.
- (not a change) The exception to this is `Int`s and `Float`s, which
will intermix, `Strings` and `Glob`s, which will intermix, and `None` as
described above. Additionally, natural sort will intermix strings with
ints and floats (see below).
- Natural sort no longer coerce all inputs to strings.
- I did originally make natural only apply to strings, but @fdncred
pointed out that the previous behavior also allowed you to sort numeric
strings with numbers. This seems like a useful feature if we are trying
to support sorting with mixed types, so I settled on coercing only
numbers (int, float). This can be reverted if people don't like it.
- Here is an example of this behavior in action, which is the same
before and after this PR:
```nushell
$ [1 "4" 3 "2"] | sort --natural
╭───┬───╮
│ 0 │ 1 │
│ 1 │ 2 │
│ 2 │ 3 │
│ 3 │ 4 │
╰───┴───╯
```
# User-Facing Changes
## New features
- Replaces the `columns` string parameter of `sort-by` with a cell path
or a closure.
- The cell path parameter works exactly as you would expect
- By default, the `closure` parameter acts as a "key sort"; that is,
each element is transformed by the closure into a sorting key
- With the `--custom` (`-c`) parameter, you can define a comparison
function for completely custom sorting order.
## Behavior changes
<details>
<summary><code>sort -v</code> does not coerce record values to
strings</summary>
This was a bit of a surprising behavior, and is now unified with the
behavior of `sort` and `sort-by`. Here's an example where you can
observe the values being implicitly coerced into strings for sorting, as
they are sorted like strings rather than numbers:
Old behavior:
```nushell
$ {foo: 9 bar: 10} | sort -v
╭─────┬────╮
│ bar │ 10 │
│ foo │ 9 │
╰─────┴────╯
```
New behavior:
```nushell
$ {foo: 9 bar: 10} | sort -v
╭─────┬────╮
│ foo │ 9 │
│ bar │ 10 │
╰─────┴────╯
```
</details>
<details>
<summary>Changed <code>sort-by</code> parameters from
<code>string</code> to <code>cell-path</code> or <code>closure</code>.
Typical interactive usage is the same as before, but if passing a
variable to <code>sort-by</code> it must be a cell path (or closure),
not a string</summary>
Old behavior:
```nushell
$ let sort = "modified"
$ ls | sort-by $sort
╭───┬──────┬──────┬──────┬────────────────╮
│ # │ name │ type │ size │ modified │
├───┼──────┼──────┼──────┼────────────────┤
│ 0 │ foo │ file │ 0 B │ 10 hours ago │
│ 1 │ bar │ file │ 0 B │ 35 seconds ago │
╰───┴──────┴──────┴──────┴────────────────╯
```
New behavior:
```nushell
$ let sort = "modified"
$ ls | sort-by $sort
Error: nu:🐚:type_mismatch
× Type mismatch.
╭─[entry #10:1:14]
1 │ ls | sort-by $sort
· ──┬──
· ╰── Cannot sort using a value which is not a cell path or closure
╰────
$ let sort = $."modified"
$ ls | sort-by $sort
╭───┬──────┬──────┬──────┬───────────────╮
│ # │ name │ type │ size │ modified │
├───┼──────┼──────┼──────┼───────────────┤
│ 0 │ foo │ file │ 0 B │ 10 hours ago │
│ 1 │ bar │ file │ 0 B │ 2 minutes ago │
╰───┴──────┴──────┴──────┴───────────────╯
```
</details>
<details>
<summary>Insensitve and natural sorting behavior reworked</summary>
Previously, the `-i` and `-n` worked differently for `sort` and
`sort-by` (see "Making sort more consistent"). Here are examples of how
these options result in different sorts now:
1. `sort -n`
- Old behavior (types other than numbers, strings, dates, and binary
sorted incorrectly)
```nushell
$ [2sec 1sec] | sort -n
╭───┬──────╮
│ 0 │ 2sec │
│ 1 │ 1sec │
╰───┴──────╯
```
- New behavior
```nushell
$ [2sec 1sec] | sort -n
╭───┬──────╮
│ 0 │ 1sec │
│ 1 │ 2sec │
╰───┴──────╯
```
2. `sort-by -i`
- Old behavior (uppercase words appear before lowercase words as they
would in a typical sort, indicating this is not actually an insensitive
sort)
```nushell
$ ["BAR" "bar" "foo" 2 "FOO" 1] | wrap a | sort-by -i a
╭───┬─────╮
│ # │ a │
├───┼─────┤
│ 0 │ 1 │
│ 1 │ 2 │
│ 2 │ BAR │
│ 3 │ FOO │
│ 4 │ bar │
│ 5 │ foo │
╰───┴─────╯
```
- New behavior (strings are sorted stably, indicating this is an
insensitive sort)
```nushell
$ ["BAR" "bar" "foo" 2 "FOO" 1] | wrap a | sort-by -i a
╭───┬─────╮
│ # │ a │
├───┼─────┤
│ 0 │ 1 │
│ 1 │ 2 │
│ 2 │ BAR │
│ 3 │ bar │
│ 4 │ foo │
│ 5 │ FOO │
╰───┴─────╯
```
3. `sort-by -n`
- Old behavior (natural sort does not work when data contains non-string
values)
```nushell
$ ["10" 8 "9"] | wrap a | sort-by -n a
╭───┬────╮
│ # │ a │
├───┼────┤
│ 0 │ 8 │
│ 1 │ 10 │
│ 2 │ 9 │
╰───┴────╯
```
- New behavior
```nushell
$ ["10" 8 "9"] | wrap a | sort-by -n a
╭───┬────╮
│ # │ a │
├───┼────┤
│ 0 │ 8 │
│ 1 │ 9 │
│ 2 │ 10 │
╰───┴────╯
```
</details>
<details>
<summary>
Sorting a list of non-record values with a non-existent column/path now
errors instead of sorting the values directly (<code>sort</code> should
be used for this, not <code>sort-by</code>)
</summary>
Old behavior:
```nushell
$ [2 1] | sort-by foo
╭───┬───╮
│ 0 │ 1 │
│ 1 │ 2 │
╰───┴───╯
```
New behavior:
```nushell
$ [2 1] | sort-by foo
Error: nu:🐚:incompatible_path_access
× Data cannot be accessed with a cell path
╭─[entry #29:1:17]
1 │ [2 1] | sort-by foo
· ─┬─
· ╰── int doesn't support cell paths
╰────
```
</details>
<details>
<summary><code>sort</code> and <code>sort-by</code> output
<code>List</code> instead of <code>ListStream</code> </summary>
This isn't a meaningful change (unless I misunderstand the purpose of
ListStream), since `sort` and `sort-by` both need to collect in order to
do the sorting anyway, but is user observable.
Old behavior:
```nushell
$ ls | sort | describe -d
╭──────────┬───────────────────╮
│ type │ stream │
│ origin │ nushell │
│ subtype │ {record 3 fields} │
│ metadata │ {record 1 field} │
╰──────────┴───────────────────╯
```
```nushell
$ ls | sort-by name | describe -d
╭──────────┬───────────────────╮
│ type │ stream │
│ origin │ nushell │
│ subtype │ {record 3 fields} │
│ metadata │ {record 1 field} │
╰──────────┴───────────────────╯
```
New behavior:
```nushell
ls | sort | describe -d
╭────────┬─────────────────╮
│ type │ list │
│ length │ 22 │
│ values │ [table 22 rows] │
╰────────┴─────────────────╯
```
```nushell
$ ls | sort-by name | describe -d
╭────────┬─────────────────╮
│ type │ list │
│ length │ 22 │
│ values │ [table 22 rows] │
╰────────┴─────────────────╯
```
</details>
- `sort` now errors when nothing is piped in (`sort-by` already did
this)
# Tests + Formatting
I added lots of unit tests on the new sort implementation to enforce new
sort behaviors and prevent regressions.
# After Submitting
See [docs PR](https://github.com/nushell/nushell.github.io/pull/1568),
which is ~2/3 finished.
---------
Co-authored-by: NotTheDr01ds <32344964+NotTheDr01ds@users.noreply.github.com>
Co-authored-by: Ian Manske <ian.manske@pm.me>
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx
you can also mention related issues, PRs or discussions!
-->
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.
Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
Maybe we can deprecate `encode new-base64` and `decode new-base64`
first, to make the code clean and simple I'd rather remove the old
`encode base64` and `decode base64` and replace them with the `*
new-base64` commands.
Related PR: https://github.com/nushell/nushell/pull/13428
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
- `encode new-base64` --> `encode base64`
- `decode new-base64` --> `decode base64`
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
It's a breaking change
# Description
Fixes a couple panics:
```
> {} | inspect
Error: x Main thread panicked.
|-> at crates/nu-command/src/debug/inspect_table.rs:87:15
`-> attempt to divide by zero
```
```
> {} | explore
# see an empty column, press Down
Error: x Main thread panicked.
|-> at crates/nu-explore/src/views/cursor/mod.rs:39:9
`-> attempt to subtract with overflow
```
# User-Facing Changes
`{} | inspect` now outputs an empty table:
```
╭─────────────┬────────╮
│ description │ record │
├─────────────┴────────┤
│ │
├──────────────────────┤
```
`{} | explore` opens the help menu.
Both match the empty list behavior.
# Tests
I'm not sure how to test `explore`, as it waits for interaction.
# Description
Partialy addresses #13868. `try` does not catch non-zero exit code
errors from the last command in a pipeline if the result is assigned to
a variable using `let` (or `mut`).
This was fixed by adding a new `OutDest::Value` case. This is used when
the pipeline is in a "value" position. I.e., it will be collected into a
value. This ended up replacing most of the usages of `OutDest::Capture`.
So, this PR also renames `OutDest::Capture` to `OutDest::PipeSeparate`
to better fit the few remaining use cases for it.
# User-Facing Changes
Bug fix.
# Tests + Formatting
Added two tests.
# Description
Similar to #13870 (thanks @WindSoilder), this PR adds a boolean which
determines whether to ignore any errors from an external command. This
is in order to fix#13876. I.e., `do -p` does not wait for externals to
complete before continuing.
# User-Facing Changes
Bug fix.
# Tests + Formatting
Added a test.
This allows parsing of data (e.g. key-value pairs) where the last column
may contain the delimiter.
- this PR should close#13742
# Description
Adds a `--number (-n)` flag to `split column`, analogous to `split row
--number`.
```
~> ['author: Salina Yoon' r#'title: Where's Ellie?: A Hide-and-Seek Book'#] | split column --number 2 ': ' key value
╭───┬────────┬──────────────────────────────────────╮
│ # │ key │ value │
├───┼────────┼──────────────────────────────────────┤
│ 0 │ author │ Salina Yoon │
│ 1 │ title │ Where's Ellie?: A Hide-and-Seek Book │
╰───┴────────┴──────────────────────────────────────╯
```
# User-Facing Changes
* `split column` gains a `--number` option
# Tests + Formatting
Tests included in strings::split::column::test::test_examples and
commands::split_column::to_column.
# After Submitting
Reference documentation is auto-generated from code. No other
documentation updates necessary.
# Description
Fixes: #13460
The issue is caused by `try_exists` method on path, it will return
`Err(NotADirectory)` if user tried to check for a path under a regular
file.
To fix it, I think it's ok to use `exists` rather than `try_exists`,
although
[Path::exists()](https://doc.rust-lang.org/std/path/struct.Path.html#method.exists)
only checks whether or not a path was both found and readable. I think
it's ok, and we can add this information under `extra_description`.
# User-Facing Changes
The following code will no longer raise error:
```
touch a; 'a/b' | path exists
```
# Tests + Formatting
Added 1 test.
# Description
Fixes a bug in the IR for `try` to match that of the regular evaluator
(continuing from #13515):
```nushell
# without IR:
try { ^false } catch { 'caught' } # == 'caught'
# with IR:
try { ^false } catch { 'caught' } # error, non-zero exit code
```
In this PR, both now evaluate to `caught`. For the implementation, I had
to add another instruction, and feel free to suggest better
alternatives. In the future, it might be possible to get rid of this
extra instruction.
# User-Facing Changes
Bug fix, `try { ^false } catch { 'caught' }` now works in IR.
# Description
This PR makes it so that non-zero exit codes and termination by signal
are treated as a normal `ShellError`. Currently, these are silent
errors. That is, if an external command fails, then it's code block is
aborted, but the parent block can sometimes continue execution. E.g.,
see #8569 and this example:
```nushell
[1 2] | each { ^false }
```
Before this would give:
```
╭───┬──╮
│ 0 │ │
│ 1 │ │
╰───┴──╯
```
Now, this shows an error:
```
Error: nu:🐚:eval_block_with_input
× Eval block failed with pipeline input
╭─[entry #1:1:2]
1 │ [1 2] | each { ^false }
· ┬
· ╰── source value
╰────
Error: nu:🐚:non_zero_exit_code
× External command had a non-zero exit code
╭─[entry #1:1:17]
1 │ [1 2] | each { ^false }
· ──┬──
· ╰── exited with code 1
╰────
```
This PR fixes#12874, fixes#5960, fixes#10856, and fixes#5347. This
PR also partially addresses #10633 and #10624 (only the last command of
a pipeline is currently checked). It looks like #8569 is already fixed,
but this PR will make sure it is definitely fixed (fixes#8569).
# User-Facing Changes
- Non-zero exit codes and termination by signal now cause an error to be
thrown.
- The error record value passed to a `catch` block may now have an
`exit_code` column containing the integer exit code if the error was due
to an external command.
- Adds new config values, `display_errors.exit_code` and
`display_errors.termination_signal`, which determine whether an error
message should be printed in the respective error cases. For
non-interactive sessions, these are set to `true`, and for interactive
sessions `display_errors.exit_code` is false (via the default config).
# Tests
Added a few tests.
# After Submitting
- Update docs and book.
- Future work:
- Error if other external commands besides the last in a pipeline exit
with a non-zero exit code. Then, deprecate `do -c` since this will be
the default behavior everywhere.
- Add a better mechanism for exit codes and deprecate
`$env.LAST_EXIT_CODE` (it's buggy).
# Description
The content-type was not being handled appropriately when sending
requests with a string value.
# User-Facing Changes
- Passing a string value through a pipeline with a content-type set as
metadata is handled correctly
- Passing a string value as a parameter with a content-type set as a
flag is handled correctly.
# Description
After looking at #13751 I noticed that the config setting
`use_grid_icons` seems out of place. So, I removed it from the config
and made it a parameter to the `grid` command.
Closes#13687Closes#13686
# Description
Light refactoring of `send_request `in `client.rs`. In the end there are
more lines but now the logic is more concise and facilitates adding new
conditions in the future. Unit tests ran fine and I tested a few cases
manually.
Cool project btw, I'll be using nushell from now on.
# Description
This changes the behavior of `tee` to be more transparent when given a
value that isn't a list or range. Previously, anything that wasn't a
byte stream would converted to a list stream using the iterator
implementation, which led to some surprising results. Instead, now, if
the value is a string or binary, it will be treated the same way a byte
stream is, and the output of `tee` is a byte stream instead of the
original value. This is done so that we can synchronize with the other
thread on collect, and potentially capture any error produced by the
closure.
For values that can't be converted to streams, the closure is just run
with a clone of the value instead on another thread. Because we can't
wait for the other thread, there is no way to send an error back to the
original thread, so instead it's just written to stderr using
`report_error_new()`.
There are a couple of follow up edge cases I see where byte streams
aren't necessarily treated exactly the same way strings are, but this
should mostly be a good experience.
Fixes#13489.
# User-Facing Changes
Breaking change.
- `tee` now outputs and sends string/binary stream for string/binary
input.
- `tee` now outputs and sends the original value for any other input
other than lists/ranges.
# Tests + Formatting
Added for new behavior.
# After Submitting
- [ ] release notes: breaking change, command change
Mistakes have been made. I forgot about a bunch of `todo`s in the helper
functions. So, this PR replaces them with proper errors. It also adds
tests for parse-time evaluation, because one `todo` I missed was in a
`run_const` function.
Based on the discussion in #13419.
## Description
Reworks the `decode`/`encode` commands by adding/changing the following
bases:
- `base32`
- `base32hex`
- `hex`
- `new-base64`
The `hex` base is compatible with the previous version of `hex` out of
the box (it only adds more flags). `base64` isn't, so the PR adds a new
version and deprecates the old one.
All commands have `string -> binary` signature for decoding and `string
| binary -> string` signature for encoding. A few `base64` encodings,
which are not a part of the
[RFC4648](https://datatracker.ietf.org/doc/html/rfc4648#section-6), have
been dropped.
## Example usage
```Nushell
~/fork/nushell> "string" | encode base32 | decode base32 | decode
string
```
```Nushell
~/fork/nushell> "ORSXG5A=" | decode base32
# `decode` always returns a binary value
Length: 4 (0x4) bytes | printable whitespace ascii_other non_ascii
00000000: 74 65 73 74 test
```
## User-Facing Changes
- New commands: `encode/decode base32/base32hex`.
- `encode hex` gets a `--lower` flag.
- `encode/decode base64` deprecated in favor of `encode/decode
new-base64`.
# Description
The meaning of the word usage is specific to describing how a command
function is *used* and not a synonym for general description. Usage can
be used to describe the SYNOPSIS or EXAMPLES sections of a man page
where the permitted argument combinations are shown or example *uses*
are given.
Let's not confuse people and call it what it is a description.
Our `help` command already creates its own *Usage* section based on the
available arguments and doesn't refer to the description with usage.
# User-Facing Changes
`help commands` and `scope commands` will now use `description` or
`extra_description`
`usage`-> `description`
`extra_usage` -> `extra_description`
Breaking change in the plugin protocol:
In the signature record communicated with the engine.
`usage`-> `description`
`extra_usage` -> `extra_description`
The same rename also takes place for the methods on
`SimplePluginCommand` and `PluginCommand`
# Tests + Formatting
- Updated plugin protocol specific changes
# After Submitting
- [ ] update plugin protocol doc
# Description
The previous behaviour of `into record` on lists was to create a new
record with each list index as the key. This was not very useful for
creating meaningful records, though, and most people would end up using
commands like `headers` or `transpose` to turn a list of keys and values
into a record.
This PR changes that instead to do what I think the most ergonomic thing
is, and instead:
- A list of records is merged into one record.
- A list of pairs (two element lists) is folded into a record with the
first element of each pair being the key, and the second being the
value.
The former is just generally more useful than having to use `reduce`
with `merge` for such a common operation, and the latter is useful
because it means that `$a | zip $b | into record` *just works* in the
way that seems most obvious.
Example:
```nushell
[[foo bar] [baz quux]] | into record # => {foo: bar, baz: quux}
[{foo: bar} {baz: quux}] | into record # => {foo: bar, baz: quux}
[foo baz] | zip [bar quux] | into record # => {foo: bar, baz: quux}
```
The support for range input has been removed, as it would no longer
reflect the treatment of an equivalent list.
The following is equivalent to the old behavior, in case that's desired:
```
0.. | zip [a b c] | into record # => {0: a, 1: b, 2: c}
```
# User-Facing Changes
- `into record` changed as described above (breaking)
- `into record` no longer supports range input (breaking)
# Tests + Formatting
Examples changed to match, everything works. Some usage in stdlib and
`nu_plugin_nu_example` had to be changed.
# After Submitting
- [ ] release notes (commands, breaking change)
# Description
Previously when nushell failed to parse the content type header, it
would emit an error instead of returning the response. Now it will fall
back to `text/plain` (which, in turn, will trigger type detection based
on file extension).
May fix (potentially) nushell/nushell#11927
Refs:
https://discord.com/channels/601130461678272522/614593951969574961/1272895236489613366
Supercedes: #13609
# User-Facing Changes
It's now possible to fetch content even if the server returns an invalid
content type header. Users may need to parse the response manually, but
it's still better than not getting the response at all.
# Tests + Formatting
Added a test for the new behaviour.
# After Submitting
# Description
Fixes: #13479
# User-Facing Changes
Given the following setup:
```
cd /tmp
touch src_file.txt
ln -s src_file.txt link1
```
### Before
```
ls -lf link1 | get target.0 # It outputs src_file.txt
```
### After
```
ls -lf link1 | get target.0 # It outputs /tmp/src_file.txt
```
# Tests + Formatting
Added a test for the change
# Description
Something I meant to add a long time ago. We currently don't have a
convenient way to print raw binary data intentionally. You can pipe it
through `cat` to turn it into an unknown stream, or write it to a file
and read it again, but we can't really just e.g. generate msgpack and
write it to stdout without this. For example:
```nushell
[abc def] | to msgpack | print --raw
```
This is useful for nushell scripts that will be piped into something
else. It also means that `nu_plugin_nu_example` probably doesn't need to
do this anymore, but I haven't adjusted it yet:
```nushell
def tell_nushell_encoding [] {
print -n "\u{0004}json"
}
```
This happens to work because 0x04 is a valid UTF-8 character, but it
wouldn't be possible if it were something above 0x80.
`--raw` also formats other things without `table`, I figured the two
things kind of go together. The output is kind of like `to text`.
Debatable whether that should share the same flag, but it was easier
that way and seemed reasonable.
# User-Facing Changes
- `print` new flag: `--raw`
# Tests + Formatting
Added tests.
# After Submitting
- [ ] release notes (command modified)
This PR closes [Issue
#13482](https://github.com/nushell/nushell/issues/13482)
# Description
This PR tend to make all math function to be constant.
# User-Facing Changes
The math commands now can be used as constant methods.
### Some Example
```
> const MODE = [3 3 9 12 12 15] | math mode
> $MODE
╭───┬────╮
│ 0 │ 3 │
│ 1 │ 12 │
╰───┴────╯
> const LOG = [16 8 4] | math log 2
> $LOG
╭───┬──────╮
│ 0 │ 4.00 │
│ 1 │ 3.00 │
│ 2 │ 2.00 │
╰───┴──────╯
> const VAR = [1 3 5] | math variance
> $VAR
2.6666666666666665
```
# Tests + Formatting
Tests are added for all of the math command to test there constant
behavior.
I mostly focused on the actual user experience, not the correctness of
the methods and algorithms.
# After Submitting
I think this change don't require any additional documentation. Feel
free to correct me in this topic please.
# Description
Attempt to guess the content type of a file when opening with --raw and
set it in the pipeline metadata.
<img width="644" alt="Screenshot 2024-08-02 at 11 30 10"
src="https://github.com/user-attachments/assets/071f0967-c4dd-405a-b8c8-f7aa073efa98">
# User-Facing Changes
- Content of files can be directly piped into commands like `http post`
with the content type set appropriately when using `--raw`.
# Description
Part 4 of replacing std::path types with nu_path types added in
https://github.com/nushell/nushell/pull/13115. This PR migrates various
tests throughout the code base.
- **Doccomment style fixes**
- **Forgotten stuff in `nu-pretty-hex`**
- **Don't `for` around an `Option`**
- and more
I think the suggestions here are a net positive, some of the suggestions
moved into #13498 feel somewhat arbitrary, I also raised
https://github.com/rust-lang/rust-clippy/issues/13188 as the nightly
`byte_char_slices` would require either a global allow or otherwise a
ton of granular allows or possibly confusing bytestring literals.
# Description
Fixes: #13260
When user run a command like this:
```nushell
$env.FOO = " New";
$env.BAZ = " New Err";
do -i {nu -n -c 'nu --testbin echo_env FOO; nu --testbin echo_env_stderr BAZ'} | save -a -r save_test_22/log.txt
```
`save` command sinks the output of previous commands' stderr output. I
think it should be `stderr`.
# User-Facing Changes
```nushell
$env.FOO = " New";
$env.BAZ = " New Err";
do -i {nu -n -c 'nu --testbin echo_env FOO; nu --testbin echo_env_stderr BAZ'} | save -a -r save_test_22/log.txt
```
The command will output ` New Err` to stderr.
# Tests + Formatting
Added 2 cases.
- **Suggested default impl for the new `*Stack`s**
- **Change a hashmap to make clippy happy**
- **Clone from fix**
- **Fix conditional unused in test**
- then **Bump rust toolchain**
# Description
This makes assignment operations and `const` behave the same way `let`
and `mut` do, absorbing the rest of the pipeline.
Changes the lexer to be able to recognize assignment operators as a
separate token, and then makes the lite parser continue to push spans
into the same command regardless of any redirections or pipes if an
assignment operator is encountered. Because the pipeline is no longer
split up by the lite parser at this point, it's trivial to just parse
the right hand side as if it were a subexpression not contained within
parentheses.
# User-Facing Changes
Big breaking change. These are all now possible:
```nushell
const path = 'a' | path join 'b'
mut x = 2
$x = random int
$x = [1 2 3] | math sum
$env.FOO = random chars
```
In the past, these would have led to (an attempt at) bare word string
parsing. So while `$env.FOO = bar` would have previously set the
environment variable `FOO` to the string `"bar"`, it now tries to run
the command named `bar`, hence the major breaking change.
However, this is desirable because it is very consistent - if you see
the `=`, you can just assume it absorbs everything else to the right of
it.
# Tests + Formatting
Added tests for the new behaviour. Adjusted some existing tests that
depended on the right hand side of assignments being parsed as
barewords.
# After Submitting
- [ ] release notes (breaking change!)
# Description
Close: #12083Close: #12084
# User-Facing Changes
It's a breaking change because we have switched the position of
`<initial>` and `<closure>`, after the change, initial value will be
optional. So it's possible to do something like this:
```nushell
> let f = {|fib = [0, 1]| {out: $fib.0, next: [$fib.1, ($fib.0 + $fib.1)]} }
> generate $f | first 5
╭───┬───╮
│ 0 │ 0 │
│ 1 │ 1 │
│ 2 │ 1 │
│ 3 │ 2 │
│ 4 │ 3 │
╰───┴───╯
```
It will also raise error if user don't give initial value, and the
closure don't have default parameter.
```nushell
❯ let f = {|fib| {out: $fib.0, next: [$fib.1, ($fib.0 + $fib.1)]} }
❯ generate $f
Error: × The initial value is missing
╭─[entry #5:1:1]
1 │ generate $f
· ────┬───
· ╰── Missing intial value
╰────
help: Provide <initial> value in generate, or assigning default value to closure parameter
```
# Tests + Formatting
Added some test cases.
---------
Co-authored-by: Stefan Holderbach <sholderbach@users.noreply.github.com>
# Description
Following from #13377, this PR refactors the related `window` command.
In the case where `window` should behave exactly like `chunks`, it now
reuses the same code that `chunks` does. Otherwise, the other cases have
been rewritten and have resulted in a performance increase:
| window size | stride | old time (ms) | new time (ms) |
| -----------:| ------:| -------------:| -------------:|
| 20 | 1 | 757 | 722 |
| 2 | 1 | 364 | 333 |
| 1 | 1 | 343 | 293 |
| 20 | 20 | 90 | 63 |
| 2 | 2 | 215 | 175 |
| 20 | 30 | 74 | 60 |
| 2 | 4 | 141 | 110 |
# User-Facing Changes
`window` will now error if the window size or stride is 0, which is
technically a breaking change.
# Description
This grew quite a bit beyond its original scope, but I've tried to make
`$in` a bit more consistent and easier to work with.
Instead of the parser generating calls to `collect` and creating
closures, this adds `Expr::Collect` which just evaluates in the same
scope and doesn't require any closure.
When `$in` is detected in an expression, it is replaced with a new
variable (also called `$in`) and wrapped in `Expr::Collect`. During
eval, this expression is evaluated directly, with the input and with
that new variable set to the collected value.
Other than being faster and less prone to gotchas, it also makes it
possible to typecheck the output of an expression containing `$in`,
which is nice. This is a breaking change though, because of the lack of
the closure and because now typechecking will actually happen. Also, I
haven't attempted to typecheck the input yet.
The IR generated now just looks like this:
```gas
collect %in
clone %tmp, %in
store-variable $in, %tmp
# %out <- ...expression... <- %in
drop-variable $in
```
(where `$in` is the local variable created for this collection, and not
`IN_VARIABLE_ID`)
which is a lot better than having to create a closure and call `collect
--keep-env`, dealing with all of the capture gathering and allocation
that entails. Ideally we can also detect whether that input is actually
needed, so maybe we don't have to clone, but I haven't tried to do that
yet. Theoretically now that the variable is a unique one every time, it
should be possible to give it a type - I just don't know how to
determine that yet.
On top of that, I've also reworked how `$in` works in pipeline-initial
position. Previously, it was a little bit inconsistent. For example,
this worked:
```nushell
> 3 | do { let x = $in; let y = $in; print $x $y }
3
3
```
However, this causes a runtime variable not found error on the second
`$in`:
```nushell
> def foo [] { let x = $in; let y = $in; print $x $y }; 3 | foo
Error: nu:🐚:variable_not_found
× Variable not found
╭─[entry #115:1:35]
1 │ def foo [] { let x = $in; let y = $in; print $x $y }; 3 | foo
· ─┬─
· ╰── variable not found
╰────
```
I've fixed this by making the first element `$in` detection *always*
happen at the block level, so if you use `$in` in pipeline-initial
position anywhere in a block, it will collect with an implicit
subexpression around the whole thing, and you can then use that `$in`
more than once. In doing this I also rewrote `parse_pipeline()` and
hopefully it's a bit more straightforward and possibly more efficient
too now.
Finally, I've tried to make `let` and `mut` a lot more straightforward
with how they handle the rest of the pipeline, and using a redirection
with `let`/`mut` now does what you'd expect if you assume that they
consume the whole pipeline - the redirection is just processed as
normal. These both work now:
```nushell
let x = ^foo err> err.txt
let y = ^foo out+err>| str length
```
It was previously possible to accomplish this with a subexpression, but
it just seemed like a weird gotcha that you couldn't do it. Intuitively,
`let` and `mut` just seem to take the whole line.
- closes#13137
# User-Facing Changes
- `$in` will behave more consistently with blocks and closures, since
the entire block is now just wrapped to handle it if it appears in the
first pipeline element
- `$in` no longer creates a closure, so what can be done within an
expression containing `$in` is less restrictive
- `$in` containing expressions are now type checked, rather than just
resulting in `any`. However, `$in` itself is still `any`, so this isn't
quite perfect yet
- Redirections are now allowed in `let` and `mut` and behave pretty much
how you'd expect
# Tests + Formatting
Added tests to cover the new behaviour.
# After Submitting
- [ ] release notes (definitely breaking change)
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx
you can also mention related issues, PRs or discussions!
-->
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.
Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
Replaces the `dirs_next` family of crates with `dirs`. `dirs_next` was
born when the `dirs` crates were abandoned three years ago, but they're
being maintained again and most projects depend on `dirs` nowadays.
`dirs_next` has been abandoned since.
This came up while working on
https://github.com/nushell/nushell/pull/13382.
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
None.
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
Tests and formatter have been run.
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
# Description
Before this change `default` would dive into lists and replace `null`
elements with the default value.
Therefore there was no easy way to process `null|list<null|string>`
input and receive `list<null|string>`
(IOW to apply the default on the top level only).
However it's trivially easy to apply default values to list elements
with the `each` command:
```nushell
[null, "a", null] | each { default "b" }
```
So there's no need to have this behavior in `default` command.
# User-Facing Changes
* `default` no longer dives into lists to replace `null` elements with
the default value.
# Tests + Formatting
Added a couple of tests for the new (and old) behavior.
# After Submitting
* Update docs.
# Description
The name of the `group` command is a little unclear/ambiguous.
Everything I look at it, I think of `group-by`. I think `chunks` more
clearly conveys what the `group` command does. Namely, it divides the
input list into chunks of a certain size. For example,
[`slice::chunks`](https://doc.rust-lang.org/std/primitive.slice.html#method.chunks)
has the same name. So, this PR adds a new `chunks` command to replace
the now deprecated `group` command.
The `chunks` command is a refactored version of `group`. As such, there
is a small performance improvement:
```nushell
# $data is a very large list
> bench { $data | chunks 2 } --rounds 30 | get mean
474ms 921µs 190ns
# deprecation warning was disabled here for fairness
> bench { $data | group 2 } --rounds 30 | get mean
592ms 702µs 440ns
> bench { $data | chunks 200 } --rounds 30 | get mean
374ms 188µs 318ns
> bench { $data | group 200 } --rounds 30 | get mean
481ms 264µs 869ns
> bench { $data | chunks 1 } --rounds 30 | get mean
642ms 574µs 42ns
> bench { $data | group 1 } --rounds 30 | get mean
981ms 602µs 513ns
```
# User-Facing Changes
- `group` command has been deprecated in favor of new `chunks` command.
- `chunks` errors when given a chunk size of `0` whereas `group` returns
chunks with one element.
# Tests + Formatting
Added tests for `chunks`, since `group` did not have any tests.
# After Submitting
Update book if necessary.
# Description
Fixes#13359
In an attempt to generate names for flat columns resulting from a nested
accesses #3016 generated new column names on nested selection, out of
convenience, that composed the cell path as a string (including `.`) and
then simply replaced all `.` with `_`. As we permit `.` in column names
as long as you quote this surprisingly alters `select`ed columns.
# User-Facing Changes
New columns generated by selection with nested cell paths will for now
be named with a string containing the keys separated by `.` instead of
`_`. We may want to reconsider the semantics for nested access.
# Tests + Formatting
- Alter test to breaking change on nested `select`
Somehow this logic was missed on my end. ( I mean I was not even
thinking about it in original patch 😄 )
Please recheck
Added a regression test too.
close#13336
cc: @fdncred
GOOD CATCH.............................................................
SORRY
I've added a test to catch regression just in case.
close#13319
cc: @fdncred
# Description
Bug fix: `PipelineData::check_external_failed()` was not preserving the
original `type_` and `known_size` attributes of the stream passed in for
streams that come from children, so `external-command | into binary` did
not work properly and always ended up still being unknown type.
# User-Facing Changes
The following test case now works as expected:
```nushell
> head -c 2 /dev/urandom | into binary
# Expected: pretty hex dump of binary
# Previous behavior: just raw binary in the terminal
```
# Tests + Formatting
Added a test to cover this to `into binary`
# Description
Provides the ability to use http commands as part of a pipeline.
Additionally, this pull requests extends the pipeline metadata to add a
content_type field. The content_type metadata field allows commands such
as `to json` to set the metadata in the pipeline allowing the http
commands to use it when making requests.
This pull request also introduces the ability to directly stream http
requests from streaming pipelines.
One other small change is that Content-Type will always be set if it is
passed in to the http commands, either indirectly or throw the content
type flag. Previously it was not preserved with requests that were not
of type json or form data.
# User-Facing Changes
* `http post`, `http put`, `http patch`, `http delete` can be used as
part of a pipeline
* `to text`, `to json`, `from json` all set the content_type metadata
field and the http commands will utilize them when making requests.
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx
you can also mention related issues, PRs or discussions!
-->
fixes#13245
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.
Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
In addition to addressing #13245, this PR also updated one of the doc
example to the `find` command to document that non-regex mode is case
insensitive, which may surprise some users.
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
---------
Co-authored-by: Ben Yang <ben@ya.ng>
Co-authored-by: Darren Schroeder <343840+fdncred@users.noreply.github.com>
