When specifying `seek=N` and *not* specifying `conv=notrunc`, truncate
the output file to `N` blocks instead of truncating it to zero before
starting to write output. For example
$ printf "abc" > outfile
$ printf "123" | dd bs=1 skip=1 seek=1 count=1 status=noxfer of=outfile
1+0 records in
1+0 records out
$ cat outfile
a2
Fixes#3068.
When this option is present, the files argument is not processed. This option processes the file list from provided file, splitting them by the ascii NUL (\0) character. When files0-from is '-', the file list is processed from stdin.
Using this escaped character will cause `printf` to stop generating characters.
For instance,
```rust
hbina@akarin ~/g/uutils (hbina-add-test-for-additional-escape)> cargo run --quiet -- printf "%s\c%s" a b
a⏎
```
Signed-off-by: Hanif Ariffin <hanif.ariffin.4326@gmail.com>
* test_sort: Output sorted files to a file with different name
Signed-off-by: Hanif Bin Ariffin <hanif.ariffin.43262@gmail.com>
* Fix the test by saving the environment variable
Signed-off-by: Hanif Bin Ariffin <hanif.ariffin.43262@gmail.com>
Co-authored-by: Hanif Bin Ariffin <hanif.ariffin.43262@gmail.com>
This avoids hacking around the short options of these command line
arguments that have been introduced by clap. Additionally, we test and
correctly handle the combination of both version and help. The GNU
binary will ignore both arguments in this case while clap would perform
the first one. A test for this edge case was added.
This allows for `-t` to take invalid unicode (but still single-byte) values
on unix-like platforms. Other platforms, which as of the time of this commit
do not support `OsStr::as_bytes()`, could possibly be supported in the future,
but would require design decisions as to what that means.
- Change the main! proc_macro to a bin! macro_rules macro.
- Reexport uucore_procs from uucore
- Make utils to not import uucore_procs directly
- Remove the `syn` dependency and don't parse proc_macro input (hopefully for faster compile times)
Prevent usize underflow when reducing the size of a file by more than
its current size. For example, if `f` is a file with 3 bytes, then
truncate -s-5 f
will now set the size of the file to 0 instead of causing a panic.
Improve the error message that gets printed when a directory does not
exist. After this commit, the error message is
truncate: cannot open '{file}' for writing: No such file or directory
where `{file}` is the name of a file in a directory that does not
exist.
Change a word in the error message displayed when an increment value
of 0 is provided to `seq`. This commit changes the message from "Zero
increment argument" to "Zero increment value" to match the GNU `seq`
error message.
Add an error for division by zero. Previously, running `truncate -s /0
file` or `-s %0` would panic due to division by zero. After this
change, it writes an error message "division by zero" to stderr and
terminates with an error code.
Add support for the `-f FORMAT` option to `seq`. This option instructs
the program to render each value in the generated sequence using a
given `printf`-style floating point format. For example,
$ seq -f %.2f 0.0 0.1 0.5
0.00
0.10
0.20
0.30
0.40
0.50
Fixes issue #2616.
Fix a bug where `tail -f` would terminate with an error due to failing
to parse a UTF-8 string from a sequence of bytes read from the
followed file. This commit replaces the call to `BufRead::read_line()`
with a call to `BufRead::read_until()` so that any sequence of bytes
regardless of encoding can be read.
Fixes#1050.
Correct the behavior of `dd` with the `status=noxfer` option. Before
this commit, the status output was entirely suppressed (as happens
with `status=none`). This was incorrect behavior. After this commit,
the input/output counts are printed to stderr as expected.
For example,
$ printf "" | dd status=noxfer
0+0 records in
0+0 records out
This commit also updates a unit test that was enforcing the wrong
behavior.
Fix the behavior of truncate when given a non-existent file so that it
correctly creates the file before truncating it (unless the
`--no-create` option is also given).
Fix a bug when getting all but the first NUM lines or bytes of a file
via `tail -n +NUM <file>` or `tail -c +NUM <file>`. The bug only
existed when a file is given as an argument; it did not exist when the
input data came from stdin.
Support `-z` option when the input is not a seekable file. Previously,
the option was accepted by the argument parser, but it was being
ignored by the application logic.
This expands the error message that is printed if either input file has
an unsorted line. Both the program name (join) and the offending line
are printed out with the message to match the behaviour of the GNU
utility.
This commit replaces generic Results with UResults in some key
functions in numfmt. As a result of this, we can provide different
exit codes for different errors, which resolves ~70 failing test
cases in the GNU numfmt.pl test suite.
Fix two issues with the filename creation algorithm. First, this
corrects the behavior of the `-a` option. This commit ensures a
failure occurs when the number of chunks exceeds the number of
filenames representable with the specified fixed width:
$ printf "%0.sa" {1..11} | split -d -b 1 -a 1
split: output file suffixes exhausted
Second, this corrects the behavior of the default behavior when `-a`
is not specified on the command line. Previously, it was always
settings the filenames to have length 2 suffixes. This commit corrects
the behavior to follow the algorithm implied by GNU split, where the
filename lengths grow dynamically by two characters once the number of
chunks grows sufficiently large:
$ printf "%0.sa" {1..91} | ./target/debug/coreutils split -d -b 1 \
> && ls x* | tail
x81
x82
x83
x84
x85
x86
x87
x88
x89
x9000
* print error in the correct order by flushing the stdout buffer before printing an error
* print correct GNU error codes
* correct formatting for config.inode, and for dangling links
* correct padding for Format::Long
* remove colors after the -> link symbol as this doesn't match GNU
* correct the major, minor #s for char devices, and correct padding
* improve speed for all metadata intensive ops by not allocating metadata unless in a Sort mode
* new tests, have struggled with how to deal with stderr, stdout ordering in a test though
* tried to implement UIoError, but am still having issues matching the formatting of GNU
Co-authored-by: electricboogie <32370782+electricboogie@users.noreply.github.com>
Fix a bug in which a negative decimal input would not be displayed with
the correct width in the output. Before this commit, the output was
incorrectly
$ seq -w -.1 .1 .11
-0.1
0.0
0.1
After this commit, the output is correctly
$ seq -w -.1 .1 .11
-0.1
00.0
00.1
The code was failing to take into account that the input decimal "-.1"
needs to be displayed with a leading zero, like "-0.1".
Pad infinity and negative infinity values with spaces when using the
`-w` option to `seq`. This corrects the behavior of `seq` to match that
of the GNU version:
$ seq -w 1.000 inf inf | head -n 4
1.000
inf
inf
inf
Previously, it incorrectly padded with 0s instead of spaces.
* seq: use BigDecimal to represent floats
Use `BigDecimal` to represent arbitrary precision floats in order to
prevent numerical precision issues when iterating over a sequence of
numbers. This commit makes several changes at once to accomplish this
goal.
First, it creates a new struct, `PreciseNumber`, that is responsible for
storing not only the number itself but also the number of digits (both
integer and decimal) needed to display it. This information is collected
at the time of parsing the number, which lives in the new
`numberparse.rs` module.
Second, it uses the `BigDecimal` struct to store arbitrary precision
floating point numbers instead of the previous `f64` primitive
type. This protects against issues of numerical precision when
repeatedly accumulating a very small increment.
Third, since neither the `BigDecimal` nor `BigInt` types have a
representation of infinity, minus infinity, minus zero, or NaN, we add
the `ExtendedBigDecimal` and `ExtendedBigInt` enumerations which extend
the basic types with these concepts.
* fixup! seq: use BigDecimal to represent floats
* fixup! seq: use BigDecimal to represent floats
* fixup! seq: use BigDecimal to represent floats
* fixup! seq: use BigDecimal to represent floats
* fixup! seq: use BigDecimal to represent floats
GNU rm allows the `-r` flag to be specified multiple times, but
uutils/coreutils would previously exit with an error.
I encountered this while attempting to run `make clean` on the
Postgres source tree (github.com/postgres/postgres).
Updates #1663.
This makes uu_tail pass the "gnu/tests/tail-2/descriptor-vs-rename" test.
* add tests for descriptor-vs-rename (with/without verbose)
* fix some minor error messages
* Change data structure from Vec to HashMap in order to better
keep track of files while watching them with `--follow=name`.
E.g. file paths that were removed while watching them and exit
if no files are remaining, etc.
* Move all logic related to file handling into a FileHandling trait
* Simplify handling of the verbose flag.
For the CICD on macOS, this fixes:
```
---- common::util::tests::test_check_coreutil_version stdout ----
---- common::util::tests::test_expected_result stdout ----
thread 'common::util::tests::test_expected_result' panicked at
'byte index 4 is out of bounds of `9.0`', tests/common/util.rs:1172:41
```
Replace two loops that print leading and trailing 0s when printing a
number in fixed-width mode with a single call to `write!()` with the
appropriate formatting parameters.
Ensure that the `print_seq_integers()` function is called when the first
number and the increment are integers, regardless of the type of the
last value specified.
Change the way `seq` computes the number of digits needed to print a
number so that it works for inputs given in scientific notation.
Specifically, this commit parses the input string to determine whether
it is an integer, a float in decimal notation, or a float in scientific
notation, and then computes the number of integral digits and the number
of fractional digits based on that. This also supports floating point
negative zero, expressed in both decimal and scientific notation.
This fixes an issue on macOS/bsd where multiple calls
to `host_name_for` each added a prefix 'g'.
The issue led to some tests being skipped because `util_name` was
renamed to e.g. "ggid" instead of "gid".
This makes it no longer possible to pass multiple filenames, but every
other implementation I tried (GNU, busybox, FreeBSD, sbase) also
forbids that so I think it's for the best.
* chown: support '.' as 'user.group' separator (like ':')
It also manages the case:
user.name:group (valid too)
Should fix:
gnu/tests/chown/separator.sh
* Fixed some documentation of display_item_long that was missed in #2623
* Implemented coloring of `ls -l` symlink targets( after the arrow `->`).
* Documented display_file_name to some extent.
* Ran rustfmt as part of mitigating CI chain errors.
* Removed unused variables and code in test_ls_long_format as per #2623
specifically, as per
https://github.com/uutils/coreutils/pull/2623#pullrequestreview-742386707
* Added a thorough test for `ls -laR --color` symlink coloring implemented in this branch.
* renamed test files and dirs to shorter names and ran rustfmt.
* Changed the order with which files are expected to match the change in their name.
* Bettered some comments.
* Removed needless borrow. Fixed that one clippy warning.
* Moved the cfg not windows up to the function level
because this function is meant to only run in non-windows OS (with
groups and unames).
Fixes the unused variable warning in CI.
- prevent duplicate errors from both us and `walkdir` by instructing `walkdir'
to skip directories we failed to read metadata for.
- don't directly display `walkdir`'s errors, but format them ourselves to
match gnu's format
Fix a mix-up between ownership and mode. The latter (mode / file permissions)
can also be set on windows (which however only affects the read-only flag),
while there doesn't seem to be a straight-forward way to change file ownership
on windows.
Copy the acl as well when copying the mode. This is a non-default feature and can be
enabled with --features feat_acl, because it doesn't seem to work on CI.
It is only available for unix so far.
Copy the SELinux context if possible.
Makes the -o auto option construct a format at initialization, rather
than try to handle it as a special case when printing lines. Fixes bugs
when combined with -e, especially when combined with -a.
* Used .as_path() and .as_str() when required:
when the argument required is a Path and not a PathBuf,
or an str and not a Path, respectively.
* Changed display_items to take Vec<PathData>, which is passed, instead of [PathData]
* Added a pad_right function.
* Implemented column-formating to mimic the behavior of GNU coreutils's ls
Added returns in display_dir_entry_size that keep track of uname and
group lengths.
Renamed variables to make more sense.
Changed display_item_long to take all the lengths it needs to render
correctly.
Implemented owner, group, and author padding right to mimic GNU ls.
* Added a todo for future quality-of-life cache addition.
* Documented display_item_long, as a first step in documenting all functions.
* Revert "Used .as_path() and .as_str() when required:"
This reverts commit b88db6a817.
* Revert "Changed display_items to take Vec<PathData>, which is passed, instead of [PathData]"
This reverts commit 0c690dda8d.
* Ran cargo fmt to get rid of Style/format `fmt` testing error.
* Added a test for `ls -l` and `ls -lan` line formats.
* Changed uname -> username for cspell. Removed extra blank line for rustfmt.
Fix a bug in `seq` where the number of characters needed to print the
number was computed incorrectly in some cases. This commit changes the
computation of the width to be after parsing the number instead of
before, in order to accommodate inputs like `1e3`, which requires four
digits when printing the number, not three.
- Implement all of GNU's fiddly little details
- Don't assume Linux for error codes
- Accept badly-encoded filenames
- Report errors after the fact instead of checking ahead of time
- General cleanup
rmdir now passes GNU's tests.
Errors are now always shown with the corresponding filename.
Errors are no longer converted into warnings. Previously `wc < .`
would cause a loop.
Checking whether something is a directory is no longer done in
advance. This removes race conditions and the edge case where stdin is
a directory.
The custom error type is removed because io::Error is now enough.
Report errors properly instead of panicking.
Replace zero_copy by a simpler specialized private module.
Do not assume splices move all data at once.
Use the modern uutils machinery.
Remove the "latency" feature. The time it takes to prepare the buffer
is drowned out by the startup time anyway.
yes: Add tests
yes: Fix long input test on Windows
Fix a bug in `tac` where multi-character line separators would cause
incorrect behavior when there was overlap between candidate matches in
the input string. This commit adds a dependency on `memchr` in order to
use the `memchr::memmem::rfind_iter()` function to scan for
non-overlapping instances of the specified line separator characters,
scanning from right to left.
Fixes#2580.
* hashsum: support --check for var. length outputs
Add the ability for `hashsum --check` to work with algorithms with
variable output length. Previously, the program would terminate with an
error due to constructing an invalid regular expression.
* fixup! hashsum: support --check for var. length outputs
* tac: correct behavior of -b option
Correct the behavior of `tac -b` to match that of GNU coreutils
`tac`. Specifically, this changes `tac -b` to assume *leading* line
separators instead of the default *trailing* line separators.
Before this commit, the (incorrect) behavior was
$ printf "/abc/def" | tac -b -s "/"
def/abc/
After this commit, the behavior is
$ printf "/abc/def" | tac -b -s "/"
/def/abc
Fixes#2262.
* fixup! tac: correct behavior of -b option
* fixup! tac: correct behavior of -b option
Co-authored-by: Sylvestre Ledru <sylvestre@debian.org>
Hopefully will be feature parity with GNU `tr`.
Signed-off-by: Hanif Bin Ariffin <hanif.ariffin.4326@gmail.com>
Implemented a bit of new expansion module
Signed-off-by: Hanif Bin Ariffin <hanif.ariffin.4326@gmail.com>
Implemented delete operation
Signed-off-by: Hanif Bin Ariffin <hanif.ariffin.4326@gmail.com>
Partially implemented delete operation
Will go through translate next.
Signed-off-by: Hanif Bin Ariffin <hanif.ariffin.4326@gmail.com>
Fix formatting...
Signed-off-by: Hanif Bin Ariffin <hanif.ariffin.4326@gmail.com>
Implemented translation feature
Signed-off-by: Hanif Bin Ariffin <hanif.ariffin.4326@gmail.com>
Setting the current directory in tests affects other tests, even if the
change is reverted after, because tests are run in parallel.
This should fix the flaky cp tests.
Since some tests run multiple commands, we have to re-calculate the
expected result for every run.
This is because the expected results depend on file timestamps, but the
files are re-created for every new TestScenario.
If options::WIDTH is not given, we should try to use the terminal width.
If that is unavailable, we should fall back to the 'COLUMNS' environment variable.
If that is unavailable (or invalid), we should fall back to a default of 80.
The following test case read stdin instead of file:
```
echo abcdefg > file
cargo run -- od --format x1 file
```
This is because the -t/--format argument was able to absorb multiple
arguments after it. This has now been fixed, and a test case is added
to ensure it will not happen again.
When hitting Ctrl+C sort now deletes any temporary files. To make this easier I
created a new struct `TmpDirWrapper` that handles the creation of new temporary
files and the registration of the signal handler.
The ToDo list was updated to mark `chcon` as done.
Building and testing `chcon` requires enabling the `feat_selinux` feature. If `make` is used for building, then please specify `SELINUX_ENABLED=1` if building and testing on a system where SELinux is not enabled.
Change the argument names in
* `AtPath::hard_link()`,
* `AtPath::symlink_file()`, and
* `AtPath::symlink_dir()`,
from `src` and `dest` to `original` and `link` to match the arguments of
the corresponding functions from the Rust standard library. For example,
see `std::os::unix::fs::symlink()`.
Since sort exits early due to the nonexistent file, it might no longer
be around when we try to send it the input.
This is "by design" and can be ignored.
Fix an issue where `basename` would print the empty path when provided
an input comprising only slashes (for example, "/" or "//" or
"///"). Now it correctly prints the path "/".
In such cases we have to create a temporary copy of the input file to prevent
overwriting the input with the output. This only affects merge sort, because it
is the only mode where we start writing to the output before having read all inputs.
Since lines that compare equal should be sorted together, we need to first
compare the lines (taking settings into account). Only if they do not compare
equal we should compare the hashes.
Add support for
* space-separated list of tab stops, like `--tabs="2 4 6"`,
* mixed comma- and space-separated lists, like `--tabs="2,4 6"`,
* the slash specifier in the last tab stop, like `--tabs=1,/5`,
* the plus specifier in the last tab stop, like `--tabs=1,+5`.
Change the behavior of `tac` when there are no line separators in the
input. Previously, it was appending an extra line separator; this commit
prevents that from happening. The input is now writted directly to
stdout.
Correct the error message produced by `tac` when trying to read from a
directory. Previously if the path 'a' referred to a directory, then
running `tac a` would produce the error message
dir: read error: Invalid argument
after this commit it produces
a: read error: Invalid argument
which matches GNU `tac`.
Handle additional edge cases arising from test(1)’s lack of reserved words.
For example, left parenthesis followed by an operator could indicate
either
* string comparison of a literal left parenthesis, e.g. `( = foo`
* parenthesized expression using an operator as a literal, e.g. `( = != foo )`
- Adds words to cspell exceptions
- Converts test macros to use Default trait.
- Converts parser to use Default trait.
- Adds Windows-friendly test files for block/unblock when nl is present
in test/spec file.
- Removes dd from feat_require_unix (keeps it in feat_common_core)
- Changes name of make_linux_iflags parameter from oflags to iflags.
- Removes roughed out SIGINFO impl.
- Renames plen -> target_len.
- Removes internal fn def for build_blocks and replaces with a call to
chunks from std.
- Renames apply_ct to the more descriptive apply_conversion
- Replaces manual swap steps in perform_swab with call to swap from std.
- Replaces manual solution with chunks where appropriate (in Read, and Write impl).
- Renames xfer -> transfer (in local variable names).
- Improves documentation for dd_filout/dd_stdout issue.
- Removes commented debug_assert statements.
- Modifies ProdUpdate to contain ReadStat and WriteStat rather than
copying their fields.
- Addresses verbose return in `Output<File>::new(...)`
- Resoves compiler warning when built as release when signal handler fails to
register.
- Derives _Default_ trait on ReadStat.
- Adds comments for truncated lines in block unblock tests
- Removes `as u8` in block unblock tests.
- Removes unecessary `#[inline]`
- Delegates multiplier string parsing to uucore::parse_size.
- Renames 'unfailed' -> 'succeeded' for clairity.
- Removes #dead_code warnings. No clippy warnings on my local machine.
- Reworks signal handler to better accomodate platform-specific signals.
- Removes explicit references to "if" and "of" in dd.rs.
- Removes explicit references to "bs", "ibs", "cbs" and "status" in
parseargs.rs.
- Removes `#[allow(deadcode)]` for OFlags, and IFlags.
- Removes spellchecker ignore from all dd files.
- Adds tests for 'traditional' and 'modern' CLI.
- Address Rust fmt issue
- ignores run-without-noatime test which fails on some build machines
- adds cspell:disable to all project files.
- adds .../dd/fixtures/cspell.json to ignore test fixtures.
- Adds cspell.json. Hopefully this will make you happy, spellchecker.
- Removes non-functional spellchecker-ignore tags
- Adds a sleep call to the no noatime test. Some systems were did not notice a changed
atime without the option present.
- Adds a test with a unicode filename.
- Addresses clippy lints and rustfmt issues.
- adds project header to multiple files
- updates spell check skip words
- removes linux only flags direct,noatime from mac_os build
- applies rustfmt to test_dd
- runs rustfmt on test_dd.rs
- eliminates compiler warnings
- adds many words to spellchecker ignore list
- adds sanity test for vexing conv=nocreat issue. Still WIP.
The `test_du_bytes` testcase for the `du --bytes` command is written to perform
a dynamic comparison on linux hosts, i.e. it compares the output of the command
to that of the hosts `du` from the GNU coreutils.
Previously the test was written such that it would *first* perform the dynamic
comparison, and *after that* continue to a static comparison which may fail for
specific hosts that have a filesystem different from what the test expects.
This patch excludes linux hosts from the static comparison to ensure the test
performs only the dynamic comparison.
This reimplements version_cmp, which is used in sort and ls to sort
according to versions.
However, it is not bug-for-bug identical with GNU's implementation.
I reported a bug with GNU here:
https://lists.gnu.org/archive/html/bug-coreutils/2021-06/msg00045.html
This implementation does not contain the bugs regarding the handling of
file extensions and null bytes.
`LANGUAGE=C` is not enough, `LC_ALL=C` is needed as the environment
variable that overrides all the other localization settings.
e.g.
```bash
$ LANGUAGE=C id foobar
id: ‘foobar’: no such user
$ LC_ALL=C id foobar
id: 'foobar': no such user
```
* replace `LANGUAGE` with `LC_ALL` as environment variable in the tests
* fix the the date string of affected uutils
* replace `‘` and `’` with `'`
Calling `cmd_keepenv("mv")` spawned the system `mv` instead of
the uutils `mv`. Also, `keepenv` isn't necessary because it doesn't
need to inherit environment variables.
We now actually check the stderr, because previously the result of
`ends_with` was not used, making the test pass even when it shouldn't.
I disabled the test on windows because `mkdir` does not support `-m` on
windows, making the test fail because there will be no permission error.
On FreeBSD there isn't a permission error either, and `mv` succeeds.
On some Windows machines this would otherwise cause `std::env::temp_dir`
to fall back to a path that is not writeable (C:\\Windows).
Since by default integration tests don't inherit env vars from the
parent, we have to override this in some cases.
When invoked via '[' name, last argument must be ']' or we bail out with
syntax error. Then the trailing ']' is simply disregarded and processing
happens like usual.
* --check=silent and --check=quiet, which are equivalent to -C.
* --check=diagnose-first, which is the same as --check
We also allow -c=<value>, which confuses GNU sort.
To prevent clap from parsing flags for the command to run as flags for
timeout, remove the "args" positional argument, but allow to pass flags
via the "command" positional arg.
As discussed here: https://github.com/uutils/coreutils/pull/2361
the group IDs returned for GNU's 'group' and GNU's 'id --groups'
starts with the effective group ID.
This implements a wrapper for `entris::get_groups()` which mimics
GNU's behaviour.
* add tests for `id`
* add tests for `groups`
* fix `id --groups --real` to no longer ignore `--real`
consistent
* add tests for each flag that takes NUM/SIZE arguments
* fix bug in tail where 'quiet' and 'verbose' flags did not override each other POSIX style
dst may or may not exist. In case it exists it might already be a symlink.
In neither case we should try to canonicalize dst, only its parent directory.
https://www.gnu.org/software/coreutils/manual/html_node/ln-invocation.html
> Relative symbolic links are generated based on their canonicalized
> **containing directory**, and canonicalized targets.
We were reporting "no match" when sorting something like "0 ". This is
because we don't distinguish between 0 and invalid lines when sorting.
For debug output we have to get this information back.
GNU seq does not support -t, but always outputs a newline at the end.
Therefore, our default for -t should be \n.
Also removes support for escape sequences (interpreting a literal "\n"
as a newline). This is not what GNU seq is doing, and unexpected.
If we notice that we can represent all arguments as BigInts, take a
different code path. Just like GNU seq this means we can print an
infinite amount of numbers in this case.
When a single directory is passed to ls in recursive mode, uutils ls
won't print the directory name
======================
GNU ls:
z:
======================
======================
uutils ls:
======================
This commit fixes this minor inconsistency and adds corresponding test.
Closes#2254. We should only inherit global settings for keys when there
are absolutely no options attached to the key.
The default key (matching the whole line) is implicitly added only if no
keys are supplied.
Improved some error messages by including more context.
* expr: support arbitrary precision integers
Instead of i64s we now use BigInts for integer operations. This means
that no result or input can be out of range.
The representation of integer flags was changed from i64 to u8 to make
their intention clearer.
* expr: allow big numbers as arguments as well
Also adds some tests
* expr: use num-traits to check bigints for 0 and 1
* expr: remove obsolete refs
match ergonomics made these avoidable.
* formatting
Co-authored-by: Sylvestre Ledru <sylvestre@debian.org>
* sort: crash when failing to open an input file
Instead of ignoring files we fail to open, crash.
The error message does not exactly match gnu, but that would require
more effort.
* use split_whitespace instead of a manual implementation
* fix expected error on windows
* sort: update expected error message
* sort: disable support for thousand separators
In order to be compatible with GNU, we have to disable thousands
separators. GNU does not enable them for the C locale, either.
Once we add support for locales we can add this feature back.
* sort: delete unused fixtures
* sort: compare -0 and 0 equal
I must have misunderstood this when implementing, but GNU considers
-0, 0, and invalid numbers to be equal.
* sort: strip blanks before applying the char index
* sort: don't crash when key start is after key end
* sort: add "no match" for months at the first non-whitespace char
We should put the "^ no match for key" indicator at the first
non-whitespace character of a field.
* sort: improve support for e notation
* sort: use maches! macros
Add some abstractions to simplify the `rbuf_but_last_n_lines()`
function, which implements the "take all but the last `n` lines"
functionality of the `head` program. This commit adds
- `RingBuffer`, a fixed-size ring buffer,
- `ZLines`, an iterator over zero-terminated "lines",
- `TakeAllBut`, an iterator over all but the last `n` elements of an
iterator.
These three together make the implementation of
`rbuf_but_last_n_lines()` concise.
Fails from time to time with
```
---- test_numfmt::test_should_calculate_implicit_padding_per_free_argument stdout ----
current_directory_resolved:
run: /target/x86_64-unknown-linux-musl/debug/coreutils numfmt --from=auto 1Ki 2K
thread 'test_numfmt::test_should_calculate_implicit_padding_per_free_argument' panicked at 'failed to write to stdin of child: Broken pipe (os error 32)', tests/common/util.rs:859:21
```
`sort` supports three ways to specify the sort mode: a long option
(e.g. --numeric-sort), a short option (e.g. -n) and the sort flag
(e.g. --sort=numeric).
This adds support for the sort flag.
Additionally, sort modes now conflict, which means that an error is
shown when multiple modes are passed, instead of silently picking a mode.
For consistency, I added the `random` sort mode to the `SortMode` enum,
instead of it being a bool flag.
Instead of overflowing when calculating the buffer size, use
saturating_{pow, mul}.
When failing to parse the buffer size, we now crash instead of silently
ignoring the error.
To make this work we make default sort a special case of external sort.
External sorting uses auxiliary files for intermediate chunks. However,
when we can keep our intermediate chunks in memory, we don't write them
to the file system at all. Only when we notice that we can't keep them
in memory they are written to the disk.
Additionally, we don't allocate buffers with the capacity of their
maximum size anymore. Instead, they start with a capacity of 8kb and are
grown only when needed.
This makes sorting smaller files about as fast as it was before
(I'm seeing a regression of ~3%), and allows us to seamlessly continue
with auxiliary files when needed.
A lot of tests depend on GNU's coreutils to be installed in order
to obtain reference values during testing.
In these cases testing is limited to `target_os = linux`.
This PR installs GNU's coreutils on "github actions" and adjusts the
tests for `who`, `stat` and `pinky` in order to be compatible with macOS.
* `brew install coreutils` (prefix is 'g', e.g. `gwho`, `gstat`, etc.
* switch paths for testing to something that's available on both OSs,
e.g. `/boot` -> `/bin`, etc.
* switch paths for testing to the macOS equivalent,
e.g. `/dev/pts/ptmx` -> `/dev/ptmx`, etc.
* exclude paths when no equivalent is available,
e.g. `/proc`, `/etc/fstab`, etc.
* refactor tests to make better use of the testing API
* fix a warning in utmpx.rs to print to stderr instead of stdout
* fix long_usage text in `who`
* fix minor output formatting in `stat`
* the `expected_result` function should be refactored
to reduce duplicate code
* more tests should be adjusted to not only run on `target_os = linux`
Fix a bug in which the incorrect character was being used to indicate
"round up to the nearest multiple" mode. The character was "*" but it
should be "%". This commit corrects that.
Change the error message for when the reference file (the `-r` argument)
is not found to match GNU coreutils. This commit also eliminates a
redundant call to `File::open`; the file need not be opened because the
size in bytes can be read from the result of `std::fs::metadata()`.
Change the interface provided by the `parse_size()` function to reduce
its responsibilities to just a single task: parsing a number of bytes
from a string of the form '123KB', etc. Previously, the function was
also responsible for deciding which mode truncate would operate in.
Furthermore, this commit simplifies the code for parsing the number and
unit to be less verbose and use less mutable state.
Finally, this commit adds some unit tests for the `parse_size()`
function.
Change the behavior of `head` to display an error for each problematic
file, instead of displaying an error message for the first problematic
file and terminating immediately at that point. This change now matches
the behavior of GNU `head`.
Before this commit, the first error caused the program to terminate
immediately:
$ head a b c
head: error: head: cannot open 'a' for reading: No such file or directory
After this commit:
$ head a b c
head: cannot open 'a' for reading: No such file or directory
head: cannot open 'b' for reading: No such file or directory
head: cannot open 'c' for reading: No such file or directory
Instead of using a BufReader and reading each line separately,
allocating a String for each one, we read to a chunk. Lines are
references to this chunk. This makes the allocator's job much easier
and yields performance improvements.
Chunks are read on a separate thread to further improve performance.
Fix a bug in which `head` failed to print headings for `stdin` inputs
when reading from multiple files, and fix another bug in which `head`
failed to print a blank line between the contents of a file and the
heading for the next file when reading multiple files. The output now
matches that of GNU `head`.
Fix two issues with the string formatting width for counts displayed
by `wc`.
First, the output was previously not using the default minimum width
(seven characters) when reading from `stdin`. This commit corrects
this behavior to match GNU `wc`. For example,
$ cat alice_in_wonderland.txt | wc
5 57 302
Second, if at least 10^7 bytes were read from `stdin` *after* reading
from a smaller regular file, then every output row would have width
8. This disagrees with GNU `wc`, in which only the `stdin` row and the
total row would have width 8. This commit corrects this behavior to
match GNU `wc`. For example,
$ printf "%.0s0" {1..10000000} | wc emptyfile.txt -
0 0 0 emptyfile.txt
0 1 10000000
0 1 10000000 total
Fixes#2186.
Change the error messages that get printed to `stderr` for compatibility
with GNU `wc` when an input is a directory and when an input does not
exist.
Fixes#2211.
This closes#2181.
`who --lookup` is failing with a runtime panic (double free).
Since `crate::dns-lookup` already includes a safe wrapper for `getaddrinfo`
I used this crate instead of further debugging the existing code in
utmpx::canon_host().
* It was neccessary to remove the version constraint for libc in uucore.
- add `==` as undocumented alias of `=`
- handle negated comparison of `=` as literal
- negation generally applies to only the first expression of a Boolean chain,
except when combining evaluation of two literal strings
When merging files we need to prioritize files that occur earlier in the
command line arguments with -m.
This also makes the extsort merge step (and thus extsort itself) stable again.
This is a refactor to reduce duplicate code, it affects chmod/ls/stat.
* merge `stat/src/fsext::pretty_access` into `uucore/src/lib/feature/fs::display_permissions_unix`
* move tests for `fs::display_permissions` from `test_stat::test_access` to `uucore/src/lib/features/fs::test_display_permissions`
* adjust `uu_chmod`, `uu_ls` and `uu_stat` to use `uucore::fs::display_permissions`
* ls: Implement total size feature
- Implement total size reporting that was missing
- Fix minor formatting / readability nits
* tests: Add tests for ls total sizes feature
* ls: Fix MSRV build errors due to unsupported attributes for if blocks
* ls: Add windows support for total sizes feature
- Add windows support (defaults to file size as block sizes related
infromation is not avialable on windows)
- Renamed some functions
- Replace the parser with a recursive descent implementation that handles
parentheses and produces a stack of operations in postfix order.
Parsing now operates directly on OsStrings passed by the uucore framework.
- Replace the dispatch mechanism with a stack machine operating on the
symbol stack produced by the parser.
- Add tests for parenthesized expressions.
- Begin testing character encoding handling.
Moved argument parsing to clap and added tests to cover using "-" as
stdin, passing in too many file arguments, and updated the "wrap" error
message in the tests.
Add crossterm as dependency
Complete the paging portion
Fixed tests
cp: extract linux COW logic into function
cp: add --reflink support for macOS
Fixes#1773
Fix error in Cargo.lock
Quit automatically if not much output is left
Remove unnecessary redox and windows specific code
Handle line wrapping
Put everything according to uutils coding standards
Add support for multiple files
Fix failing test
Use the args argument to get cli arguments
Fix bug where text is repeated multiple times during printing
Add a little prompt
Add a top file prompt for multiple files
Change println in loops to stdout.write and setup terminal only once
Fix bug where all lines were printed in a single row
Remove useless file and fix failing test
Fix another test
* ls: added creation time
* ls: Added most time features
Missing support for posix-,Format+, translating via locales. Also required more tests
* ls: rustfmt
* ls: Additional changes and fixes
Fixed the argument order, fixed a wrong iso format.
* ls: additional tests for styles
* ls: perfected arg parsing on time styles
* fix birthime test
* ls: Use 'stdout_str' in new tests
* ls: Disabled birthtime test for windows
* ls: removed indoc as a dependency
* ls: birthime test, sync first created file
* ls: birthime test, add comment explaining sync
* Removed ruby testfile birth_test.rb
This accidentally got commited in a merge
Note, I needed to change the error messages in one of the tests because
getopt and clap have different error messages when not providing a
default value
* Change unchecked unwrapping to unwrap_or_default for argument parsing (resolving #1845)
* Added unit-testing for the collect_str function on invalid utf8 OsStrs
* Added a warning-message for identification purpose to the collect_str method.
* - Add removal of wrongly encoded empty strings to basename
- Add testing of broken encoding to basename
- Changed UCommand to use collect_str in args method to allow for integration testing of that method
- Change UCommand to use unwarp_or_default in arg method to match the behaviour of collect_str
* Trying out a new pattern for convert_str for getting a feeling of how the API feels with more control
* Adding convenience API for compact calls
* Add new API to everywhere, fix test for basename
* Added unit-testing for the conversion options
* Added unit-testing for the conversion options for windows
* fixed compilation and some merge hiccups
* Remove windows tests in order to make merge request build
* Fix formatting to match rustfmt for the merged file
* Improve documentation of the collect_str method and the unit-tests
* Fix compilation problems with test
Co-authored-by: Christopher Regali <chris.vdop@gmail.com>
Co-authored-by: Sylvestre Ledru <sylvestre@debian.org>
* ls: ignore leading period when sorting by name
ls now behaves like GNU ls with respect to sorting files by ignoring
leading periods when sorting by main.
Added tests to ensure "touch a .a b .b ; ls" returns ".a a .b b"
* Replaced clone/collect calls.
- All assert_eq in tests/common/util.rs now print a pretty diff on test
failures.
- {stdout, stderr}_is_fixture now compare the expected output and the
fixture as Strings, which leads to more usable diffs.
This adds a --debug flag, which, when activated, will draw lines below
the characters that are actually used for comparisons.
This is not a complete implementation of --debug. It should, quoting the man page
for GNU sort: "annotate the part of the line used to sort, and warn
about questionable usage to stderr". Warning about "questionable usage"
is not part of this patch.
This change required some adjustments to be able to get the range that
is actually used for comparisons. Most notably, general numeric comparisons
were rewritten, fixing some bugs along the lines.
Testing is mostly done by adding fixtures for the expected debug output of
existing tests.
* ls: Remove allocations by eliminating collect/clones
* ls: Introduce PathData structure
- PathData will hold Path related metadata / strings that are required
frequently in subsequent functions
- All data is precomputed and cached and subsequent functions just
use cached data
* ls: Cache more data related to paths
- Cache filename and sort by filename instead of full path
- Cache uid->usr and gid->grp mappings
https://github.com/uutils/coreutils/pull/2099/files
* ls: Add BENCHMARKING.md
* ls: Document PathData structure
* tests/ls: Add testcase for error paths with width option
* ls: Fix unused import warning
cached will be only used for unix currently as current use of
caching gid/uid mappings is only relevant on unix
* ls: Suggest checking syscall count in BENCHMARKING.md
* ls: Remove mentions of sort in BENCHMARKING.md
* ls: Remove dependency on cached
Implement caching using HashMap and lazy_static
* ls: Fix MSRV error related to map_or
Rust 1.40 did not support map_or for result types
GNU sort disallows these combinations, presumably because they are
likely not what the user really wants.
Ignoring characters would cause things to be put together that aren't
together in the input. For example, -dn would cause "0.12" or "0,12" to
be parsed as "12" which is highly unexpected and confusing.
- Passing `never` to `--reflink` does not raise an error anymore.
- Remove `Options::reflink` flag as it was redundant with
`reflink_mode`.
- Add basic tests for this option. Does not check that a copy-on-write
rather than a regular copy was made.
* sort: implement numeric string comparison
This implements -n and -h using a string comparison algorithm instead
of parsing each number to a f64 and comparing those.
This should result in a moderate performance increase and eliminate loss
of precision.
* cache parsed f64 numbers
For general numeric comparisons we have to parse numbers as f64,
as this behavior is explicitly documented by GNU coreutils.
We can however cache the parsed value to speed up comparisons.
* fix leading zeroes for negative numbers
* use more appropriate name for exponent
* improvements to the parse function
* move checks into main loop and fix thousands separator condition
* remove unneeded checks
* rustfmt
* du error output should match GNU
* Created a new error macro which allows the customization of the
"error:" string part
* Match the du output based on the type of error encountered. Can extend
to handling other errors I guess.
* Rustfmt updates
* Added non-windows test for du no permission output
* Various fixes and performance improvements
* fix a typo
Co-authored-by: Michael Debertol <michael.debertol@gmail.com>
* Fix month parse for months with leading whitespace
* Implement test for months whitespace fix
* Confirm human numeric works as expected with whitespace with a test
* Correct arg help value name for --parallel
* Fix SemVer non version lines/empty line sorting with a test
Co-authored-by: Sylvestre Ledru <sledru@mozilla.com>
Co-authored-by: Michael Debertol <michael.debertol@gmail.com>
* cat: Unrevert splice patch
* cat: Add fifo test
* cat: Add tests for error cases
* cat: Add tests for character devices
* wc: Make sure we handle short splice writes
* cat: Fix tests for 1.40.0 compiler
* cat: Run rustfmt on test_cat.rs
* Run 'cargo +1.40.0 update'
* sort: implement basic -k and -t support
This allows to specify keys after the -k flag and a custom field
separator using -t.
Support for options for specific keys is still missing, and the -b flag
is not passed down correctly.
* sort: implement support for key options
* remove unstable feature use
* don't pipe in input when we expect a failure
* only tokenize when needed, remove a clone()
* improve comments
* fix clippy lints
* re-add test
* buffer writes to stdout
* fix ignore_non_printing
and make the test fail in case it is broken :)
* move attribute to the right position
* add more tests
* add my name to the copyright section
* disallow dead code
* move a comment
* re-add a loc
* use smallvec for a perf improvement in the common case
* add BENCHMARKING.md
* add ignore_case to benchmarks
* Various fixes and performance improvements
* fix a typo
Co-authored-by: Michael Debertol <michael.debertol@gmail.com>
Co-authored-by: Sylvestre Ledru <sledru@mozilla.com>
Co-authored-by: Michael Debertol <michael.debertol@gmail.com>
* if we want to test an irregular scenario, ignoring errors caused by
writing to stdin of the command can be uselful.
* for example, when writing some text to stdin of cksum in a scenario
where it doesn't consume this input, the child process might have
exited before the text was written. therefore, this test sometimes
fails with a 'Broken pipe'.
* Replace outdated time 0.1 dependancy with latest version of chrono
I also noticed that times are being miscalculated on linux, so I fixed that.
* Add time test for issue #2042
* Cleanup use declarations
* Tie time test to `touch` feature
- if we compile with the right OS feature flag then we should have it,
even on Windows
refactor `is_wsl` to `is_wsl_1` and `is_wsl_2`
On my tests with wsl2 ubuntu2004 all tests pass without special cases
I moved wsl detection into uucore so that it can be shared instead of duplicated
I moved `parse_mode` into uucode as it seemed to fit there better and anyway requires libc feature
Treat tab chars as advancing to the next tab stop rather than having a fixed
8-column width.
Also treat tab as a whitespace split target only when splitting on word
boundaries.
- Adds support for calling dd fn from cl
- Adds basic cl tests from project root
- Adds support for multiplier strings (c, w, b, kB, KB, KiB, ... EB, E,
EiB.
The '-d' flag should create all ancestors (or components) of a
directory regardless of the presence of the "-D" flag.
From the man page:
-d, --directory
treat all arguments as directory names; create all components of the specified directories
With GNU:
$ install -v -d dir1/di2
install: creating directory 'dir1'
install: creating directory 'dir1/di2'
With this version:
$ ./target/release/install -v -d dir3/di4
install: dir3/di4: No such file or directory (os error 2)
install: dir3/di4: chmod failed with error No such file or directory (os error 2)
install: created directory 'dir3/di4'
Also, one of the unit tests misinterprets what a "component" is,
and hence was fixed.
* Issue #1622 port `du` to windows
* Attempt to support Rust 1.32
Old version was getting "attributes are not yet allowed on `if`
expressions" on Rust 1.32
* Less #[cfg]
* Less duplicate code.
I need the return and the semicolon after if otherwise the second #[cfg]
leads to unexpected token complilation error
* More accurate size on disk calculations for windows
* Expect the same output on windows as with WSL
* Better matches output from du on WSL
* In the absence of feedback I'm disabling these tests on Windows.
They require `ln`. Windows does not ship with this utility.
* Use the coreutils version of `ln` to test `du`
`fn ccmd` is courtesy of @Artoria2e5
* Look up inodes (file ids) on Windows
* One more #[cfg(windows)] to prevent unreachable statement warning on linux
* wc: Don't read() if we only need to count number of bytes
* Resolve a few code review comments
* Use write macros instead of print
* Fix wc tests in case only one thing is printed
* wc: Fix style
* wc: Use return value of first splice rather than second
* wc: Make main loop more readable
* wc: Don't unwrap on failed write to stdout
* wc: Increment error count when stats fail to print
* Re-add Cargo.lock
* Implemented --indicator-style flag on ls.
* Rust fmt
* Grouped indicator_style args.
* Added tests for sockets and pipes.
Needed to modify util.rs to add support for pipes (aka FIFOs).
* Updated util.rs to remove FIFO operations on Windows
* Fixed slight error in specifying (not(windows))
* Fixed style violations and added indicator_style test for non-unix systems
+ aligned 'tee' output with GNU tee when one of the files is '/dev/full'
+ don't stop tee when one of the outputs fails; just continue and return
error status from tee in the end
Co-authored-by: Ivan Rymarchyk <irymarchyk@arlo.com>
* mkfifo: general refactor, move to clap, add unimplemented flags
* chore: update Cargo.lock
* chore: delete unused variables, simplify multiple lines with crash
* test: add tests
* chore: revert the use of crash
* test: use even more invalid mod mode
* install: implement `-C` / `--compare`
GNU coreutils [1] checks the following: whether
- either file is nonexistent,
- there's a sticky bit or set[ug]id bit in play,
- either file isn't a regular file,
- the sizes of both files mismatch,
- the destination file's owner differs from intended, or
- the contents of both files mismatch.
[1] https://git.savannah.gnu.org/cgit/coreutils.git/tree/src/install.c?h=v8.32#n174
* Add test: non-regular files
* Forgot a #[test]
* Give up on non-regular file test
* `cargo fmt` install.rs
* date: change tests to expect failure
Although these tests contain valid dates, the parsing logic is not
implemented yet. It should be changed to expect success when
the parsing logic is done.
* date: fix test build errors
* feat: move unexpand to clap
* chore: allow muliple files
* test: add test fixture, test reading from a file
* test: fix typo on file name, add test for multiple inputs
* chore: use 'success()' instead of asserting
* chore: delete unused variables
* chore: use help instead of long_help, break long line
* fix: use settings to allow leading hyphen and trailing var arg
fixes: https://github.com/uutils/coreutils/issues/1873
* test: add test cases
* test: add more test cases with different order in hyphen values
* chore: add comment to explain why we need TrailingVarArg
- changed some error return codes to match GNU implementation
- changed warning/error messages to match GNU nohup
- replaced getopts dependency with clap
- added a test
This PR adds the options to customize what information is shown in long format regarding author, group & owner. Specifically it adds:
- `--author`: shows the author, which is always the same as the owner. GNU has this feature because GNU/Hurd supports a difference between author and owner, but I don't think Rust supports GNU/Hurd, so I just used the owner.
- `-G` & `--no-group`: hide the group information.
- `-o`: hide the group and use long format (equivalent to `-lG`).
- `-g`: hide the owner and use long format.
The `-o` and `-g` options have some interesting behaviour that I had to account for. Some examples:
- `-og` hides both group and owner.
- `-ol` still hides the group. Same behaviour with variations such as `-o --format=long`, `-gl`, `-g --format=long` and `-ogl`.
- They even retain some information when overridden by another format: `-oCl` (or `-o --format=vertical --format=long`) still hides the group.
My previous solution for handling the behaviour where `-l1` shows the long format did not fit with these additions, so I had to rewrite that as well.
The tests only cover the how many names (author, group and owner) are present in the output, so it can't distinguish between, for example, author & group and group & owner.
* date: implement set date for unix and windows
Parsing the date string is not fully implemented yet, as in it relies
on the internals of chrono - things like "Mon, 14 Aug 2006 02:34:56 -0600"
do not work, nor does "2006-08-14 02:34:56" (no TZ / local time). This
is no different to using the "--date" option however, and will get fixed
when `parse_date` is a bit smarter.
Only supports unix and Windows platforms for now.
Current implementation of the skip fields logic does not handle
multibyte code points correctly. It assumes each code point (`char`) is
one byte. If the skipped part of the input line has any multibyte code
points then this can cause fields not being skipped correctly (field
start index is calculated to be before it actually starts).
* touch: use arggroup for sources
* tests/touch: add tests for multiple sources
* touch: turn macros into functions
* test/touch: fmt
* touch: constant for the sources ArgGroup
- added `-` as the default input, since `paste` reads stdin if no file
is provided
- `paste` also supports providing `-` multiple times
- added a test for it
* muted test not for windows and added windows temp file convention
* Update mktemp.rs
Revert windows mktmp template difference
Co-authored-by: Chad Brewbaker <chad@flyingdogsolutions.com>
When converting to SI or IEC, produce values that align with the conventions
used by GNU numfmt.
- values > 10 are represented without a decimal place, so 10000 becomes 10K
instead of 10.0K
- when truncating, take the ceiling of the value, so 100001 becomes 101K
- values < 10 are truncated to the highest tenth, so 1001 becomes 1.1K
closes#1726
Adjust header option handling to prohibit passing a value of 0 to align
with GNU numfmt. Also report header option parse errors as GNU does.
closes#1708
Align with GNU numfmt by trimming leading whitespace from supplied values.
If the user did not specify a padding, calculate an implied padding from
the leading whitespace and the value.
Also track closer to GNU numfmt’s error message format.
This little check, allows us to hide the files that
shouldn't be shown on the listing on Windows operating
systems.
Just like the "dot" in UNIX based operating systems
Windows uses its own file attributes to determine if a file
is hidden or not.
The lack of support for this option is normally an annoyance
for many users, this commit adds full support for this feature
