Summary of changes in tests/util:
* Introduce global constant TESTS_BINARY holding the path to `coreutils`
* Implement running an arbitrary command in a sh or cmd shell per default.
* Implement std::fmt::Display for UCommand.
* Change usages of UCommand::util_name from &Option<OsStr> to Option<OsStr>
* `UCommand::new_from_tmp`: Use OsStr directly instead of TempDir -> String -> OsStr
* Collect arguments in `UCommand::args` field
* Build environment variables in `UCommand` itself instead of `std::process::Command`
* Move building of std::process:Command from fields in UCommand to own method `build`
* Remove assertions of UCommand::has_run in arg, args and env.
Summary of changes in tests/by-util:
* Remove usages of UCommand::raw. Fix tests to use UCommand::to_string.
* test_test: Adjust test_invalid_utf8_integer_compare to use `UCommand::args`
* test_chmod: run_single_test
* test_pwd: symlinked_env
* test_cp:
Fix the usage of &Option<OsStr> in `test_src_base_dot`
Refactor test_src_base_dot to not use UCommand::new but ts.ucmd() instead
* tests/util: Fix documentation of UCommand::stderr_only and usage_error
* tests/util: Remove trimming from CmdResult::stdout_matches and stdout_does_not_match. Fix tests.
The tests are fixed to match the trailing newline instead of ignoring it.
* cp: require preserve only certain attributes
# Conflicts:
# src/uu/cp/src/copydir.rs
# src/uu/cp/src/cp.rs
* tests/cp: preserve all and preserve xattr tests with todos
* tests/cp: rename preserve tests
* tests/cp: add android fail test for preserve=xattr
On Android, this cp with explicit preserve of xattr must fail, because of the limitations of the filesystem setup used on Android.
* cp: verify some metadata in cp preserve tests
# Conflicts:
# tests/by-util/test_cp.rs
* cp: run test_cp_preserve_all in all OS's but only check metadata on linux
* test/cp: don't expect the mode to change in explicit cp preserve
* cp: attributes struct instead of enum for unified required tracking
* cp: refactor preserver and handle_preserve
# Conflicts:
# src/uu/cp/src/cp.rs
* cp: update preserve attr to max
* test/cp: fix the preserve xattr test
Access timestamps appear to be modified only in this test. Running the command directly does not alter the access timestamp.
* cp/test: preserve all and context case
* cp: fix preserve args value source
* test/cp: don't check mtime on freebsd
* test/cp: don't check mtime on macos
* test/cp: fix freebsd deps
* test/cp: support freebsd tests
* cp: simplify try_set_from_string
* cp: parse context attr in preserve in any case to show warning later
* cp: print warnings for attribute errors if not required
* cp: show SELinux warning only once
* cp: show SELinux warning without error
* Revert "cp: show SELinux warning without error"
This reverts commit d130cf0d8c8e28ac2c903413992613241decf879.
* cp: add documentation for preserve components
* cp: simplify try_set_from_string
* cp: EN_US "behavior" spelling for cspell
This introduces an argument preprocessing step for the nice tool in
order to support the legacy nice adjustment syntax (`nice -1 foo`,
`nice --1 foo`, `nice -+1 foo` and so forth).
This is a preprocessing step because the interaction between these
arguments and the existing `nice -n -1` syntax becomes context
dependent, in a way that is currently difficult to capture through
clap.
Signed-off-by: Ed Smith <ed.smith@collabora.com>
Other changes summmary:
* Use module level imports instead of fully qualified imports where appropriate
* Fix intermittent failing test_positive_zero_bytes because of broken pipe
* Cleanup old disabled test for warning of following indefinitely
* install: fix installing one file when using -Dt options
* install: fix installing multiple files with -Dt
Code was missing the logic to create the target dir when multiple files
should be copied and target dir is given by -t option.
This simplifies the copy logic also when only one file should be copied
to the target dir.
* install: fix verbose output when using -D
Also adds a unit test to verify the same behaviour as the gnu tools.
* install: add more testcases for create leading dir
Tests various combinations of "-D" with and w/o "-t" when installing
either a single file or multiple files into a non existing directory.
install -D file1 file2 (-t) not_existing_dir
install -D file1 (-t) not_existing_dir/
Also fixes file formatting, spelling and adds some more test asserts.
* install: fix error for nonex. dir path ending on /
The install command failed with a different error message than the
original GNU install tool. Checking for a trailing slash fixes this.
Only works on unix though.
* install: add windows support when checking for '/'
* install.rs: fix spelling
* install.rs: add more tests regarding omitting dir
This increases the CI test coverage and also checks for more corner
cases to ensure uu_install is compliant with GNU's original.
export C=coreutils/target/debug/
rm -rf dir1 no-dir2 dir3 file
mkdir -p dir1 dir3
touch file
${C}install dir1/file1 dir1/.. no-dir2
${C}install dir1/file1 dir1/.. dir3
${C}install dir1/.. dir3
* install: improve test_install_missing_arguments
Also check that install returns the correct error messages, when only a
target directory is given via -t and that is is not created (-D option).
* install: rework the checks for missing file args
This ensures correct (GNU install like) behavior. Tests from the last
commit will pass now.
This commit corrects the behavior of `cp -r --parents --verbose` when
the source path is a directory, so that it prints the copied ancestor
directories. For example,
$ mkdir -p a/b/c d
$ cp -r --verbose --parents a/b/c d
a -> d/a
a/b -> d/a/b
'a/b/c' -> 'd/a/b/c'
This commit corrects the behavior of `cp --parents --verbose` when the
source path is a file so that it prints the copied ancestor
directories. For example,
$ mkdir -p a/b d
$ touch a/b/c
$ cp --verbose --parents a/b/c d
a -> d/a
a/b -> d/a/b
'a/b/c' -> 'd/a/b/c'
Fixes#3332.
Adjust the rendering of the concise byte counts in both SI and IEC
units to better match the behavior of GNU dd.
Before this commit,
$ head -c 1024 /dev/zero | dd > /dev/null
2+0 records in
2+0 records out
1024 bytes (1 KB, 1024 B) copied, 0.0 s, 1.0 MB/s
After this commit,
$ head -c 1024 /dev/zero | dd > /dev/null
2+0 records in
2+0 records out
1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0 s, 1.0 MB/s
For comparison, GNU dd produces the following:
$ head -c 1024 /dev/zero | dd > /dev/null
2+0 records in
2+0 records out
1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000332864 s, 3.1 MB/s
Prevent a panic in `cp -a` when the target of a hard link already
exists in the target directory structure.
For example,
$ mkdir -p src dest/src
$ touch src/f dest/src/f
$ ln src/f src/link
$ cp -a src dest
The `cp` command now succeeds without error.
Summary:
* Disable test_retry6 on android because of intermittent failures
* Use wait() instead of wait_with_output in test_cat, test_cp, test_sort
* tests/sort: Simplify usage of test_sigpipe_panic
* Fix tests in test_tee.
tests/tac:
There was a change in the `tests/common/util.rs` test api concerning piped input which may have
revealed a bug in the implementation of tac. Please see also
https://github.com/uutils/coreutils/pull/4136
A short summary of changes:
* Add some basic tests for UChild and the run methods.
* Try more often in a fixed interval to create the tempfile for CapturedOutput.
* Fix drop order of struct fields for better cleanup of temporary files/dirs.
* Mark UChild::wait_with_output and UChild::pipe_in_and_wait_with_output deprecated
* Make CapturedOutput private
* Panic in stdout_all, stdout_all_bytes etc. if output is not captured.
* Rename some methods, refactor, clean up, fix documentation, add try_... methods
Before this commit, if `sparsefile` were a regular file of non-zero
size whose contents are all null bytes, then
dd if=sparsefile of=outfile conv=notrunc
would have resulted in `outfile` having zero size as reported by
`stat`. After this commit, `outfile` will have the same size as
`sparsefile` (even if the contents are represented sparsely by the
filesystem).
Move some tests that simulate a slow reader from `dd.rs` to
`tests/by-util/test_dd.rs`, and employ a FIFO and `sleep()` to
simulate the slow reader instead of a custom struct that implements
`Read`. This change restricts the type of `Input`s the
`Output::dd_out()` function can accept, facilitating a future change
to make `Input` an enum.
Allow uppercase "B" on its own as a unit specifier for the `count`,
`seek`, and `skip` arguments to `dd`.
For example,
$ printf "abcdef" | dd count=3B status=none
abc
Correct the behavior of `cp --reflink=never --sparse=always` so that
it performs a sparse copy. Before this commit, it was incorrectly
performing a dense copy.
Implement the `--copy-contents` option when the source is a FIFO, so
that the contents of the FIFO are copied (when the bytes become
available for reading) instead of the FIFO object itself. For example,
$ mkfifo fifo
$ cp --copy-contents fifo outfile &
[1] 1614080
$ echo foo > fifo
$ cat outfile
foo
[1]+ Done cp --copy-contents fifo outfile
Fix the behavior of `cp` when both `--backup` and `--force` are
specified and the source and destination are the same file. Before
this commit, `cp` terminated without copying and without making a
backup. After this commit, the copy is made and the backup file is
made. For example,
$ touch f
$ cp --force --backup f f
results in a backup file `f~` being created.
If the arg starts with an id numeric value, the group isn't set but the separator is provided,
we should fail with an error
Should fix tests/chown/separator.sh
Change `cp` to terminate with an error when attempting to copy through
a dangling symbolic link with the `--force` option. Before this
commit,
touch src
ln -s no-such-file dest
cp -f src dest
would incorrectly replace `dest` with the contents of `src`. After
this commit, it correctly fails with the error message
cp: not writing through dangling symlink 'dest'
Fix a bug where `cp` failed to copy ancestor directories when using
the `--parents` option. For example, before this commit:
$ mkdir -p a/b/c d
$ cp --parents a/b/c d
$ find d
d
d/c
After this commit
$ mkdir -p a/b/c d
$ cp --parents a/b/c d
$ find d
d
d/a
d/a/b
d/a/b/c
This commit also adds the correct messages for `--verbose` mode:
$ cp -r --parents --verbose a/b/c d
a -> d/a
a/b -> d/a/b
'a/b/c' -> 'd/a/b/c'
Fixes#3332.
Implement distributing lines of a file in a round-robin manner to a
specified number of chunks. For example,
$ (seq 1 10 | split -n r/3) && head -v xa[abc]
==> xaa <==
1
4
7
10
==> xab <==
2
5
8
==> xac <==
3
6
9
* hashsum: test b3sum::test_nonames for real
Signed-off-by: Huijeong Kim <herehuijeong@gmail.com>
* apply cargo format
Signed-off-by: Huijeong Kim <herehuijeong@gmail.com>
Before, the sort could work faster and we could be late with
the signal.
Now we create a new big file, `sort` can't process it in a minute,
so we can safely wait for the temporary directory to be created
and send a signal afterwards
Before the change it slept for 0.1 seconds and right after that
asserted if `sort` has created the directory. Sometimes `sort`
didn't manage to create the directory in 0.1 seconds.
So the change is it tries to wait for `timeout` starting with
0.1 seconds, and if directory was not found, it tries 4 more times,
each time increasing timeout twice. Once the directory is found
it breaks.
Make cp preserve the permissions of a directory when copying
it. Before this commit,
cp -pR src/ dest/
failed to copy the permissions of `src/` to `dest/`. After this
commit, the permissions are correctly copied.
While the rust coreutils semantics were arguably more correct,
they were different than the gnu split semantics when handling a
file without a trailing EOF. This patch addresses that difference
and allows passing one more GNU test suite.
Correct the error message produced when attempting to copy a directory
into itself with `cp`. Before this commit, the error message was
$ cp -R d d
cp: cannot copy a directory, 'd', into itself, 'd'
After this commit, the error message is
$ cp -R d d
cp: cannot copy a directory, 'd', into itself, 'd/d'
Allow `cp --remove-destination` to remove a symbolic link loop (or a
symbolic link that initiates a chain of too many symbolic
links). Before this commit, if `loop` were a symbolic link to itself,
then
cp --remove-destination file loop
would fail with an error message. After this commit, it succeeds. This
matches the behaviotr of GNU cp.
Add special handling in `mktemp` for when the directory that will
contain the temporary file is not found. This situation now produces
the message
mktemp: failed to create file via template 'XXX': No such file or directory
to match the behavior of GNU mktemp.
Update the usage message when too many template arguments are given on
the command line to match that of GNU mktemp:
mktemp: too many templates
Try 'mktemp --help' for more information.
This fixes the test case `too-many` in the GNU test suite file
`tests/misc/mktemp.pl`.
`cp` in interactive mode used to write to stdout asking for
overwrite. GNU version writes to stderr.
Changed: write to stderr to make compatible with GNU.
Change `mktemp` so that it respects the value of the `TMPDIR`
environment variable if no directory is otherwise specified in its
arguments. For example, before this commit
$ TMPDIR=. mktemp
/tmp/tmp.WDJ66MaS1T
After this commit,
$ TMPDIR=. mktemp
./tmp.h96VZBhv8P
This matches the behavior of GNU `mktemp`.
It seems that `chrono` is the reason of deadlock or UB in android
CI. Also `chrono` had some security issues and wasn't maintained for
two years until March 2022, so other unstabilities can happen. Plus
`chrono` uses old `time` dependency.
Previously, if stdin redirect pointed to a regular file,
tailing started at the beginning of the file. However,
tailing needs to start at the current position because this
is expected by tests/tail-2/start-middle.sh.
This fixes the issue by taking the current offset into account
while going backwards through the stdin redirected file.
This fixes a bug where calling `tail - < file.txt` would result
in invoking `unbounded_tail()`.
However, it is a stdin redirect to a seekable regular file and
therefore `bounded_tail` should be invoked as if `tail file.txt` had
been called.
* Fix a bug in split where chunking would be skipped when the chunk size
happened to be an exact divisor of the buffer size used to read the
input stream.
The issue here was that file was being split byte-wise in chunks of 1G.
The input stream was being read in chunks of 8KB, which evenly divides
the chunk size. Because the check to allocate the next output chunk was
done at the bottom of the loop previously, it would never occur because
the current input chunk was fully consumed at that point. By moving the
check to the top of the loop (but still late enough that we know we have
bytes to write) we resolve this issue.
This scenario is unfortunately hard to write a test for, since we don't
explicitly control the input chunk size.
Fixes https://github.com/uutils/coreutils/issues/3790
* test/cp: -p changes ctime and added sleep for better timestamp testing
* test/cp: add nanoseconds checking for copied timestamps
* test/cp: made compilable on other OSes
* test/cp: added error messages to assert_eq calls
* cp: Refactor `reflink`/`sparse` handling to enable `--sparse` flag
`--sparse` and `--reflink` options have a lot of similarities:
- They have similar options (`always`, `never`, `auto`)
- Both need OS specific handling
- They can be mutually exclusive
Prior to this change, `sparse` was defined as `CopyMode`, but `reflink`
wasn't. Given the similarities, it makes sense to handle them similarly.
The idea behind this change is to move all OS specific file copy
handling in the `copy_on_write_*` functions. Those function then
dispatch to the correct logic depending on the arguments (at the moment,
the tuple `(reflink, sparse)`).
Also, move the handling of `--reflink=never` from `copy_file` to the
`copy_on_write_*` functions, at the cost of a bit of code duplication,
to allow `copy_on_write_*` to handle all cases (and later handle
`--reflink=never` with `--sparse`).
* cp: Implement `--sparse` flag
This begins to address #3362
At the moment, only the `--sparse=always` logic matches the requirement
form GNU cp info page, i.e. always make holes in destination when
possible.
Sparse copy is done by copying the source to the destination block by
block (blocks being of the destination's fs block size). If the block
only holds NUL bytes, we don't write to the destination.
About `--sparse=auto`: according to GNU cp info page, the destination
file will be made sparse if the source file is sparse as well. The next
step are likely to use `lseek` with `SEEK_HOLE` detect if the source
file has holes. Currently, this has the same behaviour as
`--sparse=never`. This `SEEK_HOLE` logic can also be applied to
`--sparse=always` to improve performance when copying sparse files.
About `--sparse=never`: from my understanding, it is not guaranteed that
Rust's `fs::copy` will always produce a file with no holes, as
["platform-specific behavior may change in the
future"](https://doc.rust-lang.org/std/fs/fn.copy.html#platform-specific-behavior)
About other platforms:
- `macos`: The solution may be to use `fcntl` command `F_PUNCHHOLE`.
- `windows`: I only see `FSCTL_SET_SPARSE`.
This should pass the following GNU tests:
- `tests/cp/sparse.sh`
- `tests/cp/sparse-2.sh`
- `tests/cp/sparse-extents.sh`
- `tests/cp/sparse-extents-2.sh`
`sparse-perf.sh` needs `--sparse=auto`, and in particular a way to skip
holes in the source file.
Co-authored-by: Sylvestre Ledru <sylvestre@debian.org>
* ls: Implement --zero flag. (#2929)
This flag can be used to provide a easy machine parseable output from
ls, as discussed in the GNU bug report
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=49716.
There are some peculiarities with this flag:
- Current implementation of GNU ls of the `--zero` flag implies some
other flags. Those can be overridden by setting those flags after
`--zero` in the command line.
- This flag is not compatible with `--dired`. This patch is not 100%
compliant with GNU ls: GNU ls `--zero` will fail if `--dired` and
`-l` are set, while with this patch only `--dired` is needed for the
command to fail.
We also add `--dired` flag to the parser, with no additional behaviour
change.
Testing done:
```
$ bash util/build-gnu.sh
[...]
$ bash util/run-gnu-test.sh tests/ls/zero-option.sh
[...]
PASS: tests/ls/zero-option.sh
============================================================================
Testsuite summary for GNU coreutils 9.1.36-8ec11
============================================================================
# TOTAL: 1
# PASS: 1
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
============================================================================
```
* Use the US way to spell Behavior
* Fix formatting with cargo fmt -- tests/by-util/test_ls.rs
* Simplify --zero flag overriding logic by using index_of
Also, allow multiple --zero flags, as this is possible with GNU ls
command. Only the last one is taken into account.
Co-authored-by: Sylvestre Ledru <sledru@mozilla.com>
* wc: specialize scanning loop on settings.
The primary computational loop in wc (iterating over all the
characters and computing word lengths, etc) is configured by a
number of boolean options that control the text-scanning behavior.
If we monomorphize the code loop for each possible combination of
scanning configurations, the rustc is able to generate better code
for each instantiation, at the least by removing the conditional
checks on each iteration, and possibly by allowing things like
vectorization.
On my computer (aarch64/macos), I am seeing at least a 5% performance
improvement in release builds on all wc flag configurations
(other than those that were already specialized) against
odyssey1024.txt, with wc -l showing the greatest improvement at 15%.
* Reduce the size of the wc dispatch table by half.
By extracting the handling of hand-written fast-paths to the
same dispatch as the automatic specializations, we can avoid
needing to pass `show_bytes` as a const generic to
`word_count_from_reader_specialized`. Eliminating this parameter
halves the number of arms in the dispatch.
