The previous encoding handling was unnecessarily complex. This commit removes the enum that specifies the handling and instead has two separate methods to collect the strings either with lossy conversion or by ignoring invalidly encoded strings.
Outside of tests, only `accept_any` was used, meaning that this unnecessarily complicated the code. The behaviour of `accept_any` is now the default (and only) option.
* Fix a bug in split where chunking would be skipped when the chunk size
happened to be an exact divisor of the buffer size used to read the
input stream.
The issue here was that file was being split byte-wise in chunks of 1G.
The input stream was being read in chunks of 8KB, which evenly divides
the chunk size. Because the check to allocate the next output chunk was
done at the bottom of the loop previously, it would never occur because
the current input chunk was fully consumed at that point. By moving the
check to the top of the loop (but still late enough that we know we have
bytes to write) we resolve this issue.
This scenario is unfortunately hard to write a test for, since we don't
explicitly control the input chunk size.
Fixes https://github.com/uutils/coreutils/issues/3790
* test/cp: -p changes ctime and added sleep for better timestamp testing
* test/cp: add nanoseconds checking for copied timestamps
* test/cp: made compilable on other OSes
* test/cp: added error messages to assert_eq calls
* cp: Refactor `reflink`/`sparse` handling to enable `--sparse` flag
`--sparse` and `--reflink` options have a lot of similarities:
- They have similar options (`always`, `never`, `auto`)
- Both need OS specific handling
- They can be mutually exclusive
Prior to this change, `sparse` was defined as `CopyMode`, but `reflink`
wasn't. Given the similarities, it makes sense to handle them similarly.
The idea behind this change is to move all OS specific file copy
handling in the `copy_on_write_*` functions. Those function then
dispatch to the correct logic depending on the arguments (at the moment,
the tuple `(reflink, sparse)`).
Also, move the handling of `--reflink=never` from `copy_file` to the
`copy_on_write_*` functions, at the cost of a bit of code duplication,
to allow `copy_on_write_*` to handle all cases (and later handle
`--reflink=never` with `--sparse`).
* cp: Implement `--sparse` flag
This begins to address #3362
At the moment, only the `--sparse=always` logic matches the requirement
form GNU cp info page, i.e. always make holes in destination when
possible.
Sparse copy is done by copying the source to the destination block by
block (blocks being of the destination's fs block size). If the block
only holds NUL bytes, we don't write to the destination.
About `--sparse=auto`: according to GNU cp info page, the destination
file will be made sparse if the source file is sparse as well. The next
step are likely to use `lseek` with `SEEK_HOLE` detect if the source
file has holes. Currently, this has the same behaviour as
`--sparse=never`. This `SEEK_HOLE` logic can also be applied to
`--sparse=always` to improve performance when copying sparse files.
About `--sparse=never`: from my understanding, it is not guaranteed that
Rust's `fs::copy` will always produce a file with no holes, as
["platform-specific behavior may change in the
future"](https://doc.rust-lang.org/std/fs/fn.copy.html#platform-specific-behavior)
About other platforms:
- `macos`: The solution may be to use `fcntl` command `F_PUNCHHOLE`.
- `windows`: I only see `FSCTL_SET_SPARSE`.
This should pass the following GNU tests:
- `tests/cp/sparse.sh`
- `tests/cp/sparse-2.sh`
- `tests/cp/sparse-extents.sh`
- `tests/cp/sparse-extents-2.sh`
`sparse-perf.sh` needs `--sparse=auto`, and in particular a way to skip
holes in the source file.
Co-authored-by: Sylvestre Ledru <sylvestre@debian.org>
* ls: Implement --zero flag. (#2929)
This flag can be used to provide a easy machine parseable output from
ls, as discussed in the GNU bug report
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=49716.
There are some peculiarities with this flag:
- Current implementation of GNU ls of the `--zero` flag implies some
other flags. Those can be overridden by setting those flags after
`--zero` in the command line.
- This flag is not compatible with `--dired`. This patch is not 100%
compliant with GNU ls: GNU ls `--zero` will fail if `--dired` and
`-l` are set, while with this patch only `--dired` is needed for the
command to fail.
We also add `--dired` flag to the parser, with no additional behaviour
change.
Testing done:
```
$ bash util/build-gnu.sh
[...]
$ bash util/run-gnu-test.sh tests/ls/zero-option.sh
[...]
PASS: tests/ls/zero-option.sh
============================================================================
Testsuite summary for GNU coreutils 9.1.36-8ec11
============================================================================
# TOTAL: 1
# PASS: 1
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
============================================================================
```
* Use the US way to spell Behavior
* Fix formatting with cargo fmt -- tests/by-util/test_ls.rs
* Simplify --zero flag overriding logic by using index_of
Also, allow multiple --zero flags, as this is possible with GNU ls
command. Only the last one is taken into account.
Co-authored-by: Sylvestre Ledru <sledru@mozilla.com>
* wc: specialize scanning loop on settings.
The primary computational loop in wc (iterating over all the
characters and computing word lengths, etc) is configured by a
number of boolean options that control the text-scanning behavior.
If we monomorphize the code loop for each possible combination of
scanning configurations, the rustc is able to generate better code
for each instantiation, at the least by removing the conditional
checks on each iteration, and possibly by allowing things like
vectorization.
On my computer (aarch64/macos), I am seeing at least a 5% performance
improvement in release builds on all wc flag configurations
(other than those that were already specialized) against
odyssey1024.txt, with wc -l showing the greatest improvement at 15%.
* Reduce the size of the wc dispatch table by half.
By extracting the handling of hand-written fast-paths to the
same dispatch as the automatic specializations, we can avoid
needing to pass `show_bytes` as a const generic to
`word_count_from_reader_specialized`. Eliminating this parameter
halves the number of arms in the dispatch.
* ls: handle looping symlinks infinite printing
* ls: better coloring and printing symlinks when dereferenced
* tests/ls: add dereferencing and symlink loop tests
* ls: reformat changed using rustfmt
* ls: follow clippy advice for cleaner code
* uucore/fs: fix FileInformation to open directory handles in Windows as
well
This is part of fixing the tee tests. 'yes' is used by the GNU test
suite to identify what the SIGPIPE exit code is on the target
platform. By trapping SIGPIPE, it creates a requirement that other
utilities also trap SIGPIPE (and exit 0 after SIGPIPE). This is
sometimes at odds with their desired behaviour.
This has the following behaviours. On Unix:
- The default is to exit on pipe errors, and warn on other errors.
- "--output-error=warn" means to warn on all errors
- "--output-error", "--output-error=warn-nopipe" and "-p" all mean
that pipe errors are suppressed, all other errors warn.
- "--output-error=exit" means to warn and exit on all errors.
- "--output-error=exit-nopipe" means to suppress pipe errors, and to
warn and exit on all other errors.
On non-Unix platforms, all pipe behaviours are ignored, so the default
is effectively "--output-error=warn" and "warn-nopipe" is identical.
The only meaningful option is "--output-error=exit" which is identical
to "--output-error=exit-nopipe" on these platforms.
Note that warnings give a non-zero exit code, but do not halt writing
to non-erroring targets.
Fix a bug in which `cp` incorrectly exited with an error when
attempting to copy the attributes of a dangling symbolic link (that
is, when running `cp -P -p`).
Fixes#3531.
* tail: reduce CPU load for polling
This reduces the CPU load for polling drastically (from ~80% down to ~5%)
by removing/fixing several previous workarounds related to polling,
while still passing all related GNU test-suite checks.
* set Notify::PollWatcher delay to: sleep_sec/10 instead of
sleep_sec/100
* set recv_timeout to sleep_sec instead of sleep_sec/100
* remove the manual polling of watched files
Bugs:
* fix an issue with headers to consistently pass
"test_follow_name_retry_headers" and "gnu/tests/tail-2/overlay-headers.sh"
Code clean-up and refactor
* make fields of struct FileHandling private (and add getters/setters)
to ensure that the paths are absolute and match the paths returned by
Notify::Events
* replace calls to "crash!" with "return USimpleError"
* clean-up formatting
When `--backup` is supplied, `cp` will take a backup of *destination* before *source* is copied. When `--backup=simple` is supplied, it is possible for the backup path for *destination* to equal the path for *source*, destroying source before the copy is made. This change prevents this by returning an error instead.
This fixes https://github.com/uutils/coreutils/issues/3629
Update `dd` to only print a concise form of the number of bytes with
an SI prefix (like "1 MB" or "2 GB") if the number is at least
1000. Similarly, only print the concise form with an IEC prefix (like
"1 MiB" or "2 GiB") if the number is at least 1024. For example,
$ head -c 999 /dev/zero | dd > /dev/null
1+1 records in
1+1 records out
999 bytes copied, 0.0 s, 999.0 KB/s
$ head -c 1000 /dev/zero | dd > /dev/null
1+1 records in
1+1 records out
1000 bytes (1000 B) copied, 0.0 s, 1000.0 KB/s
$ head -c 1024 /dev/zero | dd > /dev/null
2+0 records in
2+0 records out
1024 bytes (1 KB, 1024 B) copied, 0.0 s, 1.0 MB/s
Make `mktemp` exit with an error if the `--suffix` option is the empty
string and the template argument does not end in an "X". Previously,
the program succeeded.
Before this commit,
$ mktemp --suffix= aXXXb
apBEb
After this commit,
$ mktemp --suffix= aXXXb
mktemp: with --suffix, template 'aXXXb' must end in X
It is probably too hard to verify that the sync is actually performed,
so we just check that we have a test using the code path, pro forma.
Signed-off-by: anastygnome <noreplygitemail@protonmail.com>
Add a unit test for combining the directory given in `--tmpdir` with
any subdirectory structure appearing in the prefix of the template
string. For example,
$ mktemp --tmpdir=a b/cXXX
a/b/cNqJ
This behavior is currently working, but a unit test was missing.
Fix a bug in which `mktemp` would replace everything in the template
argument from the first 'X' to the last 'X' with random bytes, instead
of just replacing the last contiguous block of 'X's.
Before this commit,
$ mktemp XXX_XXX
2meCpfM
After this commit,
$ mktemp XXX_XXX
XXX_Rp5
This fixes test cases `suffix2f` and `suffix2d` in
`tests/misc/mktemp.pl` in the GNU coreutils test suite.
Simplify the logic of computing the file path parameters (the
directory, prefix, suffix, and number of random characters) for the
temporary file created by `mktemp`. This commits adds an `Options`
struct as a layer of indirection between the application logic and
`clap`, and a `Params` struct whose associated function is responsible
for determining the file path parameters from the `Options`. This is
an improvement because the previous code had some logic for
determining file path parameters in one place and some in another
place.
Show the "total" label in the "source" column or in the "target"
column if the "source" column is not visible.
Before this commit,
$ df --total --output=target .
Mounted on
/
-
After this commit,
$ df --total --output=target .
Mounted on
/
total
Include the suffix in the error message produced by `mktemp` when
there are too few Xs in the template. Before this commit,
$ mktemp --suffix=X aXX
mktemp: too few X's in template 'aXX'
After this commit,
$ mktemp --suffix=X aXX
mktemp: too few X's in template 'aXXX'
This matches the behavior of GNU `mktemp`.
Correct the error message when the template argument contains a path
separator in its suffix. Before this commit:
$ mktemp aXXX/b
mktemp: too few X's in template 'b'
After this commit:
$ mktemp aXXX/b
mktemp: invalid suffix '/b', contains directory separator
This error message is more appropriate and matches the behavior of GNU
mktemp.
On macOS path.is_dir() can be false for directories
if it was a redirect, e.g. ` tail < DIR`
* fix some tests for macOS
Cleanup:
* fix clippy/spell-checker
* fix build for windows by refactoring stdin_is_pipe_or_fifo()
Correct the error message produced by `mktemp` when `--tmpdir` is
given and the template is an absolute path:
$ mktemp --tmpdir=a /XXX
mktemp: invalid template, '/XXX'; with --tmpdir, it may not be absolute
* add various tests adapted from `gnu/tests/tail-2/follow-stdin.sh`
* explicitly set_stdin to null where needed, otherwise stdin is always
`piped`
* tighten some existing tests (no_stderr, code_is, etc)
* add test for fifo
* add various tests adapted from `gnu/tests/tail-2/follow-stdin.sh`
* explicitly set_stdin to null where needed, otherwise stdin is always
`piped`
* tighten some existing tests (no_stderr, code_is, etc)
* add test for fifo
On Android and macOS all/some tests for stdin fail with:
`cannot stat '-': No such file or directory`
Apparently the `/dev/stdin` redirect workaround doesn't work for
these targets.
Update `chown` to allow setting the owner of a file to a numeric user
ID regardless of whether a corresponding username exists on the
system.
For example,
$ touch f && sudo chown 12345 f
succeeds even though there is no named user with ID 12345.
Fixes#3380.
Correct the error that arises from a path separator in the prefix
portion of a template argument provided to `mktemp`. Before this
commit, the error message was incorrect:
$ mktemp -t a/bXXX
mktemp: failed to create file via template 'a/bXXX': No such file or directory (os error 2) at path "/tmp/a/bege"
After this commit, the error message is correct:
$ mktemp -t a/bXXX
mktemp: invalid template, 'a/bXXX', contains directory separator
The code was failing to check for a path separator in the prefix
portion of the template.
* ensure that `fn run_no_wait` is only invoked if the system running the
test has `sudo` in $path
Previously, if run inside CICD, calling `fn run_ucmd_as_root`
would provoke `fn run_no_wait` to panic because there's no `sudo`.
When using left-justify with integer conversion (like `printf '%-o'`),
default the minimum width to 1.
Closes: https://github.com/uutils/coreutils/issues/3050
Signed-off-by: Hanif Ariffin <hanif.ariffin.4326@gmail.com>
Fix a bug in `mktemp` where it was not respecting the path given by
the positional argument. Previously, it would place the temporary file
whose name is induced by a given template in the `/tmp` directory,
like this:
$ mktemp XXX
/tmp/LJr
$ mktemp d/XXX
/tmp/d/IhS
After this commit, it respects the directory given in the template
argument:
$ mktemp XXX
LJr
$ mktemp d/XXX
d/IhS
Fixes#3440.
* Fix a timing related bug with polling (---disable-inotify) where some
Events weren't delivered fast enough by `Notify::PollWatcher` to pass all
of tests/tail-2/retry.sh and test_tail::{test_retry4, retry7}.
* uu_tail now reverts to polling automatically if inotify backend reports
too many open files (this mimics the behavior of GNU's tail).
This makes uu_tail pass the "gnu/tests/tail-2/inotify-only-regular" test
again by adding support for charater devices.
test_tail:
* add test_follow_inotify_only_regular
* add clippy fixes for windows
The code for creating a Passwd from the fields of the raw syscall result
assumed that the syscall would return valid C strings in all non-error
cases. This is not true, and at least one platform (Android) will
populate the fields with null pointers where they are not supported.
To fix this and prevent the error from happening again, this commit
changes `cstr2string(ptr)` to check for a null pointer, and return an
`Option<String>`, with `None` being the null pointer case. While
arguably it should be the caller's job to check for a null pointer
before calling (since the safety precondition is that the pointer is to
a valid C string), relying on the type checker to force remembering this
edge case is safer in the long run.
Add a missing dash to the `--total` argument applied in the
`test_df_output` test case. Before this commit, the argument `-total`
was treated as a path argument. After this commit, `--total` is
treated as a command-line option that causes the total file usage to
be displayed.
Fix a bug in which the error message displayed when using
`CmdResult::no_stdout()` was incorrectly showing stderr when it should
have been showing stdout.
* hashsum: add --no-names option from official b3sum tool
The official b3sum tool has a --no-names option for only printing the
hashes, omitting the filenames. This is quite handy when used from
scripts because it spares the postprocessing with "cut" or "awk".
Since the installed b3sum symlink would also serve as a drop-in for the
official tool, the --no-names option is expected to exist for
compatibility.
Add a --no-names option not only for b3sum but for hashsum in general
(and maybe GNU coreutils will also feel inspired to add this option).
Closes https://github.com/uutils/coreutils/issues/3360
Change formula from: "Used/Size * 100" to "Used/(Used + Avail) * 100".
This formula also works if "Used" and "Avail" do not add up to "Size",
which is the case if there are reserved disk blocks.
When doing
ln b b~
ln -f --b=simple a b
First, we create a backup of b
Then, we force the override of a => b but we make sure that the backup is
done.
So, we had a bug in the ordering of the actions.
we were first removing b. Therefore, losing the capability to do a backup of this.
Change formula from: "Used/Size * 100" to "Used/(Used + Avail) * 100".
This formula also works if "Used" and "Avail" do not add up to "Size",
which is the case if there are reserved disk blocks.
Previously, individual file sizes were used to compute the number width, which
would cause misalignment when the total has a greater number of digits, and is
different from the behavior of GNU wc
```
$ ./target/debug/wc -w -l -m -c -L deny.toml GNUmakefile
95 422 3110 3110 85 deny.toml
349 865 6996 6996 196 GNUmakefile
444 1287 10106 10106 196 total
$ wc -w -l -m -c -L deny.toml GNUmakefile
95 422 3110 3110 85 deny.toml
349 865 6996 6996 196 GNUmakefile
444 1287 10106 10106 196 total
```
"df --output ." was treated as "df --output=." and hence "." was
interpreted as a column name. With this commit, "." is treated as
an argument on its own.
Fixes#3324
Print a usage error when duplicat column names are specified to the
`--output` command-line argument. For example,
$ df --output=source,source
df: option --output: field ‘source’ used more than once
Try 'df --help' for more information.
Implement the "File" column in the `df` output table. Before this
commit, a blank entry appeared in the "File" column for each
row. After this commit, a "-" entry appears when `df` is run with no
positional arguments and the filename appears when run with positional
arguments. For example:
$ touch a b c && df --output=target,file a b c
Mounted on File
/ a
/ b
/ c
