* tail: reduce CPU load for polling
This reduces the CPU load for polling drastically (from ~80% down to ~5%)
by removing/fixing several previous workarounds related to polling,
while still passing all related GNU test-suite checks.
* set Notify::PollWatcher delay to: sleep_sec/10 instead of
sleep_sec/100
* set recv_timeout to sleep_sec instead of sleep_sec/100
* remove the manual polling of watched files
Bugs:
* fix an issue with headers to consistently pass
"test_follow_name_retry_headers" and "gnu/tests/tail-2/overlay-headers.sh"
Code clean-up and refactor
* make fields of struct FileHandling private (and add getters/setters)
to ensure that the paths are absolute and match the paths returned by
Notify::Events
* replace calls to "crash!" with "return USimpleError"
* clean-up formatting
When `--backup` is supplied, `cp` will take a backup of *destination* before *source* is copied. When `--backup=simple` is supplied, it is possible for the backup path for *destination* to equal the path for *source*, destroying source before the copy is made. This change prevents this by returning an error instead.
This fixes https://github.com/uutils/coreutils/issues/3629
Update `dd` to only print a concise form of the number of bytes with
an SI prefix (like "1 MB" or "2 GB") if the number is at least
1000. Similarly, only print the concise form with an IEC prefix (like
"1 MiB" or "2 GiB") if the number is at least 1024. For example,
$ head -c 999 /dev/zero | dd > /dev/null
1+1 records in
1+1 records out
999 bytes copied, 0.0 s, 999.0 KB/s
$ head -c 1000 /dev/zero | dd > /dev/null
1+1 records in
1+1 records out
1000 bytes (1000 B) copied, 0.0 s, 1000.0 KB/s
$ head -c 1024 /dev/zero | dd > /dev/null
2+0 records in
2+0 records out
1024 bytes (1 KB, 1024 B) copied, 0.0 s, 1.0 MB/s
Make `mktemp` exit with an error if the `--suffix` option is the empty
string and the template argument does not end in an "X". Previously,
the program succeeded.
Before this commit,
$ mktemp --suffix= aXXXb
apBEb
After this commit,
$ mktemp --suffix= aXXXb
mktemp: with --suffix, template 'aXXXb' must end in X
It is probably too hard to verify that the sync is actually performed,
so we just check that we have a test using the code path, pro forma.
Signed-off-by: anastygnome <noreplygitemail@protonmail.com>
Add a unit test for combining the directory given in `--tmpdir` with
any subdirectory structure appearing in the prefix of the template
string. For example,
$ mktemp --tmpdir=a b/cXXX
a/b/cNqJ
This behavior is currently working, but a unit test was missing.
Fix a bug in which `mktemp` would replace everything in the template
argument from the first 'X' to the last 'X' with random bytes, instead
of just replacing the last contiguous block of 'X's.
Before this commit,
$ mktemp XXX_XXX
2meCpfM
After this commit,
$ mktemp XXX_XXX
XXX_Rp5
This fixes test cases `suffix2f` and `suffix2d` in
`tests/misc/mktemp.pl` in the GNU coreutils test suite.
Simplify the logic of computing the file path parameters (the
directory, prefix, suffix, and number of random characters) for the
temporary file created by `mktemp`. This commits adds an `Options`
struct as a layer of indirection between the application logic and
`clap`, and a `Params` struct whose associated function is responsible
for determining the file path parameters from the `Options`. This is
an improvement because the previous code had some logic for
determining file path parameters in one place and some in another
place.
Show the "total" label in the "source" column or in the "target"
column if the "source" column is not visible.
Before this commit,
$ df --total --output=target .
Mounted on
/
-
After this commit,
$ df --total --output=target .
Mounted on
/
total
Include the suffix in the error message produced by `mktemp` when
there are too few Xs in the template. Before this commit,
$ mktemp --suffix=X aXX
mktemp: too few X's in template 'aXX'
After this commit,
$ mktemp --suffix=X aXX
mktemp: too few X's in template 'aXXX'
This matches the behavior of GNU `mktemp`.
Correct the error message when the template argument contains a path
separator in its suffix. Before this commit:
$ mktemp aXXX/b
mktemp: too few X's in template 'b'
After this commit:
$ mktemp aXXX/b
mktemp: invalid suffix '/b', contains directory separator
This error message is more appropriate and matches the behavior of GNU
mktemp.
On macOS path.is_dir() can be false for directories
if it was a redirect, e.g. ` tail < DIR`
* fix some tests for macOS
Cleanup:
* fix clippy/spell-checker
* fix build for windows by refactoring stdin_is_pipe_or_fifo()
Correct the error message produced by `mktemp` when `--tmpdir` is
given and the template is an absolute path:
$ mktemp --tmpdir=a /XXX
mktemp: invalid template, '/XXX'; with --tmpdir, it may not be absolute
* add various tests adapted from `gnu/tests/tail-2/follow-stdin.sh`
* explicitly set_stdin to null where needed, otherwise stdin is always
`piped`
* tighten some existing tests (no_stderr, code_is, etc)
* add test for fifo
* add various tests adapted from `gnu/tests/tail-2/follow-stdin.sh`
* explicitly set_stdin to null where needed, otherwise stdin is always
`piped`
* tighten some existing tests (no_stderr, code_is, etc)
* add test for fifo
On Android and macOS all/some tests for stdin fail with:
`cannot stat '-': No such file or directory`
Apparently the `/dev/stdin` redirect workaround doesn't work for
these targets.
Update `chown` to allow setting the owner of a file to a numeric user
ID regardless of whether a corresponding username exists on the
system.
For example,
$ touch f && sudo chown 12345 f
succeeds even though there is no named user with ID 12345.
Fixes#3380.
Correct the error that arises from a path separator in the prefix
portion of a template argument provided to `mktemp`. Before this
commit, the error message was incorrect:
$ mktemp -t a/bXXX
mktemp: failed to create file via template 'a/bXXX': No such file or directory (os error 2) at path "/tmp/a/bege"
After this commit, the error message is correct:
$ mktemp -t a/bXXX
mktemp: invalid template, 'a/bXXX', contains directory separator
The code was failing to check for a path separator in the prefix
portion of the template.
When using left-justify with integer conversion (like `printf '%-o'`),
default the minimum width to 1.
Closes: https://github.com/uutils/coreutils/issues/3050
Signed-off-by: Hanif Ariffin <hanif.ariffin.4326@gmail.com>
Fix a bug in `mktemp` where it was not respecting the path given by
the positional argument. Previously, it would place the temporary file
whose name is induced by a given template in the `/tmp` directory,
like this:
$ mktemp XXX
/tmp/LJr
$ mktemp d/XXX
/tmp/d/IhS
After this commit, it respects the directory given in the template
argument:
$ mktemp XXX
LJr
$ mktemp d/XXX
d/IhS
Fixes#3440.
* Fix a timing related bug with polling (---disable-inotify) where some
Events weren't delivered fast enough by `Notify::PollWatcher` to pass all
of tests/tail-2/retry.sh and test_tail::{test_retry4, retry7}.
* uu_tail now reverts to polling automatically if inotify backend reports
too many open files (this mimics the behavior of GNU's tail).
This makes uu_tail pass the "gnu/tests/tail-2/inotify-only-regular" test
again by adding support for charater devices.
test_tail:
* add test_follow_inotify_only_regular
* add clippy fixes for windows
The code for creating a Passwd from the fields of the raw syscall result
assumed that the syscall would return valid C strings in all non-error
cases. This is not true, and at least one platform (Android) will
populate the fields with null pointers where they are not supported.
To fix this and prevent the error from happening again, this commit
changes `cstr2string(ptr)` to check for a null pointer, and return an
`Option<String>`, with `None` being the null pointer case. While
arguably it should be the caller's job to check for a null pointer
before calling (since the safety precondition is that the pointer is to
a valid C string), relying on the type checker to force remembering this
edge case is safer in the long run.
Add a missing dash to the `--total` argument applied in the
`test_df_output` test case. Before this commit, the argument `-total`
was treated as a path argument. After this commit, `--total` is
treated as a command-line option that causes the total file usage to
be displayed.
* hashsum: add --no-names option from official b3sum tool
The official b3sum tool has a --no-names option for only printing the
hashes, omitting the filenames. This is quite handy when used from
scripts because it spares the postprocessing with "cut" or "awk".
Since the installed b3sum symlink would also serve as a drop-in for the
official tool, the --no-names option is expected to exist for
compatibility.
Add a --no-names option not only for b3sum but for hashsum in general
(and maybe GNU coreutils will also feel inspired to add this option).
Closes https://github.com/uutils/coreutils/issues/3360
Change formula from: "Used/Size * 100" to "Used/(Used + Avail) * 100".
This formula also works if "Used" and "Avail" do not add up to "Size",
which is the case if there are reserved disk blocks.
When doing
ln b b~
ln -f --b=simple a b
First, we create a backup of b
Then, we force the override of a => b but we make sure that the backup is
done.
So, we had a bug in the ordering of the actions.
we were first removing b. Therefore, losing the capability to do a backup of this.
Change formula from: "Used/Size * 100" to "Used/(Used + Avail) * 100".
This formula also works if "Used" and "Avail" do not add up to "Size",
which is the case if there are reserved disk blocks.
Previously, individual file sizes were used to compute the number width, which
would cause misalignment when the total has a greater number of digits, and is
different from the behavior of GNU wc
```
$ ./target/debug/wc -w -l -m -c -L deny.toml GNUmakefile
95 422 3110 3110 85 deny.toml
349 865 6996 6996 196 GNUmakefile
444 1287 10106 10106 196 total
$ wc -w -l -m -c -L deny.toml GNUmakefile
95 422 3110 3110 85 deny.toml
349 865 6996 6996 196 GNUmakefile
444 1287 10106 10106 196 total
```
"df --output ." was treated as "df --output=." and hence "." was
interpreted as a column name. With this commit, "." is treated as
an argument on its own.
Fixes#3324
Print a usage error when duplicat column names are specified to the
`--output` command-line argument. For example,
$ df --output=source,source
df: option --output: field ‘source’ used more than once
Try 'df --help' for more information.
Implement the "File" column in the `df` output table. Before this
commit, a blank entry appeared in the "File" column for each
row. After this commit, a "-" entry appears when `df` is run with no
positional arguments and the filename appears when run with positional
arguments. For example:
$ touch a b c && df --output=target,file a b c
Mounted on File
/ a
/ b
/ c
Produce a usage error on an invalid signal argument. For example,
$ timeout --signal=invalid 1 sleep 0
timeout: 'invalid': invalid signal
Try 'timeout --help' for more information.
Return an error when a negative interval is provided as the argument
to `uucore::parse_time::from_str()`, since a `Duration` should only be
non-negative.
Fix a bug in the behavior of `split -e -n NUM` when the input file is
empty. Previously, it would panic due to overflow when subtracting 1
from 0. After this change, it will terminate successfully and produce
no output chunks.
Use `Duration::saturating_mul()` to avoid a panic due to overflow in
`uucore::parse_time::from_str()`. This change prevents panic on very
large arguments to timeout and sleep.
These are the first half of changes needed to pass the dd/bytes.sh tests:
- Add iseek and oseek options (additive with skip and seek options)
- Implement tests for the new flags, matching those from dd/bytes.sh
Implement distributing lines of a file in a round-robin manner to a
specified number of chunks. For example,
$ (seq 1 10 | split -n r/3) && head -v xa[abc]
==> xaa <==
1
4
7
10
==> xab <==
2
5
8
==> xac <==
3
6
9
Previously, given 'cp -P a b', where 'a' and 'b' were both symlinks, cp
would end up replacing the target of 'b'.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
Fix a bug where `timeout --preserve-status` was not correctly
preserving the status code of the child process if it timed out. When
that happens, the status code of the child process is considered to be
the signal number (in this case, `SIGTERM`). The exit status of
`timeout` is then 128 plus the numeric code associated with `SIGTERM`.
* Adds support for mount path prefix matching and input path
canonicalization
- Sorts mount paths in reverse lexicographical order
- Canonicalize all paths and clear invalid paths
- Checking of mount path prefix matches input path
Implement the `--line-bytes` option to `split`. In this mode, the
program tries to write as many lines of the input as possible to each
chunk of output without exceeding a specified byte limit. The new
`LineBytesChunkWriter` struct represents this functionality.
Implement the `--output` command-line argument, which allows
specifying an exact sequence of columns to display in the output
table. For example,
$ df --output=source,fstype | head -n3
Filesystem Type
udev devtmpfs
tmpfs tmpfs
(The spacing does not exactly match the spacing of GNU `df` yet.)
Fixes#3057.
Correct the column header printed by `df` when the `--block-size`
argument has a value that is a multiple of 1024. After this commit,
the header looks like "1K" or "4M" or "117G", etc., depending on the
particular value of the block size. For example:
$ df --block-size=1024 | head -n1
Filesystem 1K-blocks Used Available Use% Mounted on
$ df --block-size=2048 | head -n1
Filesystem 2K-blocks Used Available Use% Mounted on
$ df --block-size=3072 | head -n1
Filesystem 3K-blocks Used Available Use% Mounted on
$ df --block-size=4096 | head -n1
Filesystem 4K-blocks Used Available Use% Mounted on
Add support for the `--total` option to `df`, which displays the total
of each numeric column. For example,
$ df --total
Filesystem 1K-blocks Used Available Use% Mounted on
udev 3858016 0 3858016 0% /dev
...
/dev/loop14 63488 63488 0 100% /snap/core20/1361
total 258775268 98099712 148220200 40% -
Implement `-n l/k/N` option, where the `k`th chunk of the input file
is written to stdout. For example,
$ seq -w 0 99 > f; split -n l/3/10 f
20
21
22
23
24
25
26
27
28
29
Add support for `split -n l/NUM`. Previously, `split` only supported
`-n NUM`, which splits a file into `NUM` chunks by byte. The `-n
l/NUM` strategy splits a file into `NUM` chunks without splitting
lines across chunks.
https://github.com/uutils/coreutils/pull/3084 (2a333ab391) had some
missing coverage and was merged before I had a chance to fix it.
This PR adds some coverage / improved error messages that were missing
from that previous PR.
If `conv=block,sync` command-line arguments are given and there is at
least one partial record read from the input (for example, if the
length of the input is not divisible by the value of the `ibs`
argument), then output an extra block of `cbs` spaces.
For example, no extra spaces are printed in this example because the
input is of length 10, a multiple of `ibs`:
$ printf "012\nabcde\n" \
> | dd ibs=5 cbs=5 conv=block,sync status=noxfer \
> && echo $
012 abcde$
2+0 records in
0+1 records out
But in this example, 5 extra spaces are printed because the length of
the input is not a multiple of `ibs`:
$ printf "012\nabcdefg\n" \
> | dd ibs=5 cbs=5 conv=block,sync status=noxfer \
> && echo $
012 abcde $
2+1 records in
0+1 records out
1 truncated record
The number of spaces printed is the size of the conversion block,
given by `cbs`.
Prevent `dd` from terminating with an error when given the
command-line argument `of=/dev/null`. This commit allows the call to
`File::set_len()` to result in an error without causing the process to
terminate prematurely.
Place the "truncated records" line below the "records out" line in the
status report produced by `dd` and properly handle the singularization
of the word "record" in the case of 1 truncated record. This matches
the behavior of GNU `dd`.
For example
$ printf "ab" | dd cbs=1 conv=block status=noxfer > /dev/null
0+1 records in
0+1 records out
1 truncated record
$ printf "ab\ncd\n" | dd cbs=1 conv=block status=noxfer > /dev/null
0+1 records in
0+1 records out
2 truncated records
Add the `-e` flag, which indicates whether to elide (that is, remove)
empty files that would have been created by the `-n` option.
The `-n` command-line argument gives a specific number of chunks into
which the input files will be split. If the number of chunks is
greater than the number of bytes, then empty files will be created for
the excess chunks. But if `-e` is given, then empty files will not be
created.
For example, contrast
$ printf 'a\n' > f && split -e -n 3 f && cat xaa xab xac
a
cat: xac: No such file or directory
with
$ printf 'a\n' > f && split -n 3 f && cat xaa xab xac
a
Clean up unit tests in the `dd` crate to make them easier to
manage. This commit does a few things.
* move test cases that test the complete functionality of the `dd`
program from the `dd_unit_tests` module up to the
`tests/by-util/test_dd.rs` module so that they can take advantage of
the testing framework and common testing tools provided by uutils,
* move test cases that test internal functions of the `dd`
implementation into the `tests` module within `dd.rs` so that they
live closer to the code they are testing,
* replace test cases defined by macros with test cases defined by
plain old functions to make the test cases easier to read at a
glance.
* include io-blksize parameter
* format changes for including io-blksize
Co-authored-by: DevSabb <devsabb@local>
Co-authored-by: Sylvestre Ledru <sylvestre@debian.org>
Add support for the `-x` command-line option to `split`. This option
causes `split` to produce filenames with hexadecimal suffixes instead
of the default alphabetic suffixes.
Correct the accounting for partial records written by `dd` to the
output file. After this commit, if fewer than `obs` bytes are written,
then that is counted as a partial record. For example,
$ printf 'abc' | dd bs=2 status=noxfer > /dev/null
1+1 records in
1+1 records out
That is, one complete record and one partial record are read from the
input, one complete record and one partial record are written to the
output. Previously, `dd` reported two complete records and zero
partial records written to the output in this case.
Change the `filter_mount_list()` function so that it always produces
the same order of `MountInfo` objects. This change ultimately results
in `df` printing its table of filesystems in the same order on each
execution. Previously, the table was in an arbitrary order because the
`MountInfo` objects were read from a `HashMap`.
Fixes#3086.
* ls: add new optional arguments to --classify flag
The --classify flag in ls now takes an option when argument
that may have the values always, auto and none.
Modified clap argument to allow an optional parameter and
changed the classify flag value parsing logic to account for
this change.
* ls: add test for indicator-style, ind and classify with value none
* ls: require option paramter to --classify to use a = to specify flag value
* ls: account for all the undocumented possible values for the --classify flag
Added the other values for the --classify flag along with modifications to tests.
Also documented the inconsistency between GNU coreutils because we accept the
flag value even for the short version of the flag.
Replace `ByteSplitter` and `LineSplitter` with `ByteChunkWriter` and
`LineChunkWriter` respectively. This results in a more maintainable
design and an increase in the speed of splitting by lines.
Correct the `test_split::test_suffixes_exhausted` test case so that it
actually exercises the intended behavior of `split`. Previously, the
test fixture contained 26 bytes. After this commit, the test fixture
contains 27 bytes. When using a suffix width of one, only 26 filenames
should be available when naming chunk files---one for each lowercase
ASCII letter. This commit ensures that the filenames will be exhausted
as intended by the test.
Show a warning if the `skip=N` command-line argument would cause `dd`
to skip past the end of the input. For example:
$ printf "abcd" | dd bs=1 skip=5 count=0 status=noxfer
'standard input': cannot skip to specified offset
0+0 records in
0+0 records out
Show a warning when a block size includes "0x" since this is
ambiguous: the user may have meant "multiply the next number by zero"
or they may have meant "the following characters should be interpreted
as a hexadecimal number".
When specifying `seek=N` and *not* specifying `conv=notrunc`, truncate
the output file to `N` blocks instead of truncating it to zero before
starting to write output. For example
$ printf "abc" > outfile
$ printf "123" | dd bs=1 skip=1 seek=1 count=1 status=noxfer of=outfile
1+0 records in
1+0 records out
$ cat outfile
a2
Fixes#3068.
When this option is present, the files argument is not processed. This option processes the file list from provided file, splitting them by the ascii NUL (\0) character. When files0-from is '-', the file list is processed from stdin.
Using this escaped character will cause `printf` to stop generating characters.
For instance,
```rust
hbina@akarin ~/g/uutils (hbina-add-test-for-additional-escape)> cargo run --quiet -- printf "%s\c%s" a b
a⏎
```
Signed-off-by: Hanif Ariffin <hanif.ariffin.4326@gmail.com>
* test_sort: Output sorted files to a file with different name
Signed-off-by: Hanif Bin Ariffin <hanif.ariffin.43262@gmail.com>
* Fix the test by saving the environment variable
Signed-off-by: Hanif Bin Ariffin <hanif.ariffin.43262@gmail.com>
Co-authored-by: Hanif Bin Ariffin <hanif.ariffin.43262@gmail.com>
