Byte, character, and line counting can all be done on the raw bytes
of the incoming stream without decoding the Unicode characters. This
fact was previously exploited in specific fast paths for counting
characters and counting lines. This change unifies those fast paths into
a single shared fast paths, using const generics to specialize the
function for each use case. This has the benefit of making sure that all
combinations of these Unicode-oblivious fast paths benefit from the same
optimization.
On my laptop, this speeds up `wc -clm odyssey1024.txt` from 840ms to
120ms. I experimented with using a filter loop for line counting, but
continuing to use the bytecount crate came out ahead by a significant
margin.
When wc is invoked with only the -m flag, we only need to count the
number of Unicode characters in the input. In order to do so, we don't
actually need to decode the input bytes into characters. Rather, we can
simply count the number of non-continuation bytes in the UTF-8 stream,
since every character will contain exactly one non-continuation byte.
On my laptop, this speeds up `wc -m odyssey1024.txt` from 745ms to
109ms.
* wc: specialize scanning loop on settings.
The primary computational loop in wc (iterating over all the
characters and computing word lengths, etc) is configured by a
number of boolean options that control the text-scanning behavior.
If we monomorphize the code loop for each possible combination of
scanning configurations, the rustc is able to generate better code
for each instantiation, at the least by removing the conditional
checks on each iteration, and possibly by allowing things like
vectorization.
On my computer (aarch64/macos), I am seeing at least a 5% performance
improvement in release builds on all wc flag configurations
(other than those that were already specialized) against
odyssey1024.txt, with wc -l showing the greatest improvement at 15%.
* Reduce the size of the wc dispatch table by half.
By extracting the handling of hand-written fast-paths to the
same dispatch as the automatic specializations, we can avoid
needing to pass `show_bytes` as a const generic to
`word_count_from_reader_specialized`. Eliminating this parameter
halves the number of arms in the dispatch.
This change will extract a utility already present in ls to uucore.
This utility is used by dir and vdir too, which are adjusted to
look it up in uucode. No further changes to ls, dir or dirv intended.
The change here largely fiddles with the output of uu_wc to match
that of GNU wc. This is the case to the extent to make unit tests
pass, however, there are differences remaining. One specific
difference I did not tackle is that GNU wc will not align the
output columns (compute_number_width() -> 1) in the specific case
of the input for --files0-from=- being a named pipe, not real stdin.
This difference can be triggered using the following two invocations.
- wc --files0-from=- < files0 # use a named pipe, GNU does align
- cat files0- | wc --files0-from=- # use real stdin, GNU does not
align.
Previously, individual file sizes were used to compute the number width, which
would cause misalignment when the total has a greater number of digits, and is
different from the behavior of GNU wc
```
$ ./target/debug/wc -w -l -m -c -L deny.toml GNUmakefile
95 422 3110 3110 85 deny.toml
349 865 6996 6996 196 GNUmakefile
444 1287 10106 10106 196 total
$ wc -w -l -m -c -L deny.toml GNUmakefile
95 422 3110 3110 85 deny.toml
349 865 6996 6996 196 GNUmakefile
444 1287 10106 10106 196 total
```
This should correct the usage strings in both the `--help` and user documentation. Previously, sometimes the name of the utils did not show up correctly.
When this option is present, the files argument is not processed. This option processes the file list from provided file, splitting them by the ascii NUL (\0) character. When files0-from is '-', the file list is processed from stdin.
- Change the main! proc_macro to a bin! macro_rules macro.
- Reexport uucore_procs from uucore
- Make utils to not import uucore_procs directly
- Remove the `syn` dependency and don't parse proc_macro input (hopefully for faster compile times)
If reading fails midway through then a count should be reported for
what could be read.
If opening a file fails then no count should be reported.
The exit code shouldn't report the number of failures, that's fragile
in case of many failures.
Errors are now always shown with the corresponding filename.
Errors are no longer converted into warnings. Previously `wc < .`
would cause a loop.
Checking whether something is a directory is no longer done in
advance. This removes race conditions and the edge case where stdin is
a directory.
The custom error type is removed because io::Error is now enough.
- Reuse allocations for read lines
- Increase splice size
- Check if /dev/null was opened correctly
- Do not discard read bytes after I/O error
- Add fast line counting with bytecount
Fails with:
```
error: use of irregular braces for `write!` macro
--> src/uucore/src/lib/features/encoding.rs:19:17
|
19 | #[derive(Debug, Error)]
| ^^^^^
|
= note: `-D clippy::nonstandard-macro-braces` implied by `-D warnings`
help: consider writing `Error`
--> src/uucore/src/lib/features/encoding.rs:19:17
|
19 | #[derive(Debug, Error)]
| ^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#nonstandard_macro_braces
= note: this error originates in the derive macro `Error` (in Nightly builds, run with -Z macro-backtrace for more info)
```
This makes clap wrap the help text according to the terminal width,
which improves readability for terminal widths < 120 chars,
because clap defaults to a width of 120 chars without this feature.
Change the behavior of `wc` to print the counts for a file as soon as
it is computed, instead of waiting to compute the counts for all files
before writing any output to `stdout`. The new behavior matches the
behavior of GNU `wc`.
The old behavior looked like this (the word "hello" is entered on
`stdin`):
$ wc emptyfile.txt -
hello
0 0 0 emptyfile.txt
1 1 6
1 1 6 total
The new behavior looks like this:
$ wc emptyfile.txt -
0 0 0 emptyfile.txt
hello
1 1 6
1 1 6 total
Fix two issues with the string formatting width for counts displayed
by `wc`.
First, the output was previously not using the default minimum width
(seven characters) when reading from `stdin`. This commit corrects
this behavior to match GNU `wc`. For example,
$ cat alice_in_wonderland.txt | wc
5 57 302
Second, if at least 10^7 bytes were read from `stdin` *after* reading
from a smaller regular file, then every output row would have width
8. This disagrees with GNU `wc`, in which only the `stdin` row and the
total row would have width 8. This commit corrects this behavior to
match GNU `wc`. For example,
$ printf "%.0s0" {1..10000000} | wc emptyfile.txt -
0 0 0 emptyfile.txt
0 1 10000000
0 1 10000000 total
Fixes#2186.
Change the error messages that get printed to `stderr` for compatibility
with GNU `wc` when an input is a directory and when an input does not
exist.
Fixes#2211.
Refactor the counting code from the inner loop of the `wc` program
into the `WordCount::from_line()` associated function. This commit
also splits that function up into other helper functions that
encapsulate decoding characters and finding word boundaries from raw
bytes.
This commit also implements the `Sum` trait for the `WordCount`
struct, so that we can simply call `sum()` on an iterator that yields
`WordCount` instances.
Add the `WordCountable::lines()` method that returns an iterator over
lines of a file-like object. This mirrors the
`std::io::BufRead::lines()` method, with some minor differences due to
the particular use case of `wc`.
This commit also creates a new module, `countable.rs`, to contain the
`WordCountable` trait and the new `Lines` struct returned by `lines()`.
* cat: Unrevert splice patch
* cat: Add fifo test
* cat: Add tests for error cases
* cat: Add tests for character devices
* wc: Make sure we handle short splice writes
* cat: Fix tests for 1.40.0 compiler
* cat: Run rustfmt on test_cat.rs
* Run 'cargo +1.40.0 update'
