* Use buffered stdout to reduce write sys calls.
This simple change yielded the biggest performace gain.
* Use `for_byte_record_with_terminator` from the `bstr` crate.
This is to minimize the per line copying needed by
`BufReader::read_until`. The `cut_fields` and `cut_fields_delimiter`
functions used `read_until` to iterate over lines. That required copying
each input line to the line buffer. With
`for_byte_record_with_terminator` copying is minimized as it calls our
closure with a reference to BufReader's buffer most of the time. It
needs to copy (internally) only to process any incomplete lines at the
end of the buffer.
* Re-write `Searcher` to use `memchr`.
Switch from the naive implementation to one that uses `memchr`.
* Rewrite `cut_bytes` almost entirely.
This was already well optimized. The performance gain in this case is
not from avoiding copying. In fact, it needed zero copying whereas new
implementation introduces some copying similar to `cut_fields` described
above. But the occassional copying cost is more than offset by the use
of the very fast `memchr` inside `for_byte_record_with_terminator`.
This change also simplifies the code significantly. Removed the `buffer`
module.
This adds a --debug flag, which, when activated, will draw lines below
the characters that are actually used for comparisons.
This is not a complete implementation of --debug. It should, quoting the man page
for GNU sort: "annotate the part of the line used to sort, and warn
about questionable usage to stderr". Warning about "questionable usage"
is not part of this patch.
This change required some adjustments to be able to get the range that
is actually used for comparisons. Most notably, general numeric comparisons
were rewritten, fixing some bugs along the lines.
Testing is mostly done by adding fixtures for the expected debug output of
existing tests.
* ls: Remove allocations by eliminating collect/clones
* ls: Introduce PathData structure
- PathData will hold Path related metadata / strings that are required
frequently in subsequent functions
- All data is precomputed and cached and subsequent functions just
use cached data
* ls: Cache more data related to paths
- Cache filename and sort by filename instead of full path
- Cache uid->usr and gid->grp mappings
https://github.com/uutils/coreutils/pull/2099/files
* ls: Add BENCHMARKING.md
* ls: Document PathData structure
* tests/ls: Add testcase for error paths with width option
* ls: Fix unused import warning
cached will be only used for unix currently as current use of
caching gid/uid mappings is only relevant on unix
* ls: Suggest checking syscall count in BENCHMARKING.md
* ls: Remove mentions of sort in BENCHMARKING.md
* ls: Remove dependency on cached
Implement caching using HashMap and lazy_static
* ls: Fix MSRV error related to map_or
Rust 1.40 did not support map_or for result types
Trailing separators were included at the end of the last token, but they
should not be.
This changes tokenize_with_separator as suggested by @cbjadwani.
GNU sort disallows these combinations, presumably because they are
likely not what the user really wants.
Ignoring characters would cause things to be put together that aren't
together in the input. For example, -dn would cause "0.12" or "0,12" to
be parsed as "12" which is highly unexpected and confusing.
This reduces memory usage by only storing two lines of the input file at
a time. The current implementation first builds a list of all duplicate
lines ('group') and then decides which lines of the group should be
printed.
- Passing `never` to `--reflink` does not raise an error anymore.
- Remove `Options::reflink` flag as it was redundant with
`reflink_mode`.
- Add basic tests for this option. Does not check that a copy-on-write
rather than a regular copy was made.
* sort: use unstable sort when possible
This results in a very minor performance (speed) improvement.
It does however result in a memory usage reduction, because unstable
sort does not allocate auxiliary memory. There's also an improvement in
overall CPU usage.
* add benchmarking instructions
* add user time
* fix typo
* sort: implement numeric string comparison
This implements -n and -h using a string comparison algorithm instead
of parsing each number to a f64 and comparing those.
This should result in a moderate performance increase and eliminate loss
of precision.
* cache parsed f64 numbers
For general numeric comparisons we have to parse numbers as f64,
as this behavior is explicitly documented by GNU coreutils.
We can however cache the parsed value to speed up comparisons.
* fix leading zeroes for negative numbers
* use more appropriate name for exponent
* improvements to the parse function
* move checks into main loop and fix thousands separator condition
* remove unneeded checks
* rustfmt
* du error output should match GNU
* Created a new error macro which allows the customization of the
"error:" string part
* Match the du output based on the type of error encountered. Can extend
to handling other errors I guess.
* Rustfmt updates
* Added non-windows test for du no permission output
* Various fixes and performance improvements
* fix a typo
Co-authored-by: Michael Debertol <michael.debertol@gmail.com>
* Fix month parse for months with leading whitespace
* Implement test for months whitespace fix
* Confirm human numeric works as expected with whitespace with a test
* Correct arg help value name for --parallel
* Fix SemVer non version lines/empty line sorting with a test
Co-authored-by: Sylvestre Ledru <sledru@mozilla.com>
Co-authored-by: Michael Debertol <michael.debertol@gmail.com>
* cat: Unrevert splice patch
* cat: Add fifo test
* cat: Add tests for error cases
* cat: Add tests for character devices
* wc: Make sure we handle short splice writes
* cat: Fix tests for 1.40.0 compiler
* cat: Run rustfmt on test_cat.rs
* Run 'cargo +1.40.0 update'
* sort: implement basic -k and -t support
This allows to specify keys after the -k flag and a custom field
separator using -t.
Support for options for specific keys is still missing, and the -b flag
is not passed down correctly.
* sort: implement support for key options
* remove unstable feature use
* don't pipe in input when we expect a failure
* only tokenize when needed, remove a clone()
* improve comments
* fix clippy lints
* re-add test
* buffer writes to stdout
* fix ignore_non_printing
and make the test fail in case it is broken :)
* move attribute to the right position
* add more tests
* add my name to the copyright section
* disallow dead code
* move a comment
* re-add a loc
* use smallvec for a perf improvement in the common case
* add BENCHMARKING.md
* add ignore_case to benchmarks
* Various fixes and performance improvements
* fix a typo
Co-authored-by: Michael Debertol <michael.debertol@gmail.com>
Co-authored-by: Sylvestre Ledru <sledru@mozilla.com>
Co-authored-by: Michael Debertol <michael.debertol@gmail.com>
* Replace outdated time 0.1 dependancy with latest version of chrono
I also noticed that times are being miscalculated on linux, so I fixed that.
* Add time test for issue #2042
* Cleanup use declarations
* Tie time test to `touch` feature
- if we compile with the right OS feature flag then we should have it,
even on Windows
Forgot to handle the case where no arguments were passed to the COMMAND.
Because ARGS can be empty, we need two separate cases for handling
options.values_of(options::ARGS)
Fixed some minor ownership issues in converting from the options to the
arguments to the timeout COMMAND.
Additionally, fixed a rustfmt issue in other files (fold/stdbuf.rs)
Changed from optparse to clap.
None of the logic within timeout has been changed, which could use some
refactoring, but that's beyond the scope of this commit.
refactor `is_wsl` to `is_wsl_1` and `is_wsl_2`
On my tests with wsl2 ubuntu2004 all tests pass without special cases
I moved wsl detection into uucore so that it can be shared instead of duplicated
I moved `parse_mode` into uucode as it seemed to fit there better and anyway requires libc feature
Treat tab chars as advancing to the next tab stop rather than having a fixed
8-column width.
Also treat tab as a whitespace split target only when splitting on word
boundaries.
The '-d' flag should create all ancestors (or components) of a
directory regardless of the presence of the "-D" flag.
From the man page:
-d, --directory
treat all arguments as directory names; create all components of the specified directories
With GNU:
$ install -v -d dir1/di2
install: creating directory 'dir1'
install: creating directory 'dir1/di2'
With this version:
$ ./target/release/install -v -d dir3/di4
install: dir3/di4: No such file or directory (os error 2)
install: dir3/di4: chmod failed with error No such file or directory (os error 2)
install: created directory 'dir3/di4'
Also, one of the unit tests misinterprets what a "component" is,
and hence was fixed.
Also don't run chmod when we just failed to create the directory.
Behaviour before this patch:
$ ./target/release/install -v -d dir1/dir2
install: dir1/dir2: Permission denied (os error 13)
install: dir1/dir2: chmod failed with error No such file or directory (os error 2)
install: created directory 'dir1/dir2'
* Issue #1622 port `du` to windows
* Attempt to support Rust 1.32
Old version was getting "attributes are not yet allowed on `if`
expressions" on Rust 1.32
* Less #[cfg]
* Less duplicate code.
I need the return and the semicolon after if otherwise the second #[cfg]
leads to unexpected token complilation error
* More accurate size on disk calculations for windows
* Expect the same output on windows as with WSL
* Better matches output from du on WSL
* In the absence of feedback I'm disabling these tests on Windows.
They require `ln`. Windows does not ship with this utility.
* Use the coreutils version of `ln` to test `du`
`fn ccmd` is courtesy of @Artoria2e5
* Look up inodes (file ids) on Windows
* One more #[cfg(windows)] to prevent unreachable statement warning on linux
* cat: Improve performance, especially on Linux
* cat: Don't use io::copy for splice fallback
On my MacBook Pro 2020, it is around 25% faster to not use io::copy.
* cat: Only fall back to generic copy if first splice fails
* cat: Don't double buffer stdout
* cat: Don't use experimental or-pattern syntax
* cat: Remove nix symbol use from non-Linux
* wc: Don't read() if we only need to count number of bytes
* Resolve a few code review comments
* Use write macros instead of print
* Fix wc tests in case only one thing is printed
* wc: Fix style
* wc: Use return value of first splice rather than second
* wc: Make main loop more readable
* wc: Don't unwrap on failed write to stdout
* wc: Increment error count when stats fail to print
* Re-add Cargo.lock
* Implemented --indicator-style flag on ls.
* Rust fmt
* Grouped indicator_style args.
* Added tests for sockets and pipes.
Needed to modify util.rs to add support for pipes (aka FIFOs).
* Updated util.rs to remove FIFO operations on Windows
* Fixed slight error in specifying (not(windows))
* Fixed style violations and added indicator_style test for non-unix systems
