Prior to this fix, a file redirection was turned into an io_file_t. This is
annoying because every place where we want to apply the redirection, we
might fail due to open() failing. Switch to opening the file at the point
we resolve the redirection spec. This will simplify a lot of code.
Prior to this change, a process after it has been constructed by
parse_execution, but before it is executed, was given a list of
io_data_t redirections. The problem is that redirections have a
sensitive ownership policy because they hold onto fds. This made it
rather hard to reason about fd lifetime.
Change these to redirection_spec_t. This is a textual description
of a redirection after expansion. It does not represent an open file and
so its lifetime is no longer important.
This enables files to be held only on the stack, and are no longer owned
by a process of indeterminate lifetime.
fish has to ensure that the pipes it creates do not conflict with any
explicit fds named in redirections. Switch this code to using
autoclose_fd_t to make the ownership logic more explicit, and also
introduce fd_set_t to reduce the dependence on io_chain_t.
Prior to this fix, a job would hold onto any IO redirections from its
parent. For example:
begin
echo a
end < file.txt
The "echo a" job would hold a reference to the I/O redirection.
The problem is that jobs then extend the life of pipes until the job is
cleaned up. This can prevent pipes from closing, leading to hangs.
Fix this by not storing the block IO; this ensures that jobs do not
prolong the life of pipes.
Fixes#6397
Currently a job needs to know three things about its "parents:"
1. Any IO redirections for the block or function containing this job
2. The pgid for the parent job
3. Whether the parent job has been fully constructed (to defer self-disown)
These are all tracked in somewhat separate awkward ways. Collapse them
into a single new type job_lineage_t.
In preparation for concurrent execution, invert the control of function and
block execution. Allow a process to return an std::function that performs the
the execution. This can be run on either the main or a background thread
(eventually).
This splits a string into variables according to the shell's
tokenization rules, considering quoting, escaping etc.
This runs an automatic `unescape` on the string so it's presented like
it would be passed to the command. E.g.
printf '%s\n' a\ b
returns the tokens
printf
%s\n
a b
It might be useful to add another mode "--tokenize-raw" that doesn't
do that, but this seems to be the more useful of the two.
Fixes#3823.
Background fillthreads are used when we want to populate a buffer from an
external command. The most common is command substitution.
Prior to this commit, fish would spin up a fillthread whenever required.
This ended up being quite expensive.
Switch to using the iothread pool instead. This enables reusing the same
thread(s), which prevents needing to spawn new threads. This shows a big
perf win on the alias benchmark (766 -> 378 ms).
This reintroduces commits 22230a1a0d
and 9d7d70c204, now with the bug fixed.
The problem was when there was one thread waiting in the pool. We enqueue
an item onto the pool and attempt to wake up the thread. But before the
thread runs, we enqueue another item - this second enqueue will see the
thread waiting and attempt to wake it up as well. If the two work items
were dependent (reader/writer) then we would have a deadlock.
The fix is to check if the number of waiting threads is at least as large
as the queue. If the number of enqueued items exceeds the number of waiting
threads, then spawn a new thread always.
This added the function offset *again*, but it's already included in
the line for the current file.
And yes, I have explicitly tested a function file with a function
defined at a later line.
Fixes#6350
Since #6287, bare variable assignments do not parse, which broke
the "Unsupported use of '='" error message.
This commit catches parse errors that occur on bare variable assignments.
When a statement node fails to parse, then we check if there is at least one
prefixing variable assignment. If so, we emit the old error message.
See also #6347
This adds initial support for statements with prefixed variable assignments.
Statments like this are supported:
a=1 b=$a echo $b # outputs 1
Just like in other shells, the left-hand side of each assignment must
be a valid variable identifier (no quoting/escaping). Array indexing
(PATH[1]=/bin ls $PATH) is *not* yet supported, but can be added fairly
easily.
The right hand side may be any valid string token, like a command
substitution, or a brace expansion.
Since `a=* foo` is equivalent to `begin set -lx a *; foo; end`,
the assignment, like `set`, uses nullglob behavior, e.g. below command
can safely be used to check if a directory is empty.
x=/nothing/{,.}* test (count $x) -eq 0
Generic file completion is done after the equal sign, so for example
pressing tab after something like `HOME=/` completes files in the
root directory
Subcommand completion works, so something like
`GIT_DIR=repo.git and command git ` correctly calls git completions
(but the git completion does not use the variable as of now).
The variable assignment is highlighted like an argument.
Closes#6048
uClibc-ng does not expose C++11 math
functions to the std namespace, breaking
compilation. This is fine as the argument
type is double.
Signed-off-by: Rosen Penev <rosenp@gmail.com>
Improve the iothread behavior by enabling an iothread to stick around for
a while waiting for work. This reduces the amount of iothread churn, which
is useful on platforms where threads are expensive.
Also do other modernization like clean up the locking discipline and use
FLOG.
fish will react to certain variable modifications, such as "TZ." Only do
this if the main stack is modified. This has no effect now because there
is always a single stack, but will become important when concurrent
execution is supported.
This adds string-x.rst for each subcommand x of string. The main page
(string.rst) is not changed, except that examples are shown directly after
each subcommand. The subcommand sections in string.rst are created by
textual inclusion of parts of the string-x.rst files.
Subcommand man pages can be viewed with either of:
```
man string collect
man string-collect
string collect <press F1 or Alt-h>
string collect -h
```
While `string -h ...` still prints the full help.
Closes#5968
Presently the completion engine ignores builtins that are part of the
fish syntax. This can be a problem when completing a string that was
based on the output of `commandline -p`. This changes completions to
treat these builtins like any other command.
This also disables generic (filename) completion inside comments and
after strings that do not tokenize.
Additionally, comments are stripped off the output of `commandline -p`.
Fixes#5415Fixes#2705
The history search logic had a not very useful "fast path" which was also
buggy because it neglected to dedup. Switch the "fast path" to just a
history search type which always matches.
Fixes#6278
PATH and CDPATH have special behavior around empty elements. Express this
directly in env_stack_t::set rather than via variable dispatch; this is
cleaner.
This adds support for `fish_trace`, a new variable intended to serve the
same purpose as `set -x` as in bash. Setting this variable to anything
non-empty causes execution to be traced. In the future we may give more
specific meaning to the value of the variable.
The user's prompt is not traced unless you run it explicitly. Events are
also not traced because it is noisy; however autoloading is.
Fixes#3427
When considering an autosuggestion from history, we attempt to validate the
command to ensure that we don't suggest invalid (e.g. path-dependent)
commands. Prior to this fix, we would validate the last command in the
command line (e.g. in `cd /bin && ./stuff` we would validate "./stuff".
This doesn't really make sense; we should be validating the first command
because it has the potential to change the PWD. Switch to validating the
first command.
Also remove some helper functions that became dead through this change.
Until now, something like
`math '7 = 2'`
would complain about a "missing" operator.
Now we print an error about logical operators not being supported and
point the user towards `test`.
Fixes#6096
Every builtin or function shipped with fish supports flag -h or --help to
print a slightly condensed version of its manpage.
Some of those help messages are longer than a typical screen;
this commit pipes the help to a pager to make it easier to read.
As in other places in fish we assume that either $PAGER or "less" is a
valid pager and use that.
In three places (error messages for bg, break and continue) the help is
printed to stderr instead of stdout. To make sure the error message is
visible in the pager, we pass it to builtin_print_help, every call of which
needs to be updated.
Fixes#6227
The `--entire` would enable output even though the `--quiet` should
have silenced it. These two don't make any sense together so print an
error, because the user could have just left off the `-q`.
sys/sysctl.h is deprecated on glibc, so it leads to warnings.
According to fa4ec55c96, it was included for KERN_PROCARGS2 for
process expansion, but process expansion is gone, so it's unused now.
(there is another use of it in common.cpp, but that's only on FreeBSD)
Also 1f06e5f0b9 only included
tokenizer.h (present since the initial commit) if KERN_PROCARGS2
wasn't available, so it can't have been important.
This builds and passes the tests on:
- Archlinux, with glibc 2.30
- Alpine, with musl
- FreeBSD
- NetBSD
Universal exported variables (created by `set -xU`) used to show up
both as universal and global variable in child instances of fish.
As a result, when changing an exported universal variable, the
new value would only be visible after a new login (or deleting the
variable from global scope in each fish instance).
Additionally, something like `set -xU EDITOR vim -g` would be imported
into the global scope as a single word resulting in failures to
execute $EDITOR in fish.
We cannot simply give precedence to universal variables, because
another process might have exported the same variable. Instead, we
only skip importing a variable when it is equivalent to an exported
universal variable with the same name. We compare their values after
joining with spaces, hence skipping those imports does not change the
environment fish passes to its children. Only the representation in
fish is changed from `"vim -g"` to `vim -g`.
Closes#5258.
This eliminates the issue #5348 for universal variables.
Consider a group of short options, like -xzPARAM, where x and z are options and z takes an argument.
This commit enables completion of the argument to the last option (z), both within the same
token (-xzP) or in the next one (-xz P).
complete -C'-xz' will complete only parameters to z.
complete -C'-xz ' will complete only parameters to z if z requires a parameter
otherwise, it will also complete non-option parameters
To do so this implements a heuristic to differentiate such strings from single long options. To
detect whether our token contains some short options, we only require the first character after the
dash (here x) to be an option. Previously, all characters had to be short options. The last option
in our example is z. Everything after the last option is assumed to be a parameter to the last
option.
Assume there is also a single long option -x-foo, then complete -C'-x' will suggest both -x-foo and
-xy. However, when the single option x requires an argument, this will not suggest -x-foo.
However, I assume this will almost never happen in practise since completions very rarely mix
short and single long options.
Fixes#332
In e167714899 we allowed recursive calls
to complete. However, some completions use infinite recursion in their
completions and rely on `complete` to silently stop as soon as it is
called recursively twice without parameter (thus completing the
current commandline). For example:
complete -c su -s -xa "(complete -C(commandline -ct))"
su -c <TAB>
Infinite recursion happens because (commandline -ct) is an empty list,
which would print an error message. This commmit explicitly detects
such recursive calls where `complete` has no parameter and silently
terminates. This enables above completion (like before raising the
recursion limit) while still allowing legitimate cases with limited
recursion.
Closes#6171
This stops reading argument names after another option appears. It does not break any previous uses and in fact fixes uses like
```fish
function foo --argument-names bar --description baz
```
* `function` command handles options after argument names (Fixes#6186)
* Removed unneccesary test
Prior to this fix, self-insert would always wait for a new character.
Track in char_event what sequence generated the readline event, and then
if the sequence is not empty, insert that sequence.
This will support implementing the space binding via pure readline
functions.
Previously, tab-completion would move the cursor to the end of the current token, even
if no completion is inserted. This commit defers moving the cursor until we insert a completion.
Fixes#4124
If interactive, `complete` commands are highlighted like they would
be if typed. Adds a little fun contrast and it's easier to read.
Moved a function out of fish_indent to highlight.h
we now print --long options for ones I arbitrarily decided
are less likely to be remembered.
Also fixed the `--wraps` items at the end not being escaped
Most of our completion scripts are written using the short options
anyhow, and this makes it less likely the output will span several
lines per command
This sequence can be generatd by control-spacebar. Allow it to be bound
properly.
To do this we must be sure that we never round-trip the key sequence
through a C string.
Fish completes parts of words split by the separators, so things like
`dd if=/dev/sd<TAB>` work.
This commit improves interactive completion if completion strings legitimately
contain '=' or ':'. Consider this example where completion will suggest
a🅰️1 and other files in the cwd in addition to a:1
touch a:1; complete -C'ls a:'
This behavior remains unchanged, but this commit allows to quote or escape
separators, so that e.g. `ls "a:<TAB>` and `ls a\:<TAB>` successfully complete
the filename.
This also makes the completion insert those escapes automatically unless
already quoted.
So `ls a<TAB>` will give `ls a\:1`.
Both changes match bash's behavior.
Instead of warning (debug level 1), we now emit an error (debug level 0) if a known bad version of
WSL is detected. However, `FISH_NO_WSL_CHECK` can now be defined to skip both the check and the
startup message.
fish is designed to append to the history file in most cases. However
save_internal_via_appending was never returning success, so we were
always doing the slow rewrite path. Correctly return success.
Fixes#6042
It appears Gcc 4.8 doesn't get this particular expression, so we just
revert to the old `type foo = bar` style from the new `type foo{bar}`.
Fixes#6027.
Meaning empty variables, command substitutions that don't print
anything.
A switch without an argument
```fish
switch
case ...
end
```
is still a syntax error, and more than one argument is still a runtime
error.
The none-argument matches either an empty-string `case ''` or a
catch-all `case '*'`.
Fixes#5677.
Fixes#4943.
Previously when propagating explicitly separated output, we would early-out
if the buffer was empty, where empty meant contains no characters. However
it may contain one or more empty strings, in which case we should propagate
those strings.
Remove this footgun "empty" function and handle this properly.
Fixes#5987
Prior to this fix, fish would attempt to react if a local fish_complete_path
or fish_function_path were set. However this has never been very well tested
and will become impossible with concurrent execution. Always use the global
values.
Soon we will have more complicated logic around whether to call tcsetpgrp.
Prepare to centralize the logic by passing in the new term owner pgrp,
instead of having child_setup_process perform the decision.
This exitted if the cursor was at the end of the line as well (i.e. if
delete-char failed). That's a bit too eager.
Also documentation, which should have already been included.
We used to have a global notion of "is the shell interactive" but soon we
will want to have multiple independent execution threads, only some of
which may be interactive. Start tracking this data per-parser.
history now often writes to the history file asynchronously, but the history
test expects to find the text in the file immediately after running the
command. Hack a bit in history to make this test more reliable.
Prior to this diff, fish had different signal handling functions for
different signals. However it was hard to coordinate when a signal needed
to be the default handler, and when it was custom. In #5962 we overwrote
fish's custom WINCH handler with the default_handler when fish script asked
for WINCH to be handled.
Just have a single big signal handler function. That way it can never be
set to the wrong thing.
Fixes#5969
When executing a job, if the first process is fish internal, then have
fish claim the job's pgroup.
The idea here is that the terminal must be owned by a pgroup containing
the process reading from the terminal. If the first process is fish
internal (a function or builtin) then the pgroup must contain the fish
process.
This is a bit of a workaround of the behavior where the first process that
executes in a job becomes the process group leader. If there's a deferred
process, then we will execute processes out of order so the pgroup can be
wrong. Fix this by setting the process group leader explicitly as fish
when necessary.
Fixes#5855
Especially as, in this case, the documentation is quite massive.
Caught by porting string's test to littlecheck.
See #3404 - this was already supposed to be included.
25afc9b377 made this unnecessary by
having child processes wait for a signal after fork(), but this change
was later reverted. If we artificially slow down fish (e.g. with a sleep)
after the fork call, we see commands getting backgrounded by mistake.
Put back the tcsetgrp() call.
* Prevent not-yet-loaded functions from loaded when erased
Today, `functions --erase $function` does nothing if the function
hasn't been autoloaded yet.
E.g. run, in an interactive session
> functions --erase ls
> type ls
and be amazed that it still shows our default `ls --color=auto`
wrapper function.
This seems counter-intuitive - removing a function ought to remove it,
whether it had been executed before or not.
* doc/changelog
Instead of requiring a flag to enable newline trimming, invert it so the
flag (now `--no-trim-newlines`) disables newline trimming. This way our
default behavior matches that of sh's `"$(cmd)"`.
Also change newline trimming to trim all newlines instead of just one,
again to match sh's behavior.
The `string collect` subcommand behaves quite similarly in practice to
`string split0 -m 0` in that it doesn't split its output, but it also
takes an optional `--trim-newline` flag to trim a single trailing
newline off of the output.
See issue #159.
It's always a bit annoying that `*` requires quoting.
So we allow "x" as an alternative, only it needs to be followed by
whitespace to distinguish it from "0x" hexadecimal notation.
To support distinct parsers having different working directories, we need
to keep the working directory alive, and also retain a non-path reference
to it.
Because an exported universal variable must be exported in all variable
stacks, explicit invalidation is infeasible. Switch the universal variables
to a generation count.
Prior to this fix, fish would invalidate the exported variable list
whenever an exported variable changes. However we soon will not have a
single "exported variable list." If a global variable changes, it is
infeasible to find all exported variable lists and invalidate them.
Switch to a new model where we store a list of generation counts. Every
time an exported variable changes, the node gets a new generation. If the
current generation list does not match the cached one, then we know that
our exported variable list is stale.
This was undocumented, not all that useful and potentially unwanted.
In particular it means that things like
mysql -p(read)
will still keep the password in history.
Also it allows us to simply implement asking for the history deletion
term.
See #5791.
This makes the following changes:
1. Events in background threads are executed in those threads, instead of
being silently dropped
2. Blocked events are now per-parser instead of global
3. Events are posted in builtin_set instead of within the environment stack
The last one means that we no longer support event handlers for implicit
sets like (example) argv. Instead only the `set` builtin (and also `cd`)
post variable-change events.
Events from universal variable changes are still not fully rationalized.
This cuts down on the wcs2string here by ~25%.
The better solution would be to cache narrow versions of $PATH, since
we compute that over and over and over and over again, while it rarely changes.
Or we could add a full path-cache (where which command is), but that's
much harder to invalidate.
See #5905.
This sets the explicit path to the default one, which should be okay,
since the default path never changes (not even if $XDG_CONFIG_HOME
does).
Then it saves a narrow version of that, which saves most of the time
needed to `sync` in most cases.
Fixes#5905.
Note that this isn't technically *w*util, but the differences between
the functions are basically just whether they do the wcs2string
themselves or not.
This fixes a race condition in the topic monitor. A thread may decide to
enter the wait queue, but before it does the generation list changes, and
so our thread will wait forever, resulting in a hang.
It also simplifies the implementation of the topic monitor considerably;
on reflection the whole "metagen" thing isn't providing any value and we
should just compare generations directly.
In the new design, we have a lock-protected list of current generations,
along with a boolean as to whether someone is reading from the pipe. The
reader (only one at a time) is responsible for broadcasting notifications
via a condition variable.
The enum starts at 0 (defined to be!), so we can eliminate this one.
That allows us to remove a reliance on the position of
beginning_of_line, and it would trigger a "type-limits" warning.
Also leave a comment because I actually hit that.
Someone has hit the 10MiB limit (and of course it's the number of
javascript packages), and we don't handle it fantastically currently.
And even though you can't pass a variable of that size in one go, it's
plausible that someone might do it in multiple passes.
See #5267.
This widens the remaining ones that don't take a char
anywhere.
The rest either use a char _variable_ or __FUNCTION__, which from my
reading is narrow and needs to be widened manually. I've been unable
to test it, though.
See #5900.
This solves the main part of (careful linebreak)
issue #5900.
I'm betting all the errors that do use narrow IO are broken, including
a bunch of asserts.
This adds a new mechanism for logging, intended to replace debug().
The entry points are FLOG and FLOGF. FLOG can be used to log a sequence of
arguments, FLOGF is for printf-style formatted strings.
Each call to FLOG and FLOGF requires a category. If logging for a category
is not enabled, there is no effect (and arguments are not evaluated).
Categories may be enabled on the command line via the -d option.
Now that our interactive signal handlers are a strict superset of
non-interactive ones, there is no reason to "reset" signals or take action
when becoming non-interactive. Clean up how signal handlers get installed.
The signal handlers for interactive and non-interactive SIGINT were distinct
and talked to the reader. This wasn't really justified and will complicate
having multiple threads. Unify these into a single signal handler.
is_interactive_read is a suspicious flag which prevents a call to
parser_t::skip_all_blocks from a ^C signal handler. However we end
up skipping the blocks later when we exit the read loop.
This flag seems unnecessary. Bravely remove it.
When setting a variable without a specified scope, we should give priority
to an existing local or global above an existing universal variable with
the same name.
In 16fd780484 there was a regression that
made universal variables have priority.
Fixes#5883
Otherwise we'd undo the history search when you press e.g. execute,
which means you'd execute the search term.
Only `cancel` should walk it back, like it previously did hardcoded to
escape.
Fixes#5891.
These were inadvertently disabled by a bug which was introduced in
cd7e8f4103 . Fix the bug so the tests run
again.
They don't all pass yet; they regressed during the period they were
disabled.
This mainly is conceptually a bit simpler. The comment about making it
cheaper is entirely misplaced since this is quite far away from being
important.
Even expanding 1000 abbrs, it doesn't show up in the profile.
This was, under some circumstances, apparently off by one.
If a suggestion was really long, like
```fish
infocmp | string split , | string trim | string match -re . | while read -d = -l key val; test -z "$val"; and continue; string match -q '*%*' -- $val; and continue; test (string replace -ra '\e([\[\]]|\(B).*[\comJKsu]' '' -- a(tput $key)b) = ab; or echo $key $val; end > xterm
```
(I'm assuming longer than $COLUMNS), it would staircase like with a wrong wcwidth.
This reverts commit 15a5c0ed5f.
I noticed my debug output for 24bit color mode was garbled due to
this being wrong. I spent a little time trying to get the compiler
to tell us about these, but -Wformat doesn't do anything for wchar
printf functions, and __attribute__((format(printf, n, m))) will
cause an error with wchar_t's, so I gave up and decided to manually
check out every '%s' in the entire project. I found (only) one
more.
debug(0, "%s", wchars) will report warnings for incorrect
specifiers but debug(0, L"%s", wchars) is unable. Thus there may
be reason to prefer not using L"..." as an argument if all else
is equal and it's not necessary.
Allows `fish_indent -w **.fish` to restyle all fish files under the
current directory.
(This also has the sideeffect of reducing style.fish time by ~10s, as
we only need to invoke `fish_indent` once, instead of once per-file)
Blocks will soon need to be shared across parsers. Migrate the loop status
(like break or continue) from the block into the libdata. It turns out we
only ever need one, we don't need to track this per-block.
Make it an enum class.
Brace expansion with single words in it is quite useless - `HEAD@{0}`
expanding to `HEAD@0` breaks git.
So we complicate the rule slightly - if there is no variable expansion
or "," inside of braces, they are just treated as literal braces.
Note that this is technically backwards-incompatible, because
echo foo{0}
will now print `foo{0}` instead of `foo0`. However that's a
technicality because the braces were literally useless in that case.
Our tests needed to be adjusted, but that's because they are meant to
exercise this in weird ways.
I don't believe this will break any code in practice.
Fixes#5869.
Prior to this fix, a function_block stored a process_t, which was only used
when printing backtraces. Switch this to an array of arguments, and make
various other cleanups around null terminated argument arrays.
We previously checked if fish_mode_prompt existed as a function, but
that's a bad change for those who already set it to an empty function
to have a mode display elsewhere.
Updated widechar_width takes care of it.
Technically, this does ~3 comparisons more per-character (because it
checks variation selectors and such), but that shouldn't really matter.
get_current_winsize() is intended to be lazy. It does the following:
1. Gets the termsize from the kernel
2. Compares it against the current value
3. If changed, sets COLUMNS and LINES variables
Upon setting these variables, we notice that the termsize has changed
and invalidate the termsize. Thus we were doing this work multiple times
on every screen repaint.
Put back an old hack that just marked the termsize as valid at the end
of get_current_winsize().
This just sets some special characters that we use in the reader, so
it only needs to be done before the reader is set up.
Which, as it stands, is in env_init().
