Closes#9240.
Squash of the following commits (in reverse-chronological order):
commit 03b5cab3dc40eca9d50a9df07a8a32524338a807
Author: Mahmoud Al-Qudsi <mqudsi@neosmart.net>
Date: Sun Sep 25 15:09:04 2022 -0500
Handle differently declared posix_spawnxxx_t on macOS
On macOS, posix_spawnattr_t and posix_spawn_file_actions_t are declared as void
pointers, so we can't use maybe_t's bool operator to test if it has a value.
commit aed83b8bb308120c0f287814d108b5914593630a
Author: Mahmoud Al-Qudsi <mqudsi@neosmart.net>
Date: Sun Sep 25 14:48:46 2022 -0500
Update maybe_t tests to reflect dynamic bool conversion
maybe_t<T> is now bool-convertible only if T _isn't_ already bool-convertible.
commit 2b5a12ca97b46f96b1c6b56a41aafcbdb0dfddd6
Author: Mahmoud Al-Qudsi <mqudsi@neosmart.net>
Date: Sun Sep 25 14:34:03 2022 -0500
Make maybe_t a little harder to misuse
We've had a few bugs over the years stemming from accidental misuse of maybe_t
with bool-convertible types. This patch disables maybe_t's bool operator if the
type T is already bool convertible, forcing the (barely worth mentioning) need
to use maybe_t::has_value() instead.
This patch both removes maybe_t's bool conversion for bool-convertible types and
updates the existing codebase to use the explicit `has_value()` method in place
of existing implicit bool conversions.
This reverts commit 3e556b984c.
Revert "Further fix the issue and add the assert that'd have prevented it."
This reverts commit 056502001e.
Revert "Fix actual issue with allow_use_posix_spawn."
This reverts commit 85b9f3c71f.
Revert "Stop using posix_spawn when it is not allowed"
This reverts commit 9c896e1990.
Revert "don't even set up a fish_use_posix_spawn handler if unsupported"
This reverts commit 8b14ac4a9c.
Let's hope this doesn't causes build failures for e.g. musl: I just
know it's good on macOS and our Linux CI.
It's been a long time.
One fix this brings, is I discovered we #include assert.h or cassert
in a lot of places. If those ever happen to be in a file that doesn't
include common.h, or we are before common.h gets included, we're
unawaringly working with the system 'assert' macro again, which
may get disabled for debug builds or at least has different
behavior on crash. We undef 'assert' and redefine it in common.h.
Those were all eliminated, except in one catch-22 spot for
maybe.h: it can't include common.h. A fix might be to
make a fish_assert.h that *usually* common.h exports.
We store filenames in function definitions to indicate where the
function comes from. Previously these were intern'd strings. Switch them
to a shared_ptr<wcstring>, intending to remove intern'd strings.
This is a big cleanup to how tty transfer works. Recall that when job
control is active, we transfer the tty to jobs via tcsetpgrp().
Previously, transferring was done "as needed" in continue_job. That is, if
we are running a job, and the job wants the terminal and does not have it,
we will transfer the tty at that point.
This got pretty weird when running mixed pipelines. For example:
cmd1 | func1 | cmd2
Here we would run `func1` before calling continue_job. Thus the tty
would be transferred by the nested function invocation, and also restored
by that invocation, potentially racing with tty manipulation from cmd1 or
cmd2.
In the new model, migrate the tty transfer responsibility outside of
continue_job. The caller of continue_job is then responsible for setting up
the tty. There's two places where this gets done:
1. In `exec_job`, where we run a job for the first time.
2. In `builtin_fg` where we continue a stopped job in the foreground.
Fixes#8699
This is a cleanup of job groups, rationalizing a bunch of stuff. Some
notable changes (none user-visible hopefully):
1. Previously, if a job group wanted a pgid, then we would assign it to the
first process to run in the job group. Now we deliberately mark which
process will own the pgroup, via a new `leads_pgrp` flag in process_t. This
eliminates a source of ambiguity.
2. Previously, if a job were run inside fish's pgroup, we would set fish's
pgroup as the group of the job. But this meant we had to check if the job
had fish's pgroup in lots of places, for example when calling tcsetpgrp.
Now a job group only has a pgrp if that pgrp is external (i.e. the job is
under job control).
There is no undefined behavior in closing a moved pipe, since the
move constructor simply sets the fd to -1, which is ignored by close().
The move constructor of autoclose_fd_t is "fully specified" (like
unique_ptr).
It's good practice to eagerly close pipes which may be inherited by
child processes, since otherwise the writer may not get EPIPE correctly.
Closing the pipe explicitly makes it clear that the pipe does not stay
open across continue_job().
This reverts commit c014c23662.
If you make a script called `foo` somewhere in $PATH, and did not give
it a shebang, this would end up calling
sh foo
instead of
sh /usr/bin/foo
which might not match up.
Especially if the path is e.g. `--version` or `-` that would end up
being misinterpreted *by sh*.
So instead we simply pass the actual_cmd to sh, because we need it
anyway to get it to fail to execute before.
The previous layout confused me for a minute as it suggested it was
possible for `pipe_next_read` to be moved twice (once in the first
conditional block, then again when the deferred process conditional
called `continue` - if and only if the deferred process *was* the last
process in the job. This patch clarifies that can't be the case.
`pipe_next_read` is moved in the body of the loop, and not
re-initialized the last go around. However, we call
`pipe_next_read.close()` after the loop, which is undefined behavior (as
it's been moved).
Best case scenario, the compiler passed the address of our copy of the
struct to `exec_process_in_job` and beyond, it went out of scope there,
the value of `fd` was set to closed (minus one), and we explicitly call
`.close()` again, in which case it does nothing.
Worst case scenario, the compiler re-uses the storage for the now-moved
struct for something else and our call to `.close()` ends up closing
some other value of `fd` (valid or invalid) and things break.
Aside from the fact that we obviously don't need to close it since it's
not assigned for the last process in the job, it's a RAII object so we
don't have to worry about manually closing it in the first place.
When a job is placed in the background, fish will set the `$last_pid`
variable. Prior to this change, `$last_pid` was set to the process group
leader of the job. However this caussed problems when the job ran in
fish's process group, because then fish itself would be the process group
leader and commands like `wait` would not work.
Switch `$last_pid` to be the actual last pid of the pipeline. This brings
it in line with the `$!` variable from zsh and bash.
This is technically a breaking change, but it is unlikely to cause
problems, because `$last_pid` was already rather broken.
Fixes#5036Fixes#5832Fixes#7721
In preparation for using wait handles in --on-process-exit events, factor
wait handles into their own wait handle store. Also switch them to
per-process instead of per-job, which is a simplification.
This correctly sets $status when a builtin succeeds but its output fails;
for example if the output is redirected to a file and that write fails.
Fixes#7857
It's not super clear what $SHLVL is useful for, but the current
definition is essentially
"number of shells in the parent processes + 1"
which isn't *super useful*?
Bash's behavior here is a bit weird in that it increments $SHLVL
basically always, but since it auto-execs the last process it will
decrement it again, so in practice it's often not incremented.
E.g.
```
> echo $SHLVL
1
> bash -c 'echo $SHLVL; bash'
2
>> echo $SHLVL
2
```
Both bashes here end up having the same $SHLVL because this is
equivalent to `echo $SHLVL; exec bash`. Running `echo $SHLVL` and then
`bash -c 'echo $SHLVL'` in an interactive bash will have a different
result (1 and 2) because that doesn't *exec* the inner bash.
That's not something we want to get into, so what we do is increment
$SHLVL in every interactive fish. Non-interactive fish will simply
import the existing value.
That means if you had e.g. a bash that runs a fish script that ends up
opening a new fish session, you would have a $SHLVL of *2* - one for the
bash, and one for the inner fish.
We key this off is_interactive_session() (which can also be enabled
via `fish -i`) because it's easy and because `fish -i` is asking for
fish to be, in some form, "interactive".
That means most of the time $SHLVL will be "how many shells am I deep,
how often do I have to `exit`", except for when you specifically asked
for a fish to be "interactive". If that's a problem, we can rethink it.
Fixes#7864.
In preparation for concurrent execution, introduce a
`get_performer_for_builtin` function. This function itself returns a
function, which when called will run the builtin. The idea is that the
function may be called on a background thread (but not in this commit).
Several functions including wgetopt and execve operate on null-terminated
arrays of nul-terminated pointers: a list of pointers to C strings where
the last pointer is null. Prior to this change, each process_t stored its
argv in such an array. This had two problems:
1. It was awkward to work with this type, instead of using std::vector,
etc.
2. The process's arguments would be rearranged by builtins which is
surprising
Our null terminated arrays were built around a fancy type that would copy
input strings and also generate an array of pointers to them, in one big
allocation.
Switch to a new model where we construct an array of pointers over
existing strings. So you can supply a `vector<string>` and now
`null_terminated_array_t` will just make a list of pointers to them. Now
processes can just store their argv in a familiar wcstring_list_t.
This cleans up some exit code processing. Previously a failed exec
would produce exit code 125 unconditionally, while a failed posix_spawn
would produce exit code 1 (!).
With this change, fish reports exit code 126 for not-executable, and 127
for file-not-found. This matches bash.
This change modifies the fish safety check surrounding execve / spawn so
it can run shell scripts having concatenated binary content. We're using
the same safety check as FreeBSD /bin/sh [1] and the Z-shell [5]. POSIX
was recently revised to require this behavior:
"The input file may be of any type, but the initial portion of the
file intended to be parsed according to the shell grammar (XREF to
XSH 2.10.2 Shell Grammar Rules) shall consist of characters and
shall not contain the NUL character. The shell shall not enforce
any line length limits."
"Earlier versions of this standard required that input files to the
shell be text files except that line lengths were unlimited.
However, that was overly restrictive in relation to the fact that
shells can parse a script without a trailing newline, and in
relation to a common practice of concatenating a shell script
ending with an 'exit' or 'exec $command' with a binary data payload
to form a single-file self-extracting archive." [2] [3]
One example use case of such scripts, is the Cosmopolitan C Library [4]
which configuse the GNU Linker to output a polyglot shell+binary format
that runs on Linux / Mac / Windows / FreeBSD / OpenBSD / NetBSD / BIOS.
Fixesjart/cosmopolitan#88
[1] 9a1cd36331
[2] http://austingroupbugs.net/view.php?id=1250
[3] http://austingroupbugs.net/view.php?id=1226#c4394
[4] https://justine.lol/cosmopolitan/index.html
[5] 326d9c203b
dynamic_cast requires rtti to be enabled. Now, this isn't a big
problem, but since this is our only dynamic_cast in the entire
codebase, and it's not serving an important function, we can just
replace it.
See #7764
Prior to this fix, if stdin were explicitly closed, then builtins would
silently fail. For example:
count <&-
would just fail with status 1. Remove this limitation and allow each
builtin to handle a closed stdin how it sees fit.
Prior to this change, if you pipe a builtin to another process, it would
be buffered. With this fix the builtin will write directly to the pipe if
safe (that is, if the other end of the pipe is owned by some external
process that has been launched).
Most builtins do not produce a lot of output so this is somewhat tricky to
reproduce, but it can be done like so:
bash -c 'for i in {1..500}; do echo $i ; sleep .5; done' |
string match --regex '[02468]' |
cat
Here 'string match' is filtering out numbers which contain no even digits.
With this change, the numbers are printed as they come, instead of
buffering all the output.
Note that bcfc54fdaa fixed this for the case where the
builtin outputs to stdout directly. This fix extends it to all pipelines
that include only one fish internal process.
fish isn't quite sure what to do if the user specifies an fd redirection
for builtins. For example `source <&5` could potentially just read from
an arbitrary file descriptor internal to fish, like the history file.
fish has some lame code that tries to detect these, but got the sense
wrong. Fix it so that fd redirections for builtins are restricted to
range 0 through 2.
This concerns how fish prevents its own fds from interfering with
user-defined fd redirections, like `echo hi >&5`. fish has historically
done this by tracking all user defined redirections when running a job,
and ensuring that pipes are not assigned the same fds. However this is
annoying to pass around - it means that we have to thread user-defined
redirections into pipe creation.
Take a page from zsh and just ensure that all pipes we create have fds in
the "high range," which here means at least 10. The primary way to do this
is via the F_DUPFD_CLOEXEC syscall, which also sets CLOEXEC, so we aren't
invoking additional syscalls in the common case. This will free us from
having to track which fds are in user-defined redirections.
Previously we sometimes wanted to access an io_buffer_t to append to it
directly, but that's no longer true; all we really care about is its
separated_buffer_t. Make io_bufferfill_t::finish return the
separated_buffer directly, simplifying call sites. No user visible changes
expected here.
This concerns builtins writing to an io_buffer_t. io_buffer_t is how fish
captures output, especially in command substitutions:
set STUFF (string upper stuff)
Recall that io_buffer_t fills itself by reading from an fd (typically
connected to stdout of the command). However if our command is a builtin,
then we can write to the buffer directly.
Prior to this change, when a builtin anticipated writing to an
io_buffer_t, it would first write into an internal buffer, and then after
the builtin was finished, we would copy it to the io_buffer_t. This was
because we didn't have a polymorphic receiver for builtin output: we
always buffered it and then directed it to the io_buffer_t or file
descriptor or stdout or whatever.
Now that we have polymorphpic io_streams_t, we can notice ahead of time
that the builtin output is destined for an internal buffer and have it
just write directly to that buffer. This saves a buffering step, which is
a nice simplification.
Prior to this change, the functions in exec.cpp would return true or false
and it was not clear what significance that value had.
Switch to an enum to make this more explicit. In particular we have the
idea of a "pipeline breaking" error which should us to skip processes
which have not yet launched; if no process launches then we can bail out
to a different path which avoids reaping processes.
Fix an error caused by `exec_job()` assuming a job launched with the
intention of being backgrounded would have a pgid assigned in all cases,
without considering the status of `exec_error` which could have resulted
in the job failing before it was launched into its own process group.
Fixes (but doesn't close) #7423 - that can be closed if this assertion
failure doesn't happen in any released fish versions.
The 'time' prefix may come about either because the job itself is marked
with time, or because of the "inside out" weirdness of 'not time...'.
Factor this logic together and precompute it for a job.
