Commit graph

96 commits

Author SHA1 Message Date
Mahmoud Al-Qudsi
8e97fcb22c Make output_stream_t::append() fallible
Allow errors encountered by certain implementations of `output_stream_t` when
writing to the output sink to be bubbled back to the caller.
2022-10-16 15:38:11 -05:00
Aaron Gyes
14d2a6d8ff IWYU-guided #include rejiggering.
Let's hope this doesn't causes build failures for e.g. musl: I just
know it's good on macOS and our Linux CI.

It's been a long time.

One fix this brings, is I discovered we #include assert.h or cassert
in a lot of places. If those ever happen to be in a file that doesn't
include common.h, or we are before common.h gets included, we're
unawaringly working with the system 'assert' macro again, which
may get disabled for debug builds or at least has different
behavior on crash. We undef 'assert' and redefine it in common.h.

Those were all eliminated, except in one catch-22 spot for
maybe.h: it can't include common.h. A fix might be to
make a fish_assert.h that *usually* common.h exports.
2022-08-20 23:55:18 -07:00
ridiculousfish
1127d7d68f clang-format C++ files
No functional change (hopefully!)
2022-06-01 10:02:09 -07:00
Fabian Homborg
e429f76e9f append_with_separation: Default to wanting a newline
The recent change to skip the newline for `string` changed this, and
it also hit builtin path (which is in development separately, so it's
not like it broke master).

Let's pick a good default here.
2022-05-29 17:48:11 +02:00
Aaron Gyes
b514ec5fe6 append_narrow_buffer takes const reference 2022-04-07 09:25:16 -07:00
Johannes Altmanninger
745129e825 builtin string: don't print final newline if it's missing from stdin
A command like "printf nonewline | sed s/x/y/" does not print a
concluding newline, whereas "printf nnl | string replace x y" does.
This is an edge case -- usually the user input does have a newline at
the end -- but it seems still better for this command to just forward
the user's data.

Teach most string subcommands to check if stdin is missing the trailing
newline, and stop adding one in that case.
This does not apply when input is read from commandline arguments.

* Most subcommands stop adding the final newline, because they don't
  really care about newlines, so besides their normal processing,
  they just want to preserve user input. They are:
  * string collect
  * string escape/unescape
  * string join¹
  * string lower/upper
  * string pad
  * string replace
  * string repeat
  * string sub
  * string trim

* string match keeps adding the newline, following "grep". Additionally,
  for string match --regex, it's important to output capture groups
  separated by newlines, resulting in multiple output lines for an
  input line. So it is not obvious where to leave out the newline.

* string split/split0 keep adding the newline for the same reason --
  they are meant to output multiple elements for a single input line.

¹) string join0 is not changed because it already printed a trailing
   zero byte instead of the trailing newline. This is consistent
   with other tools like "find -print0".

Closes #3847
2021-11-27 19:11:24 +01:00
Rosen Penev
ffa3e0b4f4 convert const ref to value
clang-tidy wrongly sees an std::move to a const ref parameter and
believes it to be pointless. The copy constructor however is deleted.

Signed-off-by: Rosen Penev <rosenp@gmail.com>
2021-08-20 01:16:24 +02:00
ridiculousfish
5f7e03ccf4 Introduce noncopyable_t and nonmovable_t
These are little helper types that allow us to get rid of lots of
'=delete' declarations.
2021-07-23 11:19:42 -07:00
ridiculousfish
04535e9701 Fix a few mild warnings with gcc 4.8 2021-05-10 16:49:01 -07:00
ridiculousfish
0a559ac457 Reformat source files with clang-format 2021-04-21 13:31:58 -07:00
ridiculousfish
36ad116b34 Properly report errors when builtin output fails
This correctly sets $status when a builtin succeeds but its output fails;
for example if the output is redirected to a file and that write fails.

Fixes #7857
2021-04-03 16:11:25 -07:00
ridiculousfish
84d59accfc builtins to allow stdin to be closed
Prior to this fix, if stdin were explicitly closed, then builtins would
silently fail. For example:

    count <&-

would just fail with status 1. Remove this limitation and allow each
builtin to handle a closed stdin how it sees fit.
2021-02-10 17:43:10 -08:00
ridiculousfish
98b0ef532f io_buffer_t to store a promise, not a future, to satisfy TSan
io_buffer_t is a buffer that fills itself by reading from a file
descriptor (typically a pipe). When the file descriptor is widowed, the
operation completes, and it reports completion by marking a
`std::promise<void>`. The "main thread" waits for this by waiting on the
promise's future. However TSan was reporting that the future's destructor
races with its promise's wait method. It's not obvious if this is valid,
but we can fix it by keeping the promise alive until the io_buffer_t is
deallocated.

This fixes the TSan issues reported under
`complete_background_fillthread_and_take_buffer` for #7681 (but there
are other unresolved issues).
2021-02-06 13:28:01 -08:00
ridiculousfish
b5716e97cc Remove fd_set_t
Now that we no longer need to worry about pipes conflicting with
user-specified redirections, we can remove fd_set_t.
2021-02-05 18:14:50 -08:00
ridiculousfish
6588cf35f4 Move autoclose_pipes_t from io.h to fds.h 2021-02-05 17:58:08 -08:00
ridiculousfish
be9375e914 Migrate autoclose_fd_t to new file fds.h
fds.h will centralize logic around working with file descriptors. In
particular it will be the new home for logic around moving fds to high
unused values, replacing the "avoid conflicts" logic.
2021-02-05 17:58:08 -08:00
ridiculousfish
97bde2f2bf Further refactoring of io_buffer_t
Previously we sometimes wanted to access an io_buffer_t to append to it
directly, but that's no longer true; all we really care about is its
separated_buffer_t. Make io_bufferfill_t::finish return the
separated_buffer directly, simplifying call sites. No user visible changes
expected here.
2021-02-04 17:14:46 -08:00
ridiculousfish
258149fe2e Improve locking discipline in io_buffer_t
Previously we had a lock that was taken in an ad-hoc manner. Switch to
using owning_lock.
2021-02-04 17:03:54 -08:00
ridiculousfish
8bcc8c1a36 Further cleanup of separated_buffer_t and io_buffer_t
Remove some clinging tendrils of life as a template object.
2021-02-04 16:43:47 -08:00
ridiculousfish
cbf10971f0 Reorganize separated_buffer_t
Move private bits to the bottom and do some other mild cleanup.
2021-02-04 16:06:28 -08:00
ridiculousfish
d578f8d136 separated_buffer_t to accept strings by rvalue reference
This saves a copy in some cases.
2021-02-04 16:02:40 -08:00
ridiculousfish
032467f338 separated_buffer_t to stop being a template
Now that we no longer construct wide separated buffers, it doesn't have
to be templatized.
2021-02-04 15:32:11 -08:00
ridiculousfish
7d494eab5c builtins to write to buffers directly
This concerns builtins writing to an io_buffer_t. io_buffer_t is how fish
captures output, especially in command substitutions:

    set STUFF (string upper stuff)

Recall that io_buffer_t fills itself by reading from an fd (typically
connected to stdout of the command). However if our command is a builtin,
then we can write to the buffer directly.

Prior to this change, when a builtin anticipated writing to an
io_buffer_t, it would first write into an internal buffer, and then after
the builtin was finished, we would copy it to the io_buffer_t. This was
because we didn't have a polymorphic receiver for builtin output: we
always buffered it and then directed it to the io_buffer_t or file
descriptor or stdout or whatever.

Now that we have polymorphpic io_streams_t, we can notice ahead of time
that the builtin output is destined for an internal buffer and have it
just write directly to that buffer. This saves a buffering step, which is
a nice simplification.
2021-02-04 15:21:32 -08:00
ridiculousfish
cd9a035f02 Add a string_output_stream_t to collect builtin output
This is used when creating a function; this breaks a dependency on the
more complicated buffered_output_stream_t to ease refactoring.
2021-02-04 14:12:14 -08:00
ridiculousfish
86a12e1abd separated_buffer_t::append to stop being a template
In preparation for simplifying how builtins write to buffers, make
append an ordinary function rather than a template function.
2021-02-04 13:19:11 -08:00
ridiculousfish
d5d09c993e io_buffer_t to explicitly poke its item when closing
io_buffer_t is used to buffer output from a command substitution, so we
can split it into arguments. Typically io_buffer_t reads from its pipe
until it gets EOF and then stops reading. However it may be that the
cmdsub ends but EOF is not delivered because the stdout of the cmdsub
escaped with a background process.

Prior to this change we would wake up every 100 msec (select timeout) to
check if the cmdsub is finished. However this 100 msec adds latency if a
background process is launched from e.g. fish_prompt.

Switch to the new poke() function. Now when the cmdsub is finished, it
pokes its item, which explicitly wakes it up. This removes the extra
latency.

Fixes #7559
2021-01-07 11:54:31 -08:00
ridiculousfish
38a30d1798 Mark subclasses of io_data_t as final 2020-12-19 20:06:36 -08:00
ridiculousfish
d1dab22691 Ensure we don't leak half of a pipe
It was possible though unlikely for make_autoclose_pipes to close only
one side of pipe, if it fails to find a new fd. This would result in an
fd leak. Ensure that doesn't happen.
2020-09-05 13:24:26 -07:00
ridiculousfish
81a39be0bb Support explicitly separated output on stderr
In principle this would allow 'string split' or whatever to output to
stderr and not lose the item separation. In practice this is not used
but it fixes a TODO.
2020-07-30 23:00:34 -07:00
ridiculousfish
bcfc54fdaa Do not buffer builtin output if avoidable
builtins output to stdout and stderr via io_streams_t. Prior to this fix, it
contained an output_stream_t which just wraps a buffer. So all builtin output
went to this buffer (except for eval).

Switch output_stream_t to become a new abstract class which can output to a
buffer, file descriptor, or nowhere. This allows for example `string` to stream
its output as it is produced, instead of buffering it.
2020-07-30 22:45:44 -07:00
ridiculousfish
7cc99a2d80 Rename job_tree to job_group
Initially I wanted to pick a different name to avoid confusion with
process groups, but really job trees *are* process groups. So name them
to reflect that fact.

Also rename "placeholder" to "internal" which is clearer.
2020-05-30 14:22:44 -07:00
ridiculousfish
a86d3f4136 Remove job_lineage_t
Its responsibilities are now subsumed by job_tree_t except for
the block_io which we pass around explicitly.
2020-05-30 14:22:44 -07:00
ridiculousfish
e95bcfb074 Teach a job to decide its job tree
Job trees come in two flavors: “placeholders” for jobs which are only fish
functions, and non-placeholders which need to track a pgid. This adds
logic to allow a job to decide if its parent's job tree is appropriate,
and allocating a new tree if not.
2020-05-30 14:22:43 -07:00
Mahmoud Al-Qudsi
bc756a981e Recover from bad redirections in the middle of a job pipeline
Currently fish aborts execution mid-pipeline if a file redirection
failed, which can leave the shell in a broken state (job abandoned after
giving control of the terminal to an already-executed job in the
pipeline).

This patch replaces a failed fd with a closed fd and continues execution
if the affected process wasn't the first in the pipeline.

While this is a hack to address the regression behind fish-shell/#7038
introduced in d62576c, it can also be argued that this behavior is
actually more correct... right?

Closes #7038.
2020-05-30 00:27:11 -05:00
Rosen Penev
0668513138 Change C casts to C++ ones
Some were kept for compatibility.

Found with -Wold-style-cast

Signed-off-by: Rosen Penev <rosenp@gmail.com>
2020-05-01 13:30:56 -07:00
ridiculousfish
607779257c Introduce out_is_piped and err_is_piped on io_streams_t
builtin_eval needs to know whether to set up bufferfills to capture its
output and/or errput; it should do this specifically if the output and
errput is piped (and not, say, directed to a file). In preparation for
this change, add bools to io_streams_t which track whether stdout and
stderr are specifically piped.
2020-04-28 09:59:55 -07:00
ridiculousfish
a1f1b9c2d9 builtin_eval to direct output to its iostreams
Prior to this fix, builtin_eval would direct output to the io_chain of the
job. The problem is with pipes: `builtin_eval` might happily attempt to
write unlimited output to the write end of a pipe, but the corresponding
reading process has not yet been launched. This results in deadlock.

The fix is to buffer all the output from `builtin_eval`. This is not fun
but the best that can be done until we have real concurrent processes.

Fixes #6806
2020-04-26 11:05:50 -07:00
ridiculousfish
82f2d86718 Thread pgroups into builtin_eval
Ensure that if eval is invoked as part of a pipeline, any jobs spawned
by eval will have the same pgroup as the parent job.

Partially fixes #6806
2020-04-26 11:05:50 -07:00
ridiculousfish
a765026c4c Adopt fd_monitor in bufferfill
This switches bufferfills from using an exclusively-owned thread, to
sharing an fd_monitor. This allows multiple bufferfills to all use the same
thread.
2020-02-05 12:05:39 -08:00
Fabian Homborg
3bb15defbb
Replace debug() with flog
PR #6511 

Flog has the advantage of having *categories*, not severities, so it'll be easier to get output for a certain subsystem now.
2020-01-26 14:13:17 +01:00
ridiculousfish
4f205f38b4 Clean up a few bits about discarding buffers
We weren't properly propagating the 'discarded' stuff from output
streams to buffers. Fix that.
2020-01-24 16:08:56 -08:00
Fabian Homborg
024e03ab1e Replace debug(1) with FLOGF(warning) 2020-01-19 14:22:39 +01:00
ridiculousfish
c963442999 Collapse io_data switch statements
Now that each io_data knows its source and target fd, we don't need to switch
on its types any more.
2019-12-29 15:51:22 -08:00
ridiculousfish
0af5608ce8 io_data_t to store the source_fd directly
Now that all io_data_ts know their source fd, just store it directly in
the base class. This will simplify some uses of io_data_t.
2019-12-29 15:14:08 -08:00
ridiculousfish
d0cefe8b65 Always mark pipes as cloexec
There is never a reason to keep these open in exec.
2019-12-29 14:57:16 -08:00
ridiculousfish
9f7972a08b clang-format C++ files 2019-12-29 14:25:42 -08:00
ridiculousfish
2e7cbaeaba Remove io_file_t::is_dev_null
This is no longer used.
2019-12-20 14:47:54 -08:00
ridiculousfish
0531c02ce4 Remove 'user_supplied' flag for io_fd_t
user_supplied was used to distinguish IO redirections which were
explicit, vs those that came about through "transmogrphication." But
transmogrification is no more. Remove the flag.
2019-12-19 14:14:23 -08:00
ridiculousfish
1f83fb47ce Finish the IO cleanup.
Remove some dead code and add missing dtors.
2019-12-12 17:47:08 -08:00
ridiculousfish
33aff87c10 Switch io_file_t to store an fd, not a path
Prior to this fix, a file redirection was turned into an io_file_t. This is
annoying because every place where we want to apply the redirection, we
might fail due to open() failing. Switch to opening the file at the point
we resolve the redirection spec. This will simplify a lot of code.
2019-12-12 17:34:40 -08:00