Commit graph

77 commits

Author SHA1 Message Date
ridiculousfish
896ef65f8c Rename error_offset to error_offset_within_token
Hopefully clarify the role of this variable.
2019-11-08 16:56:50 -08:00
ridiculousfish
ef8b5e4fa0 Correct certain tokenizer error reporting for unclosed subshells
There was some confusion about the different pointers and offsets
in tokenizer_t::call_error.

Fixes #6281
2019-11-08 16:56:43 -08:00
ridiculousfish
eb2386f3e3 Rename tokenizer_t::buff to token_cursor
That should clarify its role vis-a-vis 'start'.
2019-11-08 16:56:18 -08:00
ridiculousfish
eeac3333df Correctly highlight input following a tokenizer error 2019-10-27 16:08:49 -07:00
ridiculousfish
afd20b8e1a Correctly report the range of tokenizer errors
This enables proper syntax highlighting of tokenizer errors.
2019-10-27 16:05:37 -07:00
ridiculousfish
0bfd897ee0 Add a special error message for |& 2019-10-27 15:24:57 -07:00
ridiculousfish
2a92e66902 Support for &> and &| as convenience redirections
This adds support for &> and &| syntax, which both redirect stdout, and
also apply a redirection of stderr to stdout.
2019-10-27 15:24:57 -07:00
ridiculousfish
35671dd9f0 Clean up and unify pipes and redirections
This cleans up how pipes and redirections are recognized by the parser,
and unifies pipes and redirections into a single type.
2019-10-15 11:26:41 -07:00
ridiculousfish
c160492d42 Remove tok_t::redirected_fd
This wasn't used for anything.
2019-10-13 23:04:18 -07:00
ridiculousfish
1a65e18ba8 Clean up some tokenization
Remove TOK_NONE
Turn token_type into an enum class
Make next() turn a maybe_t<tok_t> instead of a bool
2019-10-13 16:06:16 -07:00
ridiculousfish
82eca4bc86 Run clang-format on all files
The main change here is to reorder headers.
2019-10-13 15:50:48 -07:00
Fabian Homborg
d73ee4d54b More using FLOGF when formatting is needed
sed-patched, every time a "%" is used in a call to `FLOG`, we use
`FLOGF` instead.
2019-05-30 11:54:09 +02:00
ridiculousfish
ea9d1ad82f Convert debug(0) calls to FLOG 2019-05-27 17:31:17 -07:00
Fabian Homborg
c2970f9618 Reformat all files
This runs build_tools/style.fish, which runs clang-format on C++, fish_indent on fish and (new) black on python.

If anything is wrong with the formatting, we should fix the tools, but automated formatting is worth it.
2019-05-05 12:09:25 +02:00
Fabian Homborg
009ecfd7e6 src/tokenizer: Add ":@" to the list of non-path-component chars
This makes kill-path-component stop there.

Fixes #5841.
2019-04-26 19:16:21 +02:00
Aaron Gyes
1834e962d2 Correct carat position for unexpected } in brace expansion
before:

$ echo {}}-
fish: Unexpected '}' for unopened brace expansion

$ ./fish -c 'echo {}}}}'
fish: Unexpected '}' for unopened brace expansion
echo {}}}}
^

now:
$ echo {}}}}}}1-
fish: Unexpected '}' for unopened brace expansion
echo {}}}}}}
       ^
2019-04-11 17:18:53 -07:00
Aaron Gyes
d837eee09d remove some wcstring -> wchar_t* -> wcstring conversions
Mostly related to usage _(L"foo"), keeping in mind the _
macro does a wcstring().c_str() already.

And a smattering of other trivial micro-optimizations certain
to not help tangibly.
2019-03-14 15:21:08 -07:00
Aaron Gyes
d5ac239f68 This commit changes wchar.h includes to cwchar, and uses std::
for everything it provides.
2019-03-12 15:09:36 -07:00
Aaron Gyes
2ae6e5a585 Explicitly handle all enum values in more switch statements
This addresses a few places where -Wswitch-enum showed one or two missing
case's for enum values.

It did uncover and fix one apparent oversight:

$ function asd -p 100
   echo foo
end

$ functions --handlers-type exit
Event exit
asd

It looks like this should be showing a PID before 'asd' just like
job_exit handlers show the job id. It was falling
through to default: which just printed the function name.

$ functions --handlers-type exit
Event exit
100 asd
2019-03-11 15:02:18 -07:00
ridiculousfish
d6a5792ce2 Allow nested square brackets again
Code like echo $list[$var[1]] was producing an error because of
nested square brackets. Allow these brackets again.

Fixes #5362
2018-11-22 17:57:27 -08:00
ridiculousfish
cc99e8d510 Switch tokenizer_error back to just an error code
Rather than having tokenizer_error as pointers to objects, switch it back
to just an error code value. This makes reasoning about it easier since
it's immutable values instead of mutable objects, and it avoids allocation
during startup.
2018-09-27 21:40:51 -04:00
ridiculousfish
f28f9792b3 Remove ENUM_FLAGS
This define added operator overloading via preprocessor macros, which is
more magic than necessary. It was only used in one place.
2018-09-27 21:25:58 -04:00
ridiculousfish
6f57fef8f8 Teach the tokenizer to report escaped newlines
Add fields and flags so that escaped newlines can be reported, for the
benefit of fish_indent.
2018-05-07 21:39:30 -07:00
ridiculousfish
4194b4efee Add a feature_test() function
This is a convenience over fish_features().test()
2018-05-06 11:20:15 -07:00
ridiculousfish
902af26253 Bring back caret redirections under a feature flag
This partially reverts 5b489ca30f, with
carets acting as redirections unless the stderr-nocaret flag is set.
This flag is off by default but may be enabled on the command line:

fish --features stderr-nocaret
2018-05-06 11:20:14 -07:00
Peter Ammon
5b489ca30f Remove caret redirection
This removes the caret as a shorthand for redirecting stderr.

Note that stderr may be redirected to a file via 2>/some/path...
and may be redirected with a pipe via 2>|.

Fixes #4394
2018-04-01 13:48:21 -07:00
Mahmoud Al-Qudsi
1441cca9c5 Restore localization to tokenizer error strings
Work around #4810 by retrieving localizations at runtime to avoid issues
possibly caused by inserting into the static unordered_map during static
initialization.

Closes #810.
2018-03-13 13:45:15 -05:00
Mahmoud Al-Qudsi
054bc88b82 Add gettext operator back to strings in tokenizer
This was purposely disabled as it causes a floating point exception immediately on run:

    Program received signal SIGFPE, Arithmetic exception.
    0x000000000056eae8 in std::__detail::_Mod_range_hashing::operator()(unsigned long, unsigned long) const ()
    (gdb) bt
    #0  0x000000000056eae8 in std::__detail::_Mod_range_hashing::operator()(unsigned long, unsigned long) const ()
    #1  0x00000000005bb4f1 in std::__detail::_Hash_code_base<std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >, std::pair<std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > const, std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > >, std::__detail::_Select1st, std::hash<std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, true>::_M_bucket_index(std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > const&, unsigned long, unsigned long) const ()
    #2  0x00000000005bb22a in std::_Hashtable<std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >, std::pair<std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > const, std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > >, std::allocator<std::pair<std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > const, std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > >, std::hash<std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_bucket_index(std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > const&, unsigned long) const ()
    #3  0x00000000005bb115 in std::__detail::_Map_base<std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >, std::pair<std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > const, std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > >, std::allocator<std::pair<std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > const, std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > >, std::hash<std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>::operator[](std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > const&) ()
    #4  0x00000000005af388 in std::unordered_map<std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >, std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >, std::hash<std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > >, std::equal_to<std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > >, std::allocator<std::pair<std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > const, std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > > > >::operator[](std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > const&) ()
    #5  0x000000000067dc6b in wgettext[abi:cxx11](wchar_t const*) ()
    #6  0x0000000000551df7 in __cxx_global_var_init.1 ()
    #7  0x000000000055225e in _GLOBAL__sub_I_tokenizer.cpp ()
    #8  0x000000000069d45d in __libc_csu_init ()
    #9  0x00007ffffdc907bf in __libc_start_main (main=0x552500 <main>, argc=1, argv=0x7ffffffed298, init=0x69d410 <__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffffffed288) at ../csu/libc-start.c:247
    #10 0x0000000000552389 in _start ()
    (gdb)
2018-03-12 12:44:11 -05:00
Mahmoud Al-Qudsi
0f6c763d9e Revert "Restore sanity to line continuations"
This reverts commit 2cdacbdce4.
Our fish scripts need to be audited for compliance before this can be
safely merged. cc @faho
2018-03-12 08:35:09 -05:00
Mahmoud Al-Qudsi
2cdacbdce4 Restore sanity to line continuations
Line continuations (i.e. escaped new lines) now make sense again. With
the smart pipe support (pipes continue on to next line) recently added,
this hack to have continuations ignore comments makes no sense.

This is valid code:

```fish
echo hello |
# comment here
tr -d 'l'
```

this isn't:
```fish
echo hello | \
# comment here
tr -d 'l'
```

Reverts @snnw's 318daaffb2
Closes #2928. Closes #2929.
2018-03-12 08:12:23 -05:00
Mahmoud Al-Qudsi
0620cdf711 Fix tokenizer errors for nested, alternating {} and () 2018-03-11 20:18:03 -05:00
Mahmoud Al-Qudsi
00f95a978e Make { and } valid, first-class tokenizer elements 2018-03-11 19:36:10 -05:00
Mahmoud Al-Qudsi
7447432471 Fix tests for new ) token error 2018-03-11 17:16:53 -05:00
Mahmoud Al-Qudsi
df89d71237 Correct escaping behavior in new tokenizer code 2018-03-11 17:10:16 -05:00
Mahmoud Al-Qudsi
f508a1f274 Reset tokenizer state on start and improve slice error detection 2018-03-11 12:13:55 -05:00
Mahmoud Al-Qudsi
fb1c7a3d68 Simplify token parser 2018-03-10 18:42:56 -06:00
ridiculousfish
8ded041352 Add && and || support to tokenizer 2018-03-05 12:20:56 -08:00
Fabian Homborg
c793570f2c Fix punctuation movement with one char tokens
Previously, in

    ls ^a bcd

(with "^" as the cursor), kill-word would delete the "a" and then go
on, remove the space and the "bcd".

With this, it will only kill the "a".

Fixes #4747.
2018-02-27 22:56:15 +01:00
ridiculousfish
c4d903ff98 Rationalize how the parser reports tokenizer errors
Remove the unnecessary SQUASH_ERROR flag and correctly report errors
generated from the tokenizer.
2018-02-23 17:28:12 -08:00
ridiculousfish
99fb7bb6aa Refactor how redirections are represented by the tokenizer
Prior to this fix, each redirection type was a separate token_type.
Unify these under a single type TOK_REDIRECT and break the redirection
type out into a new sub-type redirection_type_t.
2018-02-23 15:19:58 -08:00
ridiculousfish
6673fe5457 Clean up tokenizer implementation
Rather than storing a bunch of "next_foo" fields, simply populate the
tok_t directly.
2018-02-23 14:31:13 -08:00
ridiculousfish
0f62161b2b Clean up tokenizer error handling. 2018-02-19 16:31:39 -08:00
ridiculousfish
f30bf40300 Clean up comment parsing in tokenizer
Unify the show_comments and non-show_comments path.
2018-02-19 16:20:47 -08:00
ridiculousfish
b13ee818d2 Some early cleanup of tokenizer
Prior to this the tokenizer ran "one ahead", where tokenizer_t::next()
would in fact return the last-parsed token. Switch to parsing on demand
instead of running one ahead; this is simpler and prepares for tokenizer
changes.
2018-02-19 15:10:59 -08:00
ridiculousfish
fd2a0dffa9 Reflect newlines after pipes in fish grammar
The previous attempt to support newlines after pipes changed the lexer to
swallow newlines after encountering a pipe. This has two problems that are
difficult to fix:

1. comments cannot be placed after the pipe
2. fish_indent won't know about the newlines, so it will erase them

Address these problems by removing the lexer behavior, and replacing it
with a new parser symbol "optional_newlines" allowing the newlines to be
reflected directly in the fish grammar.
2018-02-18 14:44:08 -08:00
slama
38418d6356 Enable multi-line edit when the line ends with a pipe (#1285) 2018-02-18 13:01:38 -08:00
Kurtis Rader
509ee64fc9 implement our own assert() function
I recently upgraded the software on my macOS server and was dismayed to
see that cppcheck reported a huge number of format string errors due to
mismatches between the format string and its arguments from calls to
`assert()`. It turns out they are due to the macOS header using `%lu`
for the line number which is obviously wrong since it is using the C
preprocessor `__LINE__` symbol which evaluates to a signed int.

I also noticed that the macOS implementation writes to stdout, rather
than stderr. It also uses `printf()` which can be a problem on some
platforms if the stream is already in wide mode which is the normal case
for fish.

So implement our own `assert()` implementation. This also eliminates
double-negative warnings that we get from some of our calls to
`assert()` on some platforms by oclint.

Also reimplement the `DIE()` macro in terms of our internal
implementation.

Rewrite `assert(0 && msg)` statements to `DIE(msg)` for clarity and to
eliminate oclint warnings about constant expressions.

Fixes #3276, albeit not in the fashion I originally envisioned.
2017-02-14 18:48:27 -08:00
Kurtis Rader
4ffb0adb78 lint cleanups 2017-02-11 21:30:38 -08:00
ridiculousfish
1efb81456b Use std::move instead of swap in a few places where it improves clarity 2017-01-26 16:14:50 -08:00
Kurtis Rader
2e9a349dd0 normalize use of stdio functions taking a stream
We should never use stdio functions that use stdout implicitly. Saving a
few characters isn't worth the inconsistency. Too, using the forms such
as `fwprintf()` which take an explicit stream makes it easier to find
the places we write to stdout versus stderr.

Fixes #3728
2017-01-13 20:48:55 -08:00