a5c333c3ed
This commit is an example of fixing a common parser error: infinite
loop due to error recovery.
This error typically happens when we parse a list of items and fail to
parse a specific item at the current position.
One choices is to skip a token and try to parse a list item at the
next position. This is a good, but not universal, default. When
parsing a list of arguments in a function call, you, for example,
don't want to skip over `fn`, because it's most likely that it is a
function declaration, and not a mistyped arg:
```
fn foo() {
quux(1, 2
fn bar() {
}
```
Another choice is to bail out of the loop immediately, but it isn't
perfect either: sometimes skipping over garbage helps:
```
quux(1, foo:, 92) // should skip over `:`, b/c that's part of `foo::bar`
```
In general, parser tries to balance these two cases, though we don't
have a definitive strategy yet.
However, if the parser accidentally neither skips over a token, nor
breaks out of the loop, then it becomes stuck in the loop infinitely
(there's an internal counter to self-check this situation and panic
though), and that's exactly what is demonstrated by the test.
To fix such situation, first of all, add the test case to tests/data/parser/{err,fuzz-failures}.
Then, run
```
RUST_BACKTRACE=short cargo test --package libsyntax2
````
to verify that parser indeed panics, and to get an idea what grammar
production is the culprit (look for `_list` functions!).
In this case, I see
```
10: libsyntax2::grammar::expressions::atom::match_arm_list
at crates/libsyntax2/src/grammar/expressions/atom.rs:309
```
and that's look like it might be a culprit. I verify it by adding
`eprintln!("loopy {:?}", p.current());` and indeed I see that this is
printed repeatedly.
Diagnosing this a bit shows that the problem is that
`pattern::pattern` function does not consume anything if the next
token is `let`. That is a good default to make cases like
```
let
let foo = 92;
```
where the user hasn't typed the pattern yet, to parse in a reasonable
they correctly.
For match arms, pretty much the single thing we expect is a pattern,
so, for a fix, I introduce a special variant of pattern that does not
do recovery.
|
||
|---|---|---|
| .cargo | ||
| code | ||
| crates | ||
| .gitignore | ||
| .travis.yml | ||
| appveyor.yml | ||
| bors.toml | ||
| Cargo.toml | ||
| CONTRIBUTING.md | ||
| LICENSE-APACHE | ||
| LICENSE-MIT | ||
| README.md | ||
| rustfmt.toml | ||
libsyntax2.0
libsyntax2.0 is an experimental parser of the Rust language, intended for the use in IDEs. RFC.
Quick Start
$ cargo test
$ cargo parse < crates/libsyntax2/src/lib.rs
Trying It Out
This installs experimental VS Code plugin
$ cargo install-code
It's better to remove existing Rust plugins to avoid interference. Warning: plugin is not intended for general use, has a lot of rough edges and missing features (notably, no code completion). That said, while originally libsyntax2 was developed in IntelliJ, @matklad now uses this plugin (and thus, libsytax2) to develop libsyntax2, and it doesn't hurt too much :-)
Features:
-
syntax highlighting (LSP does not have API for it, so impl is hacky and sometimes fall-backs to the horrible built-in highlighting)
-
commands (
ctrl+shift+por keybindings)- Show Rust Syntax Tree (use it to verify that plugin works)
- Rust Extend Selection (works with multiple cursors)
- Rust Matching Brace (knows the difference between
<and<) - Rust Parent Module
- Rust Join Lines (deals with trailing commas)
-
Go to symbol in file
-
Go to symbol in workspace (no support for Cargo deps yet)
-
code actions:
- Flip
,in comma separated lists - Add
#[derive]to struct/enum - Add
implblock to struct/enum - Run tests at caret
- Flip
-
Go to definition ("correct" for
mod foo;decls, index-based for functions).
Code Walk-Through
crates/libsyntax2
yellow, red/green syntax tree, heavily inspired by thisgrammar, the actual parserparser_api/parser_implbridges the tree-agnostic parser fromgrammarwithyellowtreesgrammar.ronRON description of the grammar, which is used to generatesyntax_kindsandastmodules.algo: generic tree algorithms, includingwalkfor O(1) stack space tree traversal (this is cool) andvisitfor type-driven visiting the nodes (this is double plus cool, if you understand howVisitorworks, you understand libsyntax2).
crates/libeditor
Most of IDE features leave here, unlike libanalysis, libeditor is
single-file and is basically a bunch of pure functions.
crates/libanalysis
A stateful library for analyzing many Rust files as they change.
WorldState is a mutable entity (clojure's atom) which holds current
state, incorporates changes and handles out Worlds --- immutable
consistent snapshots of WorldState, which actually power analysis.
crates/server
An LSP implementation which uses libanalysis for managing state and
libeditor for actually doing useful stuff.
crates/cli
A CLI interface to libsyntax
crate/tools
Code-gen tasks, used to develop libsyntax2:
cargo gen-kinds-- generateastandsyntax_kindscargo gen-tests-- collect inline tests from grammarcargo install-code-- build and install VS Code extension and server
code
VS Code plugin
Performance
Non-incremental, but seems pretty fast:
$ cargo build --release --package cli
$ wc -l ~/projects/rust/src/libsyntax/parse/parser.rs
7546 /home/matklad/projects/rust/src/libsyntax/parse/parser.rs
$ ./target/release/cli parse < ~/projects/rust/src/libsyntax/parse/parser.rs --no-dump > /dev/null
parsing: 21.067065ms
Getting in touch
@matklad can be found at Rust discord, in #ides-and-editors.
License
libsyntax2 is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0).
See LICENSE-APACHE and LICENSE-MIT for details.