1238: Macro queries r=edwin0cheng a=matklad
In https://github.com/rust-analyzer/rust-analyzer/pull/1231, I've added aggressive clean up of `ast_id_to_node` query.
The result of this query is a `SyntaxTree`, and we don't want to retain syntax trees in memory unless absolutely necessary.
Moreover, `SyntaxTree` has identity equality semantics, meaning that we'll get a diffferent syntax tree for a file after every reparse. That means that `ast_id_to_node` query should not genereally be used in HIR, unless it is behind some kind of salsa firewall, like the `raw` module of name resoulution.
However, that PR resulted in the abysmal performance: turns out we were using ast_id_to_node quite heavily in hir when expanding macros!
So this PR installs the more incremental-friendly query structure:
* converting source to token tree is now a query; changing source without affecting token-trees will now preserve macro expansions
* expand macro (tt -> tt) is now a query as well, so we cache macro expansions *before* parsing them into item lists or expressions, which is nice: we can cache expansion without knowing the calling context!
r? @edwin0cheng
Co-authored-by: Aleksey Kladov <aleksey.kladov@gmail.com>
Currently, when expanding macros, we look at the source code
directly (we invoke ast_id_to_node query via to_node method).
This is less then ideal, because it make us re-expand macros after
every source change.
This commit establishes a salsa-firewall: a query to get macro call's
token tree. Unlike the syntax tree, token tree changes only if we
actually modify the macro itself.
1237: Improve $ Handling in mbe parser r=matklad a=edwin0cheng
This PR improve the $ handling in mbe parser. In some rare case, the `$` may not be following an `ident` or a `Subtree`. ( For example, a macro_rules inside a macro rules).
Related issue: #1236
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1230: Desugar doc comments to `#[doc = "...."]` attributes in `syntax node` to tt conversion r=matklad a=edwin0cheng
As discussed in [Zulip](https://rust-lang.zulipchat.com/#narrow/stream/185405-t-compiler.2Fwg-rls-2.2E0/topic/MBE.20discussion/near/164446835), this PR desugar doc comments to `#[doc = "...."]` in `syntax node` to tt conversion.
Note that after this PR, all obvious mbe bugs in dogfooding are fixed. (i.e. No parsing or expanding mbe error in `env RUST_LOG=ra_hir=WARN target\release\ra_cli.exe analysis-stats`) 🎉
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1229: Mark unused mbe variable as `Binding::Empty` r=matklad a=edwin0cheng
This PR fixes a regression bug in In #1228, which incorrect expand an empty binding :
* Introduce a new Binding type `Binding::Empty`.
* Mark all unused binding variables are empty and error out in expansion instead of just by passing.
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1228: Add inner macro checker in mbe r=matklad a=edwin0cheng
This PR do the following things:
* Add an inner macro checker for allowing defining mbe in mbe. (It is a adhoc solution, we could eliminate it after we have a better mbe parser)
* Move all tests to an tests modules
* Filter empty tree while expanding mbe.
This is the final PR extracting from #1219
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1227: Add `default_type` method in `TypeParam` Node r=matklad a=edwin0cheng
This PR add a `default_type` method in `TypeParam` Node which allow future PR to handle #1099 case.
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1226: Fix Bug in mbe on mismatching bindings count r=matklad a=edwin0cheng
This PR do the followings things:
1. Fixed a bug happened in following code by inserting empty bindings before a nested bindings:
```
macro_rules foo! {
($( ($($a:ident)* => $($b:ident))*))* => { ... }
}
foo!( ( => a) (b => ) )
```
2. Fixed a bug which forget to restore `var_expanded`.
3. Some cleanup
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1224: Remove unused multchar puncts code and add space between puncts r=matklad a=edwin0cheng
After #1213 , parser only need single char punct, this PR do the following things:
* Remove code which handles multi char puncts
* Remove code which handle traversal backward in `SubtreeSource` , because we cached the result in #1195
* Add space between two consecutive puncts while `tt` to `SyntaxNode` conversion .
Note that the spaces should only be added if both puncts are not delimiters.
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1222: Skip Dollars when bump raw token r=matklad a=edwin0cheng
This PR fixed a bug while parsing token_tree, it should skip all L_DOLLAR AND R_DOLLAR.
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1220: Add macro pat parsing r=matklad a=edwin0cheng
This PR add support to parsing macro call in pattern , e.g :
```
let m!(x) = 0;
```
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1217: Sidestep two-phase borrow violation r=matklad a=pnkfelix
Sidestep two-phase borrow violation signaled by mutable_borrow_reservation_conflict.
See rust-lang/rust#59159 for further information/discussion.
Co-authored-by: Felix S. Klock II <pnkfelix@pnkfx.org>
1213: Make lexer produce only single character puncts r=matklad a=edwin0cheng
As discussed in Zulip, this PR change `lexer` to produce only single char punct.
* Remove producing `DOTDOTDOT, DOTDOTEQ, DOTDOT, COLONCOLON, EQEQ, FAT_ARROW, NEQ, THIN_ARROW` in lexer.
* Add required code in parser to make sure everythings works fine.
* Change some tests (Mainly because the `ast::token_tree` is different)
Note: i think the use of `COLON` in rust is too overloaded :)
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
