Have Derive Attribute share a token tree with it's proc macros.
The goal of this PR is to stop creating a token tree for each derive proc macro.
This is done by giving the derive proc macros an id to its parent derive element.
From running the analysis stat on the rust analyzer project I did see a small memory decrease.
```
Inference: 42.80s, 362ginstr, 591mb
MIR lowering: 8.67s, 67ginstr, 291mb
Mir failed bodies: 18 (0%)
Data layouts: 85.81ms, 609minstr, 8mb
Failed data layouts: 135 (6%)
Const evaluation: 440.57ms, 5235minstr, 13mb
Failed const evals: 1 (0%)
Total: 64.16s, 552ginstr, 1731mb
```
After Change
```
Inference: 40.32s, 340ginstr, 593mb
MIR lowering: 7.95s, 62ginstr, 292mb
Mir failed bodies: 18 (0%)
Data layouts: 87.97ms, 591minstr, 8mb
Failed data layouts: 135 (6%)
Const evaluation: 433.38ms, 5226minstr, 14mb
Failed const evals: 1 (0%)
Total: 60.49s, 523ginstr, 1680mb
```
Currently this breaks the expansion for the actual derive attribute.
## TODO
- [x] Pick a better name for the function `smart_macro_arg`
internal: Compress file text using LZ4
I haven't tested properly, but this roughly looks like:
```
1246 MB
59mb 4899 FileTextQuery
1008 MB
20mb 4899 CompressedFileTextQuery
555kb 1790 FileTextQuery
```
We might want to test on something more interesting, like `bevy`.
Setup infra for handling auto trait bounds disabled due to perf problems
This patch updates some of the partially-implemented functions of `ChalkContext as RustIrDatabase`, namely `adt_datum()` and `impl_provided_for()`. With those, we can now correctly work with auto trait bounds and distinguish methods based on them.
Resolves#7856 (the second code; the first one is resolved by #13074)
**IMPORTANT**: I don't think we want to merge this until #7637 is resolved. Currently this patch introduces A LOT of unknown types and type mismtaches as shown below. This is because we cannot resolve items like `hashbrown::HashMap` in `std` modules, leading to auto trait bounds on them and their dependents unprovable.
|crate (from `rustc-perf@c52ee6` except for r-a)|e3dc5a588f07d6f1d3a0f33051d4af26190abe9e|HEAD of this branch|
|---|---|---|
|rust-analyzer @ e3dc5a588f |exprs: 417528, ??ty: 907 (0%), ?ty: 114 (0%), !ty: 1|exprs: 417528, ??ty: 1704 (0%), ?ty: 403 (0%), !ty: 20|
|ripgrep|exprs: 62120, ??ty: 2 (0%), ?ty: 0 (0%), !ty: 0|exprs: 62120, ??ty: 132 (0%), ?ty: 58 (0%), !ty: 11|
|webrender/webrender|exprs: 94355, ??ty: 49 (0%), ?ty: 16 (0%), !ty: 2|exprs: 94355, ??ty: 429 (0%), ?ty: 130 (0%), !ty: 7|
|diesel|exprs: 132591, ??ty: 401 (0%), ?ty: 5129 (3%), !ty: 31|exprs: 132591, ??ty: 401 (0%), ?ty: 5129 (3%), !ty: 31|
feat: Introduce term search to rust-analyzer
# Introduce term search to `rust-analyzer`
_I've marked this as draft as there might be some shortcomings, please point them out so I can fix them. Otherwise I think it is kind of ready as I think I'll rather introduce extra functionality in follow up PRs._
Term search (or I guess expression search for rust) is a technique to generate code by basically making the types match.
Consider the following program
```rust
fn wrap(arg: i32) -> Option<i32> {
todo!();
}
```
From the types of values in scope and constructors of `Option`, we can produce the expected result of wrapping the argument in `Option`
Dependently typed languages such as `Idris2` and `Agda` have similar tools to help with proofs, but this can be also used in everyday development as a "auto-complete".
# Demo videos
https://github.com/rust-lang/rust-analyzer/assets/19900308/7b68a1b7-7dba-4e31-9221-6c7485e77d88https://github.com/rust-lang/rust-analyzer/assets/19900308/0fae530a-aabb-4b28-af71-e19f8d3d64b2
# What does it currently do
- It works well with locals, free functions, type constructors and non-static impl methods that take items by value.
- Works with functions/methods that take shared references, but not with unique references (very conservative).
- Can handle projections to struct fields (eg. `foo.bar.baz`) but this might me more conservative than it has to be to avoid conflicting with borrow checker
- Should create only valid programs (no type / borrow checking errors). Tested with `rust-analyzer analysis-stats /path/to/ripgrep/Cargo.toml --run-term-search --validate-term-search` (basically running `cargo check` on all of the generated programs and only error seems to be due to type inference which is more of issue of testing method.
# Performace / fitness
```txt
ripgrep (latest)
Tail Expr syntactic hits: 130/1692 (7%)
Tail Exprs found: 523/1692 (30%)
Term search avg time: 9ms
Term search: 15.64s, 97ginstr, 8mb
rust-analyzer (on this branch)
Tail Expr syntactic hits: 804/13860 (5%)
Tail Exprs found: 6757/13860 (48%)
Term search avg time: 78ms
Term search: 1088.23s, 6765ginstr, 98mb
```
Highly generic code seems to blow up the search space so currently the amount of generics allowed is functions/methods is limited down to 0 (1 didn't give much improvement and 2 is already like 0.5+s search time)
# Plans for the future (not in this PR)
- ``~~Add impl methods that do not take `self` type (should be quite straight forward)~~ Done
- Be smarter (aka less restrictive) about borrow checking - this seems quite hard but since the current approach is rather naive I think some easy improvement is available.
- ``~~See if it works as a autocomplete while typing~~ Done
_Feel free to ask questions / point of shortcoming either here or on Zulip, I'll be happy to address them. I'm doing this as part of my MSc thesis so I'll be working on it till summer anyway 😄_
feat: ignored and disabled macro expansion
Supersedes #15117, I was having some conflicts after a rebase and since I didn't remember much of it I started clean instead.
The end result is pretty much the same as the linked PR, but instead of proc macro lookups, I marked the expanders that explicitly cannot be expanded and we shouldn't even attempt to do so.
## Unresolved questions
- [ ] I introduced a `DISABLED_ID` next to `DUMMY_ID` in `hir-expand`'s `ProcMacroExpander`, that is effectively exactly the same thing with slightly different semantics, dummy macros are not (yet) expanded probably due to errors, while not expanding disabled macros is part of the usual flow. I'm not sure if it's the right way to handle this, I also thought of just adding a flag instead of replacing the macro ID, so that the disabled macro can still be expanded for any reason if needed.
Abstract more over ItemTreeLoc-like structs
Allows reducing some code duplication by using functions generic over said structs. The diff isn't negative due to me adding some additional impls for completeness.
Swap Subtree::token_trees from Vec to boxed slice
Performs one of the optimizations suggested in #16325, but a little bit more. Boxed slices guarantee `shrink_to_fit` aswell as saving a pointer width as no capacity has to be stored.
Most of the diff is:
- Changing `vec![]` to `Box::new([])`
- Changing initialize -> fill into fill -> into_boxed_slice
- Working around the lack of an owned iterator or automatic iteration over a `Box<[T]>`
I would like to use my own crate, [small-fixed-array](https://lib.rs/small-fixed-array), although I understand if it isn't mature enough for this. If I'm given the go ahead, I can rework this PR to use it instead.
internal: even more `tracing`
As part of profiling completions, I added some additional spans and moved `TyBuilder::subst_for_def` closer to its usage site (the latter had a small impact on completion performance. Thanks for the tip, Lukas!)
This commit also adds `tracing` to NotificationDispatcher/RequestDispatcher,
bumps `rust-analyzer-salsa` to 0.17.0-pre.6, `always-assert` to 0.2, and
removes the homegrown `hprof` implementation in favor of a vendored
tracing-span-tree.
feat: Support for GOTO def from *inside* files included with include! macro
close#14937
Try to implement goto def from *inside* files included with include! macro.
This implementation has two limitations:
1. Only **one** file which calls include! will be tracked. (I think multiple file be included is a rare case and we may let it go for now)
2. Mapping token from included file to macro call file (semantics.rs:646~658) works fine but I am not sure is this the correct way to implement.
Expand lint tables && make clippy happy 🎉
This PR expands the lint tables on `./Cargo.toml` and thereby makes `cargo clippy` exit successfully! 🎉Fixes#15918
## How?
In the beginning there are some warnings for rustc.
Next, and most importantly, there is the clippy lint table. There are a few sections in there.
First there are the lint groups.
Second there are all lints which are permanently allowed with the reasoning why they are allowed.
Third there is a huge list of temporarily allowed lints. They should be removed in the mid-term, but incur a substantial amount of work, therefore they are allowed for now and can be worked on bit by bit.
Fourth there are all lints which should warn.
Additionally there are a few allow statements in the code for lints which should be permanently allowed in this specific place, but not in the whole code base.
## Follow up work
- [ ] Run clippy in CI
- [ ] Remove tidy test (at least `@Veykril` wrote this in #15017)
- [ ] Work on temporarily allowed lints
fix: Acknowledge `pub(crate)` imports in import suggestions
rust-analyzer has logic that discounts suggesting `use`s for private imports, but that logic is unnecessarily strict - for instance given this code:
```rust
mod foo {
pub struct Foo;
}
pub(crate) use self::foo::*;
mod bar {
fn main() {
Foo$0;
}
}
```
... RA will suggest to add `use crate::foo::Foo;`, which not only makes the code overly verbose (especially in larger code bases), but also is disjoint with what rustc itself suggests.
This commit adjusts the logic, so that `pub(crate)` imports are taken into account when generating the suggestions; considering rustc's behavior, I think this change doesn't warrant any extra configuration flag.
Note that this is my first commit to RA, so I guess the approach taken here might be suboptimal - certainly feels somewhat hacky, maybe there's some better way of finding out the optimal import path 😅
rust-analyzer has logic that discounts suggesting `use`s for private
imports, but that logic is unnecessarily strict - for instance given
this code:
```rust
mod foo {
pub struct Foo;
}
pub(crate) use self::foo::*;
mod bar {
fn main() {
Foo$0;
}
}
```
... RA will suggest to add `use crate::foo::Foo;`, which not only makes
the code overly verbose (especially in larger code bases), but also is
disjoint with what rustc itself suggests.
This commit adjusts the logic, so that `pub(crate)` imports are taken
into account when generating the suggestions; considering rustc's
behavior, I think this change doesn't warrant any extra configuration
flag.
Note that this is my first commit to RA, so I guess the approach taken
here might be suboptimal - certainly feels somewhat hacky, maybe there's
some better way of finding out the optimal import path 😅
fix: try obligation of `IndexMut` when infer
Closes#15842.
This issue arises because `K` is ambiguous if only inferred from `Index` trait, but is unique if inferred from `IndexMut`, but r-a doesn't use this info.
fix: Correctly set and mark the proc-macro spans
This slows down analysis by 2-3s on self for me unfortunately (~2.5% slowdown)
Noisy diff due to two simple refactoring in the first 2 commits. Relevant changes are [7d762d1](7d762d18ed) and [1e1113c](1e1113cf5f) which introduce def site spans and correct marking for proc-macros respectively.
fix: Update metavariable expression implementation
Fixes https://github.com/rust-lang/rust-analyzer/issues/16154
This duplicates behavior of that before and after PR https://github.com/rust-lang/rust/pull/117050 based on the toolchain version. There are some 1.76 nightlies that are still broken (any before that PR basically) but fetching and storing the commit makes little sense to me (opposed to the toolchain version).
fix(mbe): desugar doc correctly for mbe
Fixes#16110.
The way rust desugars doc comments when expanding macros is rendering it as raw strings delimited with hashes. Rust-analyzer wasn't aware of this, so the desugared doc comments wouldn't match correctly when on the LHS of macro declarations.
This PR fixes this by porting the code used by rustc:
59096cdad0/compiler/rustc_ast/src/tokenstream.rs (L662-L671)
Fixes#16110.
The way rust desugars doc comments when expanding macros
is rendering it as raw strings delimited with hashes.
Rust-analyzer wasn't aware of this, so the desugared doc
comments wouldn't match correctly when on the LHS of macro
declarations.
This PR fixes this by porting the code used by rustc: 4cfdbd328b/compiler/rustc_ast/src/tokenstream.rs (L6837)
fix: resolve Self type references in delegate method assist
This PR makes the delegate method assist resolve any `Self` type references in the parameters or return type. It also works across macros such as the `uint_impl!` macro used for `saturating_mul` in the issue example.
Closes#14485
Before
Private functions have RawVisibility module, but were
missed because take_types returned None early. After resolve_visibility
returned None, Visibility::Public was set instead and private functions
ended up being offered in autocompletion.
Choosing such a function results in an immediate error diagnostic
about using a private function.
After
Pattern match of take_types that returns None and
query for Module-level visibility from the original_module
Fix#15134 - tested with a unit test and a manual end-to-end
test of building rust-analyzer from my branch and opening
the reproduction repository
REVIEW
Refactor to move scope_def_applicable and check function visibility
from a module
Please let me know what's the best way to add a unit tests to
nameres, which is where the root cause was
TokenMap -> SpanMap rewrite
Opening early so I can have an overview over the full diff more easily, still very unfinished and lots of work to be done.
The gist of what this PR does is move away from assigning IDs to tokens in arguments and expansions and instead gives the subtrees the text ranges they are sourced from (made relative to some item for incrementality). This means we now only have a single map per expension, opposed to map for expansion and arguments.
A few of the things that are not done yet (in arbitrary order):
- [x] generally clean up the current mess
- [x] proc-macros, have been completely ignored so far
- [x] syntax fixups, has been commented out for the time being needs to be rewritten on top of some marker SyntaxContextId
- [x] macro invocation syntax contexts are not properly passed around yet, so $crate hygiene does not work in all cases (but most)
- [x] builtin macros do not set spans properly, $crate basically does not work with them rn (which we use)
~~- [ ] remove all uses of dummy spans (or if that does not work, change the dummy entries for dummy spans so that tests will not silently pass due to havin a file id for the dummy file)~~
- [x] de-queryfy `macro_expand`, the sole caller of it is `parse_macro_expansion`, and both of these are lru-cached with the same limit so having it be a query is pointless
- [x] docs and more docs
- [x] fix eager macro spans and other stuff
- [x] simplify include! handling
- [x] Figure out how to undo the sudden `()` expression wrapping in expansions / alternatively prioritize getting invisible delimiters working again
- [x] Simplify InFile stuff and HirFIleId extensions
~~- [ ] span crate containing all the file ids, span stuff, ast ids. Then remove the dependency injection generics from tt and mbe~~
Fixes https://github.com/rust-lang/rust-analyzer/issues/10300
Fixes https://github.com/rust-lang/rust-analyzer/issues/15685
chore: remove unused `PhantomData`
This PR removes an unused `PhantomData` in `FileItemTreeId`.
*Note:* I am not sure how this should be implemented, maybe as a type instead of a wrapper struct? I'd be happy to do so if needed 👍
