The general theme of this is to make parser a better independent
library.
The specific thing we do here is replacing callback based TreeSink with
a data structure. That is, rather than calling user-provided tree
construction methods, the parser now spits out a very bare-bones tree,
effectively a log of a DFS traversal.
This makes the parser usable without any *specifc* tree sink, and allows
us to, eg, move tests into this crate.
Now, it's also true that this is a distinction without a difference, as
the old and the new interface are equivalent in expressiveness. Still,
this new thing seems somewhat simpler. But yeah, I admit I don't have a
suuper strong motivation here, just a hunch that this is better.
Originally we tried to maintain the invariant that `{}` always match.
That is, that in the parse tree the pair of corresponding `{}` is always
first and last tokens of some nodes.
We had the code to validate that, but apparently it's been broken for
**years** since we introduced tokens/nodes split. Fixing it now makes
some tests fail.
It's unclear if we want to keep this invariant: there's a strong
motivation for breaking it in the following case:
```
use std::{ // unclosed paren
fn main() {
}
} // don't actually want to pair up this with the one from `use`
```
So let's fix the code, but disable it for the time being
It's very useful when `pub` is equivalent to "this is crate's public
API", let's enforce this!
Ideally, we should enforce it for local `cargo test`, and only during
CI, but that needs https://github.com/rust-lang/cargo/issues/5034.
