This is inspired by cargo which allows you to run `cargo test --test`
and it will list the possible tests (obviously we can't support that atm
because that requires a lot of runtime processing). When we do have a
static list of possible values, we can at least show those.
Fixes#3320
This is prep for moving the derive tests. Besides organizing the test
folder for each API, this should reduce link time at the cost of
re-compiling more when a test changes.