Move compile fail tests (#13196)

# Objective

- Follow-up of #13184 :)
- We use `ui_test` to test compiler errors for our custom macros.
- There are four crates related to compile fail tests
- `bevy_ecs_compile_fail_tests`, `bevy_macros_compile_fail_tests`, and
`bevy_reflect_compile_fail_tests`, which actually test the macros.
-
[`bevy_compile_test_utils`](64c1c65783/crates/bevy_compile_test_utils),
which provides helpers and common patterns for these tests.
- All of these crates reside within the `crates` directory.
- This can be confusing, especially for newcomers. All of the other
folders in `crates` are actual published libraries, except for these 4.

## Solution

- Move all compile fail tests to a `compile_fail` folder under their
corresponding crate.
- E.g. `crates/bevy_ecs_compile_fail_tests` would be moved to
`crates/bevy_ecs/compile_fail`.
- Move `bevy_compile_test_utils` to `tools/compile_fail_utils`.

There are a few benefits to this approach:

1. An internal testing detail is less intrusive (and confusing) for
those who just want to browse the public Bevy interface.
2. Follows a pre-existing approach of organizing related crates inside a
larger crate's folder.
   - See `bevy_gizmos/macros` for an example.
4. Makes consistent the terms `compile_test`, `compile_fail`, and
`compile_fail_test` in code. It's all just `compile_fail` now, because
we are specifically testing the error messages on compiler failures.
- To be clear it can still be referred to by these terms in comments and
speech, just the names of the crates and the CI command are now
consistent.

## Testing

Run the compile fail CI command:

```shell
cargo run -p ci -- compile-fail
```

If it still passes, then my refactor was successful.
This commit is contained in:
BD103 2024-05-03 09:35:21 -04:00 committed by GitHub
parent 777bb8cfef
commit bdb4899978
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
105 changed files with 51 additions and 48 deletions

View file

@ -15,10 +15,9 @@ rust-version = "1.77.0"
[workspace]
exclude = [
"benches",
"crates/bevy_ecs_compile_fail_tests",
"crates/bevy_macros_compile_fail_tests",
"crates/bevy_reflect_compile_fail_tests",
"crates/bevy_compile_test_utils",
"crates/bevy_derive/compile_fail",
"crates/bevy_ecs/compile_fail",
"crates/bevy_reflect/compile_fail",
]
members = [
"crates/*",

View file

@ -0,0 +1 @@
/target

View file

@ -1,5 +1,5 @@
[package]
name = "bevy_macros_compile_fail_tests"
name = "bevy_derive_compile_fail"
edition = "2021"
description = "Compile fail tests for Bevy Engine's various macros"
homepage = "https://bevyengine.org"
@ -9,10 +9,10 @@ publish = false
[dependencies]
# ui_test dies if we don't specify the version. See oli-obk/ui_test#211
bevy_derive = { path = "../bevy_derive", version = "0.14.0-dev" }
bevy_derive = { path = "../", version = "0.14.0-dev" }
[dev-dependencies]
bevy_compile_test_utils = { path = "../bevy_compile_test_utils" }
compile_fail_utils = { path = "../../../tools/compile_fail_utils" }
[[test]]
name = "derive"

View file

@ -3,6 +3,6 @@
This crate is not part of the Bevy workspace in order to not fail `crater` tests for Bevy.
The tests assert on the exact compiler errors and can easily fail for new Rust versions due to updated compiler errors (e.g. changes in spans).
The `CI` workflow executes these tests on the stable rust toolchain (see [tools/ci](../../tools/ci/src/main.rs)).
The `CI` workflow executes these tests on the stable rust toolchain (see [tools/ci](../../../tools/ci/src/main.rs)).
For information on writing tests see [bevy_compile_test_utils/README.md](../bevy_compile_test_utils/README.md).
For information on writing tests see [compile_fail_utils/README.md](../../../tools/compile_fail_utils/README.md).

View file

@ -0,0 +1,4 @@
fn main() -> compile_fail_utils::ui_test::Result<()> {
compile_fail_utils::test_multiple(["tests/deref_derive", "tests/deref_mut_derive"])
}

View file

@ -0,0 +1 @@
/target

View file

@ -1,5 +1,5 @@
[package]
name = "bevy_ecs_compile_fail_tests"
name = "bevy_ecs_compile_fail"
edition = "2021"
description = "Compile fail tests for Bevy Engine's entity component system"
homepage = "https://bevyengine.org"
@ -9,10 +9,10 @@ publish = false
[dependencies]
# ui_test dies if we don't specify the version. See oli-obk/ui_test#211
bevy_ecs = { path = "../bevy_ecs", version = "0.14.0-dev" }
bevy_ecs = { path = "../", version = "0.14.0-dev" }
[dev-dependencies]
bevy_compile_test_utils = { path = "../bevy_compile_test_utils" }
compile_fail_utils = { path = "../../../tools/compile_fail_utils" }
[[test]]
name = "ui"

View file

@ -2,6 +2,6 @@
This crate is separate from `bevy_ecs` and not part of the Bevy workspace in order to not fail `crater` tests for Bevy. The tests assert on the exact compiler errors and can easily fail for new Rust versions due to updated compiler errors (e.g. changes in spans).
The `CI` workflow executes these tests on the stable rust toolchain (see [tools/ci](../../tools/ci/src/main.rs)).
The `CI` workflow executes these tests on the stable rust toolchain (see [tools/ci](../../../tools/ci/src/main.rs)).
For information on writing tests see [bevy_compile_test_utils/README.md](../bevy_compile_test_utils/README.md).
For information on writing tests see [compile_fail_utils/README.md](../../../tools/compile_fail_utils/README.md).

View file

@ -0,0 +1,3 @@
fn main() -> compile_fail_utils::ui_test::Result<()> {
compile_fail_utils::test("tests/ui")
}

View file

@ -1,3 +0,0 @@
fn main() -> bevy_compile_test_utils::ui_test::Result<()> {
bevy_compile_test_utils::test("tests/ui")
}

View file

@ -1,4 +0,0 @@
fn main() -> bevy_compile_test_utils::ui_test::Result<()> {
bevy_compile_test_utils::test_multiple(["tests/deref_derive", "tests/deref_mut_derive"])
}

View file

@ -0,0 +1 @@
/target

View file

@ -1,5 +1,5 @@
[package]
name = "bevy_reflect_compile_fail_tests"
name = "bevy_reflect_compile_fail"
edition = "2021"
description = "Compile fail tests for Bevy Engine's reflection system"
homepage = "https://bevyengine.org"
@ -9,10 +9,10 @@ publish = false
[dependencies]
# ui_test dies if we don't specify the version. See oli-obk/ui_test#211
bevy_reflect = { path = "../bevy_reflect", version = "0.14.0-dev" }
bevy_reflect = { path = "../", version = "0.14.0-dev" }
[dev-dependencies]
bevy_compile_test_utils = { path = "../bevy_compile_test_utils" }
compile_fail_utils = { path = "../../../tools/compile_fail_utils" }
[[test]]
name = "derive"

View file

@ -4,6 +4,6 @@ This crate is separate from `bevy_reflect` and not part of the Bevy workspace in
Bevy.
The tests assert on the exact compiler errors and can easily fail for new Rust versions due to updated compiler errors (e.g. changes in spans).
The `CI` workflow executes these tests on the stable rust toolchain (see [tools/ci](../../tools/ci/src/main.rs)).
The `CI` workflow executes these tests on the stable rust toolchain (see [tools/ci](../../../tools/ci/src/main.rs)).
For information on writing tests see [bevy_compile_test_utils/README.md](../bevy_compile_test_utils/README.md).
For information on writing tests see [compile_fail_utils/README.md](../../../tools/compile_fail_utils/README.md).

View file

@ -0,0 +1,3 @@
fn main() -> compile_fail_utils::ui_test::Result<()> {
compile_fail_utils::test("tests/reflect_derive")
}

View file

@ -1,3 +0,0 @@
fn main() -> bevy_compile_test_utils::ui_test::Result<()> {
bevy_compile_test_utils::test("tests/reflect_derive")
}

View file

@ -16,15 +16,26 @@ impl Prepare for CompileFailCommand {
let mut commands = vec![];
// Macro Compile Fail Tests
// Run tests (they do not get executed with the workspace tests)
// - See crates/bevy_macros_compile_fail_tests/README.md
commands.push(
PreparedCommand::new::<Self>(
cmd!(sh, "cargo test --target-dir ../../../target {no_fail_fast}"),
"Compiler errors of the macros compile fail tests seem to be different than expected! Check locally and compare rust versions.",
)
.with_subdir("crates/bevy_derive/compile_fail"),
);
// ECS Compile Fail Tests
// Run UI tests (they do not get executed with the workspace tests)
// - See crates/bevy_ecs_compile_fail_tests/README.md
commands.push(
PreparedCommand::new::<Self>(
cmd!(sh, "cargo test --target-dir ../../target {no_fail_fast}"),
cmd!(sh, "cargo test --target-dir ../../../target {no_fail_fast}"),
"Compiler errors of the ECS compile fail tests seem to be different than expected! Check locally and compare rust versions.",
)
.with_subdir("crates/bevy_ecs_compile_fail_tests"),
.with_subdir("crates/bevy_ecs/compile_fail"),
);
// Reflect Compile Fail Tests
@ -32,21 +43,10 @@ impl Prepare for CompileFailCommand {
// - See crates/bevy_reflect_compile_fail_tests/README.md
commands.push(
PreparedCommand::new::<Self>(
cmd!(sh, "cargo test --target-dir ../../target {no_fail_fast}"),
cmd!(sh, "cargo test --target-dir ../../../target {no_fail_fast}"),
"Compiler errors of the Reflect compile fail tests seem to be different than expected! Check locally and compare rust versions.",
)
.with_subdir("crates/bevy_reflect_compile_fail_tests"),
);
// Macro Compile Fail Tests
// Run tests (they do not get executed with the workspace tests)
// - See crates/bevy_macros_compile_fail_tests/README.md
commands.push(
PreparedCommand::new::<Self>(
cmd!(sh, "cargo test --target-dir ../../target {no_fail_fast}"),
"Compiler errors of the macros compile fail tests seem to be different than expected! Check locally and compare rust versions.",
)
.with_subdir("crates/bevy_macros_compile_fail_tests"),
.with_subdir("crates/bevy_reflect/compile_fail"),
);
commands

View file

@ -1,5 +1,5 @@
[package]
name = "bevy_compile_test_utils"
name = "compile_fail_utils"
edition = "2021"
description = "Utils for compile tests used in the engine"
homepage = "https://bevyengine.org"

View file

@ -29,12 +29,13 @@ An example of an error annotation would be `//~v ERROR: missing trait`. This err
This will be a rather involved process. You'll have to:
- Create an empty library crate in the [`crates`](..) directory.
- Add this crate as a `dev-dependency`.
- Create a subdirectory named `compile_fail` within the crate you are testing. (E.g. `bevy_ecs/compile_fail` for `bevy_ecs`.)
- Add `compile_fail_utils` as a `dev-dependency`.
- Create a folder called `tests` within the new crate.
- Add a test runner file to this folder. The file should contain a main function calling one of the test functions defined in this crate.
- Add a `[[test]]` table to the `Cargo.toml`. This table will need to contain `harness = false` and `name = <name of the test runner file you defined>`.
- Modify the [`Ci`](../../tools/ci/) tool to run `cargo test` on this crate.
- Add the path of the new crate under `[workspace].exclude` in the root [`Cargo.toml`](../../Cargo.toml).
- Modify the [`CI`](../../tools/ci/) tool to run `cargo test` on this crate.
- And finally, write your compile tests.
If you have any questions, don't be scared to ask for help.

Some files were not shown because too many files have changed in this diff Show more