Move compile fail tests (#13196)

# Objective

- Follow-up of #13184 :)
- We use `ui_test` to test compiler errors for our custom macros.
- There are four crates related to compile fail tests
- `bevy_ecs_compile_fail_tests`, `bevy_macros_compile_fail_tests`, and
`bevy_reflect_compile_fail_tests`, which actually test the macros.
-
[`bevy_compile_test_utils`](64c1c65783/crates/bevy_compile_test_utils),
which provides helpers and common patterns for these tests.
- All of these crates reside within the `crates` directory.
- This can be confusing, especially for newcomers. All of the other
folders in `crates` are actual published libraries, except for these 4.

## Solution

- Move all compile fail tests to a `compile_fail` folder under their
corresponding crate.
- E.g. `crates/bevy_ecs_compile_fail_tests` would be moved to
`crates/bevy_ecs/compile_fail`.
- Move `bevy_compile_test_utils` to `tools/compile_fail_utils`.

There are a few benefits to this approach:

1. An internal testing detail is less intrusive (and confusing) for
those who just want to browse the public Bevy interface.
2. Follows a pre-existing approach of organizing related crates inside a
larger crate's folder.
   - See `bevy_gizmos/macros` for an example.
4. Makes consistent the terms `compile_test`, `compile_fail`, and
`compile_fail_test` in code. It's all just `compile_fail` now, because
we are specifically testing the error messages on compiler failures.
- To be clear it can still be referred to by these terms in comments and
speech, just the names of the crates and the CI command are now
consistent.

## Testing

Run the compile fail CI command:

```shell
cargo run -p ci -- compile-fail
```

If it still passes, then my refactor was successful.

Cargo.toml

@@ -15,10 +15,9 @@ rust-version = "1.77.0"
 [workspace]
 exclude = [
   "benches",
-  "crates/bevy_ecs_compile_fail_tests",
-  "crates/bevy_macros_compile_fail_tests",
-  "crates/bevy_reflect_compile_fail_tests",
-  "crates/bevy_compile_test_utils",
+  "crates/bevy_derive/compile_fail",
+  "crates/bevy_ecs/compile_fail",
+  "crates/bevy_reflect/compile_fail",
 ]
 members = [
   "crates/*",

crates/bevy_derive/compile_fail/.gitignore

@@ -0,0 +1 @@
+/target

crates/bevy_derive/compile_fail/Cargo.toml

@@ -1,5 +1,5 @@
 [package]
-name = "bevy_macros_compile_fail_tests"
+name = "bevy_derive_compile_fail"
 edition = "2021"
 description = "Compile fail tests for Bevy Engine's various macros"
 homepage = "https://bevyengine.org"
@@ -9,10 +9,10 @@ publish = false
 
 [dependencies]
 # ui_test dies if we don't specify the version. See oli-obk/ui_test#211
-bevy_derive = { path = "../bevy_derive", version = "0.14.0-dev" }
+bevy_derive = { path = "../", version = "0.14.0-dev" }
 
 [dev-dependencies]
-bevy_compile_test_utils = { path = "../bevy_compile_test_utils" }
+compile_fail_utils = { path = "../../../tools/compile_fail_utils" }
 
 [[test]]
 name = "derive"

crates/bevy_derive/compile_fail/README.md

@@ -3,6 +3,6 @@
 This crate is not part of the Bevy workspace in order to not fail `crater` tests for Bevy.
 The tests assert on the exact compiler errors and can easily fail for new Rust versions due to updated compiler errors (e.g. changes in spans).
 
-The `CI` workflow executes these tests on the stable rust toolchain (see [tools/ci](../../tools/ci/src/main.rs)).
+The `CI` workflow executes these tests on the stable rust toolchain (see [tools/ci](../../../tools/ci/src/main.rs)).
 
-For information on writing tests see [bevy_compile_test_utils/README.md](../bevy_compile_test_utils/README.md).
+For information on writing tests see [compile_fail_utils/README.md](../../../tools/compile_fail_utils/README.md).

crates/bevy_derive/compile_fail/tests/derive.rs

@@ -0,0 +1,4 @@
+fn main() -> compile_fail_utils::ui_test::Result<()> {
+    compile_fail_utils::test_multiple(["tests/deref_derive", "tests/deref_mut_derive"])
+}
+

crates/bevy_ecs/compile_fail/.gitignore

@@ -0,0 +1 @@
+/target

crates/bevy_ecs/compile_fail/Cargo.toml

@@ -1,5 +1,5 @@
 [package]
-name = "bevy_ecs_compile_fail_tests"
+name = "bevy_ecs_compile_fail"
 edition = "2021"
 description = "Compile fail tests for Bevy Engine's entity component system"
 homepage = "https://bevyengine.org"
@@ -9,10 +9,10 @@ publish = false
 
 [dependencies]
 # ui_test dies if we don't specify the version. See oli-obk/ui_test#211
-bevy_ecs = { path = "../bevy_ecs", version = "0.14.0-dev" }
+bevy_ecs = { path = "../", version = "0.14.0-dev" }
 
 [dev-dependencies]
-bevy_compile_test_utils = { path = "../bevy_compile_test_utils" }
+compile_fail_utils = { path = "../../../tools/compile_fail_utils" }
 
 [[test]]
 name = "ui"

crates/bevy_ecs/compile_fail/README.md

@@ -2,6 +2,6 @@
 
 This crate is separate from `bevy_ecs` and not part of the Bevy workspace in order to not fail `crater` tests for Bevy. The tests assert on the exact compiler errors and can easily fail for new Rust versions due to updated compiler errors (e.g. changes in spans).
 
-The `CI` workflow executes these tests on the stable rust toolchain (see [tools/ci](../../tools/ci/src/main.rs)).
+The `CI` workflow executes these tests on the stable rust toolchain (see [tools/ci](../../../tools/ci/src/main.rs)).
 
-For information on writing tests see [bevy_compile_test_utils/README.md](../bevy_compile_test_utils/README.md).
+For information on writing tests see [compile_fail_utils/README.md](../../../tools/compile_fail_utils/README.md).

crates/bevy_ecs/compile_fail/tests/ui.rs

@@ -0,0 +1,3 @@
+fn main() -> compile_fail_utils::ui_test::Result<()> {
+    compile_fail_utils::test("tests/ui")
+}

crates/bevy_ecs_compile_fail_tests/tests/ui.rs

@@ -1,3 +0,0 @@
-fn main() -> bevy_compile_test_utils::ui_test::Result<()> {
-    bevy_compile_test_utils::test("tests/ui")
-}

crates/bevy_macros_compile_fail_tests/tests/derive.rs

@@ -1,4 +0,0 @@
-fn main() -> bevy_compile_test_utils::ui_test::Result<()> {
-    bevy_compile_test_utils::test_multiple(["tests/deref_derive", "tests/deref_mut_derive"])
-}
-

crates/bevy_reflect/compile_fail/.gitignore

@@ -0,0 +1 @@
+/target

crates/bevy_reflect/compile_fail/Cargo.toml

@@ -1,5 +1,5 @@
 [package]
-name = "bevy_reflect_compile_fail_tests"
+name = "bevy_reflect_compile_fail"
 edition = "2021"
 description = "Compile fail tests for Bevy Engine's reflection system"
 homepage = "https://bevyengine.org"
@@ -9,10 +9,10 @@ publish = false
 
 [dependencies]
 # ui_test dies if we don't specify the version. See oli-obk/ui_test#211
-bevy_reflect = { path = "../bevy_reflect", version = "0.14.0-dev" }
+bevy_reflect = { path = "../", version = "0.14.0-dev" }
 
 [dev-dependencies]
-bevy_compile_test_utils = { path = "../bevy_compile_test_utils" }
+compile_fail_utils = { path = "../../../tools/compile_fail_utils" }
 
 [[test]]
 name = "derive"

crates/bevy_reflect/compile_fail/README.md

@@ -4,6 +4,6 @@ This crate is separate from `bevy_reflect` and not part of the Bevy workspace in
 Bevy.
 The tests assert on the exact compiler errors and can easily fail for new Rust versions due to updated compiler errors (e.g. changes in spans).
 
-The `CI` workflow executes these tests on the stable rust toolchain (see [tools/ci](../../tools/ci/src/main.rs)).
+The `CI` workflow executes these tests on the stable rust toolchain (see [tools/ci](../../../tools/ci/src/main.rs)).
 
-For information on writing tests see [bevy_compile_test_utils/README.md](../bevy_compile_test_utils/README.md).
+For information on writing tests see [compile_fail_utils/README.md](../../../tools/compile_fail_utils/README.md).

crates/bevy_reflect/compile_fail/tests/derive.rs

@@ -0,0 +1,3 @@
+fn main() -> compile_fail_utils::ui_test::Result<()> {
+    compile_fail_utils::test("tests/reflect_derive")
+}

crates/bevy_reflect_compile_fail_tests/tests/derive.rs

@@ -1,3 +0,0 @@
-fn main() -> bevy_compile_test_utils::ui_test::Result<()> {
-    bevy_compile_test_utils::test("tests/reflect_derive")
-}

tools/ci/src/commands/compile_fail.rs

@@ -16,15 +16,26 @@ impl Prepare for CompileFailCommand {
 
         let mut commands = vec![];
 
+        // Macro Compile Fail Tests
+        // Run tests (they do not get executed with the workspace tests)
+        // - See crates/bevy_macros_compile_fail_tests/README.md
+        commands.push(
+            PreparedCommand::new::<Self>(
+                cmd!(sh, "cargo test --target-dir ../../../target {no_fail_fast}"),
+                "Compiler errors of the macros compile fail tests seem to be different than expected! Check locally and compare rust versions.",
+            )
+            .with_subdir("crates/bevy_derive/compile_fail"),
+        );
+
         // ECS Compile Fail Tests
         // Run UI tests (they do not get executed with the workspace tests)
         // - See crates/bevy_ecs_compile_fail_tests/README.md
         commands.push(
             PreparedCommand::new::<Self>(
-                cmd!(sh, "cargo test --target-dir ../../target {no_fail_fast}"),
+                cmd!(sh, "cargo test --target-dir ../../../target {no_fail_fast}"),
                 "Compiler errors of the ECS compile fail tests seem to be different than expected! Check locally and compare rust versions.",
             )
-            .with_subdir("crates/bevy_ecs_compile_fail_tests"),
+            .with_subdir("crates/bevy_ecs/compile_fail"),
         );
 
         // Reflect Compile Fail Tests
@@ -32,21 +43,10 @@ impl Prepare for CompileFailCommand {
         // - See crates/bevy_reflect_compile_fail_tests/README.md
         commands.push(
             PreparedCommand::new::<Self>(
-                cmd!(sh, "cargo test --target-dir ../../target {no_fail_fast}"),
+                cmd!(sh, "cargo test --target-dir ../../../target {no_fail_fast}"),
                 "Compiler errors of the Reflect compile fail tests seem to be different than expected! Check locally and compare rust versions.",
             )
-            .with_subdir("crates/bevy_reflect_compile_fail_tests"),
-        );
-
-        // Macro Compile Fail Tests
-        // Run tests (they do not get executed with the workspace tests)
-        // - See crates/bevy_macros_compile_fail_tests/README.md
-        commands.push(
-            PreparedCommand::new::<Self>(
-                cmd!(sh, "cargo test --target-dir ../../target {no_fail_fast}"),
-                "Compiler errors of the macros compile fail tests seem to be different than expected! Check locally and compare rust versions.",
-            )
-            .with_subdir("crates/bevy_macros_compile_fail_tests"),
+            .with_subdir("crates/bevy_reflect/compile_fail"),
         );
 
         commands

tools/compile_fail_utils/Cargo.toml

@@ -1,5 +1,5 @@
 [package]
-name = "bevy_compile_test_utils"
+name = "compile_fail_utils"
 edition = "2021"
 description = "Utils for compile tests used in the engine"
 homepage = "https://bevyengine.org"

tools/compile_fail_utils/README.md

@@ -29,12 +29,13 @@ An example of an error annotation would be `//~v ERROR: missing trait`. This err
 
 This will be a rather involved process. You'll have to:
 
-- Create an empty library crate in the [`crates`](..) directory.
-- Add this crate as a `dev-dependency`.
+- Create a subdirectory named `compile_fail` within the crate you are testing. (E.g. `bevy_ecs/compile_fail` for `bevy_ecs`.)
+- Add `compile_fail_utils` as a `dev-dependency`.
 - Create a folder called `tests` within the new crate.
 - Add a test runner file to this folder. The file should contain a main function calling one of the test functions defined in this crate.
 - Add a `[[test]]` table to the `Cargo.toml`. This table will need to contain `harness = false` and `name = <name of the test runner file you defined>`.
-- Modify the [`Ci`](../../tools/ci/) tool to run `cargo test` on this crate.
+- Add the path of the new crate under `[workspace].exclude` in the root [`Cargo.toml`](../../Cargo.toml).
+- Modify the [`CI`](../../tools/ci/) tool to run `cargo test` on this crate.
 - And finally, write your compile tests.
 
 If you have any questions, don't be scared to ask for help.