Add morph targets (#8158)
# Objective
- Add morph targets to `bevy_pbr` (closes #5756) & load them from glTF
- Supersedes #3722
- Fixes #6814
[Morph targets][1] (also known as shape interpolation, shape keys, or
blend shapes) allow animating individual vertices with fine grained
controls. This is typically used for facial expressions. By specifying
multiple poses as vertex offset, and providing a set of weight of each
pose, it is possible to define surprisingly realistic transitions
between poses. Blending between multiple poses also allow composition.
Morph targets are part of the [gltf standard][2] and are a feature of
Unity and Unreal, and babylone.js, it is only natural to implement them
in bevy.
## Solution
This implementation of morph targets uses a 3d texture where each pixel
is a component of an animated attribute. Each layer is a different
target. We use a 2d texture for each target, because the number of
attribute×components×animated vertices is expected to always exceed the
maximum pixel row size limit of webGL2. It copies fairly closely the way
skinning is implemented on the CPU side, while on the GPU side, the
shader morph target implementation is a relatively trivial detail.
We add an optional `morph_texture` to the `Mesh` struct. The
`morph_texture` is built through a method that accepts an iterator over
attribute buffers.
The `MorphWeights` component, user-accessible, controls the blend of
poses used by mesh instances (so that multiple copy of the same mesh may
have different weights), all the weights are uploaded to a uniform
buffer of 256 `f32`. We limit to 16 poses per mesh, and a total of 256
poses.
More literature:
* Old babylone.js implementation (vertex attribute-based):
https://www.eternalcoding.com/dev-log-1-morph-targets/
* Babylone.js implementation (similar to ours):
https://www.youtube.com/watch?v=LBPRmGgU0PE
* GPU gems 3:
https://developer.nvidia.com/gpugems/gpugems3/part-i-geometry/chapter-3-directx-10-blend-shapes-breaking-limits
* Development discord thread
https://discord.com/channels/691052431525675048/1083325980615114772
https://user-images.githubusercontent.com/26321040/231181046-3bca2ab2-d4d9-472e-8098-639f1871ce2e.mp4
https://github.com/bevyengine/bevy/assets/26321040/d2a0c544-0ef8-45cf-9f99-8c3792f5a258
## Acknowledgements
* Thanks to `storytold` for sponsoring the feature
* Thanks to `superdump` and `james7132` for guidance and help figuring
out stuff
## Future work
- Handling of less and more attributes (eg: animated uv, animated
arbitrary attributes)
- Dynamic pose allocation (so that zero-weighted poses aren't uploaded
to GPU for example, enables much more total poses)
- Better animation API, see #8357
----
## Changelog
- Add morph targets to bevy meshes
- Support up to 64 poses per mesh of individually up to 116508 vertices,
animation currently strictly limited to the position, normal and tangent
attributes.
- Load a morph target using `Mesh::set_morph_targets`
- Add `VisitMorphTargets` and `VisitMorphAttributes` traits to
`bevy_render`, this allows defining morph targets (a fairly complex and
nested data structure) through iterators (ie: single copy instead of
passing around buffers), see documentation of those traits for details
- Add `MorphWeights` component exported by `bevy_render`
- `MorphWeights` control mesh's morph target weights, blending between
various poses defined as morph targets.
- `MorphWeights` are directly inherited by direct children (single level
of hierarchy) of an entity. This allows controlling several mesh
primitives through a unique entity _as per GLTF spec_.
- Add `MorphTargetNames` component, naming each indices of loaded morph
targets.
- Load morph targets weights and buffers in `bevy_gltf`
- handle morph targets animations in `bevy_animation` (previously, it
was a `warn!` log)
- Add the `MorphStressTest.gltf` asset for morph targets testing, taken
from the glTF samples repo, CC0.
- Add morph target manipulation to `scene_viewer`
- Separate the animation code in `scene_viewer` from the rest of the
code, reducing `#[cfg(feature)]` noise
- Add the `morph_targets.rs` example to show off how to manipulate morph
targets, loading `MorpStressTest.gltf`
## Migration Guide
- (very specialized, unlikely to be touched by 3rd parties)
- `MeshPipeline` now has a single `mesh_layouts` field rather than
separate `mesh_layout` and `skinned_mesh_layout` fields. You should
handle all possible mesh bind group layouts in your implementation
- You should also handle properly the new `MORPH_TARGETS` shader def and
mesh pipeline key. A new function is exposed to make this easier:
`setup_moprh_and_skinning_defs`
- The `MeshBindGroup` is now `MeshBindGroups`, cached bind groups are
now accessed through the `get` method.
[1]: https://en.wikipedia.org/wiki/Morph_target_animation
[2]:
https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#morph-targets
---------
Co-authored-by: François <mockersf@gmail.com>
Co-authored-by: Carter Anderson <mcanders1@gmail.com>
2023-06-22 20:00:01 +00:00
|
|
|
//! Control animations of entities in the loaded scene.
|
|
|
|
use bevy::{gltf::Gltf, prelude::*};
|
|
|
|
|
|
|
|
use crate::scene_viewer_plugin::SceneHandle;
|
|
|
|
|
|
|
|
/// Controls animation clips for a unique entity.
|
|
|
|
#[derive(Component)]
|
|
|
|
struct Clips {
|
|
|
|
clips: Vec<Handle<AnimationClip>>,
|
|
|
|
current: usize,
|
|
|
|
}
|
|
|
|
impl Clips {
|
|
|
|
fn new(clips: Vec<Handle<AnimationClip>>) -> Self {
|
|
|
|
Clips { clips, current: 0 }
|
|
|
|
}
|
|
|
|
/// # Panics
|
|
|
|
///
|
|
|
|
/// When no clips are present.
|
|
|
|
fn current(&self) -> Handle<AnimationClip> {
|
|
|
|
self.clips[self.current].clone_weak()
|
|
|
|
}
|
|
|
|
fn advance_to_next(&mut self) {
|
|
|
|
self.current = (self.current + 1) % self.clips.len();
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/// Read [`AnimationClip`]s from the loaded [`Gltf`] and assign them to the
|
|
|
|
/// entities they control. [`AnimationClip`]s control specific entities, and
|
|
|
|
/// trying to play them on an [`AnimationPlayer`] controlling a different
|
|
|
|
/// entities will result in odd animations, we take extra care to store
|
|
|
|
/// animation clips for given entities in the [`Clips`] component we defined
|
|
|
|
/// earlier in this file.
|
|
|
|
fn assign_clips(
|
|
|
|
mut players: Query<(Entity, &mut AnimationPlayer, &Name)>,
|
|
|
|
scene_handle: Res<SceneHandle>,
|
|
|
|
clips: Res<Assets<AnimationClip>>,
|
|
|
|
gltf_assets: Res<Assets<Gltf>>,
|
|
|
|
mut commands: Commands,
|
|
|
|
mut setup: Local<bool>,
|
|
|
|
) {
|
|
|
|
if scene_handle.is_loaded && !*setup {
|
|
|
|
*setup = true;
|
|
|
|
} else {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
let gltf = gltf_assets.get(&scene_handle.gltf_handle).unwrap();
|
|
|
|
let animations = &gltf.animations;
|
|
|
|
if !animations.is_empty() {
|
|
|
|
let count = animations.len();
|
|
|
|
let plural = if count == 1 { "" } else { "s" };
|
|
|
|
info!("Found {} animation{plural}", animations.len());
|
|
|
|
let names: Vec<_> = gltf.named_animations.keys().collect();
|
|
|
|
info!("Animation names: {names:?}");
|
|
|
|
}
|
|
|
|
for (entity, mut player, name) in &mut players {
|
|
|
|
let clips = clips
|
|
|
|
.iter()
|
|
|
|
.filter_map(|(k, v)| v.compatible_with(name).then_some(k))
|
|
|
|
.map(|id| clips.get_handle(id))
|
|
|
|
.collect();
|
|
|
|
let animations = Clips::new(clips);
|
|
|
|
player.play(animations.current()).repeat();
|
|
|
|
commands.entity(entity).insert(animations);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
fn handle_inputs(
|
|
|
|
keyboard_input: Res<Input<KeyCode>>,
|
|
|
|
mut animation_player: Query<(&mut AnimationPlayer, &mut Clips, Entity, Option<&Name>)>,
|
|
|
|
) {
|
|
|
|
for (mut player, mut clips, entity, name) in &mut animation_player {
|
|
|
|
let display_entity_name = match name {
|
|
|
|
Some(name) => name.to_string(),
|
|
|
|
None => format!("entity {entity:?}"),
|
|
|
|
};
|
|
|
|
if keyboard_input.just_pressed(KeyCode::Space) {
|
|
|
|
if player.is_paused() {
|
|
|
|
info!("resuming animation for {display_entity_name}");
|
|
|
|
player.resume();
|
|
|
|
} else {
|
|
|
|
info!("pausing animation for {display_entity_name}");
|
|
|
|
player.pause();
|
|
|
|
}
|
|
|
|
}
|
|
|
|
if clips.clips.len() <= 1 {
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
if keyboard_input.just_pressed(KeyCode::Return) {
|
|
|
|
info!("switching to new animation for {display_entity_name}");
|
|
|
|
|
|
|
|
let resume = !player.is_paused();
|
|
|
|
// set the current animation to its start and pause it to reset to its starting state
|
API updates to the AnimationPlayer (#9002)
# Objective
Added `AnimationPlayer` API UX improvements.
- Succestor to https://github.com/bevyengine/bevy/pull/5912
- Fixes https://github.com/bevyengine/bevy/issues/5848
_(Credits to @asafigan for filing #5848, creating the initial pull
request, and the discussion in #5912)_
## Solution
- Created `RepeatAnimation` enum to describe an animation repetition
behavior.
- Added `is_finished()`, `set_repeat()`, and `is_playback_reversed()`
methods to the animation player.
- ~~Made the animation clip optional as per the comment from #5912~~
> ~~My problem is that the default handle [used the initialize a
`PlayingAnimation`] could actually refer to an actual animation if an
AnimationClip is set for the default handle, which leads me to ask,
"Should animation_clip should be an Option?"~~
- Added an accessor for the animation clip `animation_clip()` to the
animation player.
To determine if an animation is finished, we use the number of times the
animation has completed and the repetition behavior. If the animation is
playing in reverse then `elapsed < 0.0` counts as a completion.
Otherwise, `elapsed > animation.duration` counts as a completion. This
is what I would expect, personally. If there's any ambiguity, perhaps we
could add some `AnimationCompletionBehavior`, to specify that kind of
completion behavior to use.
Update: Previously `PlayingAnimation::elapsed` was being used as the
seek time into the animation clip. This was misleading because if you
increased the speed of the animation it would also increase (or
decrease) the elapsed time. In other words, the elapsed time was not
actually the elapsed time. To solve this, we introduce
`PlayingAnimation::seek_time` to serve as the value we manipulate the
move between keyframes. Consequently, `elapsed()` now returns the actual
elapsed time, and is not effected by the animation speed. Because
`set_elapsed` was being used to manipulate the displayed keyframe, we
introduce `AnimationPlayer::seek_to` and `AnimationPlayer::replay` to
provide this functionality.
## Migration Guide
- Removed `set_elapsed`.
- Removed `stop_repeating` in favour of
`AnimationPlayer::set_repeat(RepeatAnimation::Never)`.
- Introduced `seek_to` to seek to a given timestamp inside of the
animation.
- Introduced `seek_time` accessor for the `PlayingAnimation::seek_to`.
- Introduced `AnimationPlayer::replay` to reset the `PlayingAnimation`
to a state where no time has elapsed.
---------
Co-authored-by: Hennadii Chernyshchyk <genaloner@gmail.com>
Co-authored-by: François <mockersf@gmail.com>
2023-08-28 16:43:04 +00:00
|
|
|
player.seek_to(0.0).pause();
|
Add morph targets (#8158)
# Objective
- Add morph targets to `bevy_pbr` (closes #5756) & load them from glTF
- Supersedes #3722
- Fixes #6814
[Morph targets][1] (also known as shape interpolation, shape keys, or
blend shapes) allow animating individual vertices with fine grained
controls. This is typically used for facial expressions. By specifying
multiple poses as vertex offset, and providing a set of weight of each
pose, it is possible to define surprisingly realistic transitions
between poses. Blending between multiple poses also allow composition.
Morph targets are part of the [gltf standard][2] and are a feature of
Unity and Unreal, and babylone.js, it is only natural to implement them
in bevy.
## Solution
This implementation of morph targets uses a 3d texture where each pixel
is a component of an animated attribute. Each layer is a different
target. We use a 2d texture for each target, because the number of
attribute×components×animated vertices is expected to always exceed the
maximum pixel row size limit of webGL2. It copies fairly closely the way
skinning is implemented on the CPU side, while on the GPU side, the
shader morph target implementation is a relatively trivial detail.
We add an optional `morph_texture` to the `Mesh` struct. The
`morph_texture` is built through a method that accepts an iterator over
attribute buffers.
The `MorphWeights` component, user-accessible, controls the blend of
poses used by mesh instances (so that multiple copy of the same mesh may
have different weights), all the weights are uploaded to a uniform
buffer of 256 `f32`. We limit to 16 poses per mesh, and a total of 256
poses.
More literature:
* Old babylone.js implementation (vertex attribute-based):
https://www.eternalcoding.com/dev-log-1-morph-targets/
* Babylone.js implementation (similar to ours):
https://www.youtube.com/watch?v=LBPRmGgU0PE
* GPU gems 3:
https://developer.nvidia.com/gpugems/gpugems3/part-i-geometry/chapter-3-directx-10-blend-shapes-breaking-limits
* Development discord thread
https://discord.com/channels/691052431525675048/1083325980615114772
https://user-images.githubusercontent.com/26321040/231181046-3bca2ab2-d4d9-472e-8098-639f1871ce2e.mp4
https://github.com/bevyengine/bevy/assets/26321040/d2a0c544-0ef8-45cf-9f99-8c3792f5a258
## Acknowledgements
* Thanks to `storytold` for sponsoring the feature
* Thanks to `superdump` and `james7132` for guidance and help figuring
out stuff
## Future work
- Handling of less and more attributes (eg: animated uv, animated
arbitrary attributes)
- Dynamic pose allocation (so that zero-weighted poses aren't uploaded
to GPU for example, enables much more total poses)
- Better animation API, see #8357
----
## Changelog
- Add morph targets to bevy meshes
- Support up to 64 poses per mesh of individually up to 116508 vertices,
animation currently strictly limited to the position, normal and tangent
attributes.
- Load a morph target using `Mesh::set_morph_targets`
- Add `VisitMorphTargets` and `VisitMorphAttributes` traits to
`bevy_render`, this allows defining morph targets (a fairly complex and
nested data structure) through iterators (ie: single copy instead of
passing around buffers), see documentation of those traits for details
- Add `MorphWeights` component exported by `bevy_render`
- `MorphWeights` control mesh's morph target weights, blending between
various poses defined as morph targets.
- `MorphWeights` are directly inherited by direct children (single level
of hierarchy) of an entity. This allows controlling several mesh
primitives through a unique entity _as per GLTF spec_.
- Add `MorphTargetNames` component, naming each indices of loaded morph
targets.
- Load morph targets weights and buffers in `bevy_gltf`
- handle morph targets animations in `bevy_animation` (previously, it
was a `warn!` log)
- Add the `MorphStressTest.gltf` asset for morph targets testing, taken
from the glTF samples repo, CC0.
- Add morph target manipulation to `scene_viewer`
- Separate the animation code in `scene_viewer` from the rest of the
code, reducing `#[cfg(feature)]` noise
- Add the `morph_targets.rs` example to show off how to manipulate morph
targets, loading `MorpStressTest.gltf`
## Migration Guide
- (very specialized, unlikely to be touched by 3rd parties)
- `MeshPipeline` now has a single `mesh_layouts` field rather than
separate `mesh_layout` and `skinned_mesh_layout` fields. You should
handle all possible mesh bind group layouts in your implementation
- You should also handle properly the new `MORPH_TARGETS` shader def and
mesh pipeline key. A new function is exposed to make this easier:
`setup_moprh_and_skinning_defs`
- The `MeshBindGroup` is now `MeshBindGroups`, cached bind groups are
now accessed through the `get` method.
[1]: https://en.wikipedia.org/wiki/Morph_target_animation
[2]:
https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#morph-targets
---------
Co-authored-by: François <mockersf@gmail.com>
Co-authored-by: Carter Anderson <mcanders1@gmail.com>
2023-06-22 20:00:01 +00:00
|
|
|
|
|
|
|
clips.advance_to_next();
|
|
|
|
let current_clip = clips.current();
|
|
|
|
player.play(current_clip).repeat();
|
|
|
|
if resume {
|
|
|
|
player.resume();
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
pub struct AnimationManipulationPlugin;
|
|
|
|
impl Plugin for AnimationManipulationPlugin {
|
|
|
|
fn build(&self, app: &mut App) {
|
|
|
|
app.add_systems(Update, (handle_inputs, assign_clips));
|
|
|
|
}
|
|
|
|
}
|