Add parallax mapping to bevy PBR (#5928)

# Objective

Add a [parallax mapping] shader to bevy. Please note that
this is a 3d technique, NOT a 2d sidescroller feature.

## Solution

- Add related fields to `StandardMaterial`
- update the pbr shader
- Add an example taking advantage of parallax mapping

A pre-existing implementation exists at:
https://github.com/nicopap/bevy_mod_paramap/

The implementation is derived from:

https://web.archive.org/web/20150419215321/http://sunandblackcat.com/tipFullView.php?l=eng&topicid=28

Further discussion on literature is found in the `bevy_mod_paramap`
README.

### Limitations

- The mesh silhouette isn't affected by the depth map.
- The depth of the pixel does not reflect its visual position, resulting
  in artifacts for depth-dependent features such as fog or SSAO
- GLTF does not define a height map texture, so somehow the user will
  always need to work around this limitation, though [an extension is in
  the works][gltf]

### Future work

- It's possible to update the depth in the depth buffer to follow the
  parallaxed texture. This would enable interop with depth-based
  visual effects, it also allows `discard`ing pixels of materials when
  computed depth is higher than the one in depth buffer
- Cheap lower quality single-sample method using [offset limiting]
- Add distance fading, to disable parallaxing (relatively expensive)
  on distant objects
- GLTF extension to allow defining height maps. Or a workaround
  implemented through a blender plugin to the GLTF exporter that
  uses the `extras` field to add height map.
- [Quadratic surface vertex attributes][oliveira_3] to enable parallax
  mapping on bending surfaces and allow clean silhouetting.
- noise based sampling, to limit the pancake artifacts.
- Cone mapping ([GPU gems], [Simcity (2013)][simcity]). Requires
  preprocessing, increase depth map size, reduces sample count greatly.
- [Quadtree parallax mapping][qpm] (also requires preprocessing)
- Self-shadowing of parallax-mapped surfaces by modifying the shadow map
- Generate depth map from normal map [link to slides], [blender
question]


https://user-images.githubusercontent.com/26321040/223563792-dffcc6ab-70e8-4ff9-90d1-b36c338695ad.mp4

[blender question]:
https://blender.stackexchange.com/questions/89278/how-to-get-a-smooth-curvature-map-from-a-normal-map
[link to slides]:
https://developer.download.nvidia.com/assets/gamedev/docs/nmap2displacement.pdf
[oliveira_3]:
https://www.inf.ufrgs.br/~oliveira/pubs_files/Oliveira_Policarpo_RP-351_Jan_2005.pdf
[GPU gems]:
https://developer.nvidia.com/gpugems/gpugems3/part-iii-rendering/chapter-18-relaxed-cone-stepping-relief-mapping
[simcity]:
https://community.simtropolis.com/omnibus/other-games/building-and-rendering-simcity-2013-r247/
[offset limiting]:
https://raw.githubusercontent.com/marcusstenbeck/tncg14-parallax-mapping/master/documents/Parallax%20Mapping%20with%20Offset%20Limiting%20-%20A%20Per-Pixel%20Approximation%20of%20Uneven%20Surfaces.pdf
[gltf]: https://github.com/KhronosGroup/glTF/pull/2196
[qpm]:
https://www.gamedevs.org/uploads/quadtree-displacement-mapping-with-height-blending.pdf

---

## Changelog

- Add a `depth_map` field to the `StandardMaterial`, it is a grayscale
  image where white represents bottom and black the top. If `depth_map`
  is set, bevy's pbr shader will use it to do [parallax mapping] to
  give an increased feel of depth to the material. This is similar to a
  displacement map, but with infinite precision at fairly low cost.
- The fields `parallax_mapping_method`, `parallax_depth_scale` and
  `max_parallax_layer_count` allow finer grained control over the
  behavior of the parallax shader.
- Add the `parallax_mapping` example to show off the effect.

[parallax mapping]: https://en.wikipedia.org/wiki/Parallax_mapping

---------

Co-authored-by: Robert Swain <robert.swain@gmail.com>

Cargo.toml

@@ -561,6 +561,16 @@ description = "Demonstrates use of Physically Based Rendering (PBR) properties"
 category = "3D Rendering"
 wasm = true
 
+[[example]]
+name = "parallax_mapping"
+path = "examples/3d/parallax_mapping.rs"
+
+[package.metadata.example.parallax_mapping]
+name = "Parallax Mapping"
+description = "Demonstrates use of a normal map and depth map for parallax mapping"
+category = "3D Rendering"
+wasm = true
+
 [[example]]
 name = "render_to_texture"
 path = "examples/3d/render_to_texture.rs"

crates/bevy_pbr/src/lib.rs

@@ -8,6 +8,7 @@ mod environment_map;
 mod fog;
 mod light;
 mod material;
+mod parallax;
 mod pbr_material;
 mod prepass;
 mod render;
@@ -18,6 +19,7 @@ pub use environment_map::EnvironmentMapLight;
 pub use fog::*;
 pub use light::*;
 pub use material::*;
+pub use parallax::*;
 pub use pbr_material::*;
 pub use prepass::*;
 pub use render::*;
@@ -34,6 +36,7 @@ pub mod prelude {
         fog::{FogFalloff, FogSettings},
         light::{AmbientLight, DirectionalLight, PointLight, SpotLight},
         material::{Material, MaterialPlugin},
+        parallax::ParallaxMappingMethod,
         pbr_material::StandardMaterial,
     };
 }
@@ -82,6 +85,8 @@ pub const PBR_FUNCTIONS_HANDLE: HandleUntyped =
     HandleUntyped::weak_from_u64(Shader::TYPE_UUID, 16550102964439850292);
 pub const PBR_AMBIENT_HANDLE: HandleUntyped =
     HandleUntyped::weak_from_u64(Shader::TYPE_UUID, 2441520459096337034);
+pub const PARALLAX_MAPPING_SHADER_HANDLE: HandleUntyped =
+    HandleUntyped::weak_from_u64(Shader::TYPE_UUID, 17035894873630133905);
 
 /// Sets up the entire PBR infrastructure of bevy.
 pub struct PbrPlugin {
@@ -150,6 +155,12 @@ impl Plugin for PbrPlugin {
             "render/pbr_prepass.wgsl",
             Shader::from_wgsl
         );
+        load_internal_asset!(
+            app,
+            PARALLAX_MAPPING_SHADER_HANDLE,
+            "render/parallax_mapping.wgsl",
+            Shader::from_wgsl
+        );
 
         app.register_asset_reflect::<StandardMaterial>()
             .register_type::<AmbientLight>()

crates/bevy_pbr/src/parallax.rs

@@ -0,0 +1,45 @@
+use bevy_reflect::{FromReflect, Reflect};
+
+/// The [parallax mapping] method to use to compute depth based on the
+/// material's [`depth_map`].
+///
+/// Parallax Mapping uses a depth map texture to give the illusion of depth
+/// variation on a mesh surface that is geometrically flat.
+///
+/// See the `parallax_mapping.wgsl` shader code for implementation details
+/// and explanation of the methods used.
+///
+/// [`depth_map`]: crate::StandardMaterial::depth_map
+/// [parallax mapping]: https://en.wikipedia.org/wiki/Parallax_mapping
+#[derive(Debug, Copy, Clone, PartialEq, Eq, Default, Reflect, FromReflect)]
+pub enum ParallaxMappingMethod {
+    /// A simple linear interpolation, using a single texture sample.
+    ///
+    /// This method is named "Parallax Occlusion Mapping".
+    ///
+    /// Unlike [`ParallaxMappingMethod::Relief`], only requires a single lookup,
+    /// but may skip small details and result in writhing material artifacts.
+    #[default]
+    Occlusion,
+    /// Discovers the best depth value based on binary search.
+    ///
+    /// Each iteration incurs a texture sample.
+    /// The result has fewer visual artifacts than [`ParallaxMappingMethod::Occlusion`].
+    ///
+    /// This method is named "Relief Mapping".
+    Relief {
+        /// How many additional steps to use at most to find the depth value.
+        max_steps: u32,
+    },
+}
+impl ParallaxMappingMethod {
+    /// [`ParallaxMappingMethod::Relief`] with a 5 steps, a reasonable default.
+    pub const DEFAULT_RELIEF_MAPPING: Self = ParallaxMappingMethod::Relief { max_steps: 5 };
+
+    pub(crate) fn max_steps(&self) -> u32 {
+        match self {
+            ParallaxMappingMethod::Occlusion => 0,
+            ParallaxMappingMethod::Relief { max_steps } => *max_steps,
+        }
+    }
+}

crates/bevy_pbr/src/pbr_material.rs

@@ -1,6 +1,6 @@
 use crate::{
-    AlphaMode, Material, MaterialPipeline, MaterialPipelineKey, PBR_PREPASS_SHADER_HANDLE,
-    PBR_SHADER_HANDLE,
+    AlphaMode, Material, MaterialPipeline, MaterialPipelineKey, ParallaxMappingMethod,
+    PBR_PREPASS_SHADER_HANDLE, PBR_SHADER_HANDLE,
 };
 use bevy_asset::Handle;
 use bevy_math::Vec4;
@@ -231,6 +231,84 @@ pub struct StandardMaterial {
     ///
     /// [z-fighting]: https://en.wikipedia.org/wiki/Z-fighting
     pub depth_bias: f32,
+
+    /// The depth map used for [parallax mapping].
+    ///
+    /// It is a greyscale image where white represents bottom and black the top.
+    /// If this field is set, bevy will apply [parallax mapping].
+    /// Parallax mapping, unlike simple normal maps, will move the texture
+    /// coordinate according to the current perspective,
+    /// giving actual depth to the texture.
+    ///
+    /// The visual result is similar to a displacement map,
+    /// but does not require additional geometry.
+    ///
+    /// Use the [`parallax_depth_scale`] field to control the depth of the parallax.
+    ///
+    /// ## Limitations
+    ///
+    /// - It will look weird on bent/non-planar surfaces.
+    /// - The depth of the pixel does not reflect its visual position, resulting
+    ///   in artifacts for depth-dependent features such as fog or SSAO.
+    /// - For the same reason, the the geometry silhouette will always be
+    ///   the one of the actual geometry, not the parallaxed version, resulting
+    ///   in awkward looks on intersecting parallaxed surfaces.
+    ///
+    /// ## Performance
+    ///
+    /// Parallax mapping requires multiple texture lookups, proportional to
+    /// [`max_parallax_layer_count`], which might be costly.
+    ///
+    /// Use the [`parallax_mapping_method`] and [`max_parallax_layer_count`] fields
+    /// to tweak the shader, trading graphical quality for performance.
+    ///
+    /// To improve performance, set your `depth_map`'s [`Image::sampler_descriptor`]
+    /// filter mode to `FilterMode::Nearest`, as [this paper] indicates, it improves
+    /// performance a bit.
+    ///
+    /// To reduce artifacts, avoid steep changes in depth, blurring the depth
+    /// map helps with this.
+    ///
+    /// Larger depth maps haves a disproportionate performance impact.
+    ///
+    /// [this paper]: https://www.diva-portal.org/smash/get/diva2:831762/FULLTEXT01.pdf
+    /// [parallax mapping]: https://en.wikipedia.org/wiki/Parallax_mapping
+    /// [`parallax_depth_scale`]: StandardMaterial::parallax_depth_scale
+    /// [`parallax_mapping_method`]: StandardMaterial::parallax_mapping_method
+    /// [`max_parallax_layer_count`]: StandardMaterial::max_parallax_layer_count
+    #[texture(11)]
+    #[sampler(12)]
+    pub depth_map: Option<Handle<Image>>,
+
+    /// How deep the offset introduced by the depth map should be.
+    ///
+    /// Default is `0.1`, anything over that value may look distorted.
+    /// Lower values lessen the effect.
+    ///
+    /// The depth is relative to texture size. This means that if your texture
+    /// occupies a surface of `1` world unit, and `parallax_depth_scale` is `0.1`, then
+    /// the in-world depth will be of `0.1` world units.
+    /// If the texture stretches for `10` world units, then the final depth
+    /// will be of `1` world unit.
+    pub parallax_depth_scale: f32,
+
+    /// Which parallax mapping method to use.
+    ///
+    /// We recommend that all objects use the same [`ParallaxMappingMethod`], to avoid
+    /// duplicating and running two shaders.
+    pub parallax_mapping_method: ParallaxMappingMethod,
+
+    /// In how many layers to split the depth maps for parallax mapping.
+    ///
+    /// If you are seeing jaggy edges, increase this value.
+    /// However, this incurs a performance cost.
+    ///
+    /// Dependent on the situation, switching to [`ParallaxMappingMethod::Relief`]
+    /// and keeping this value low might have better performance than increasing the
+    /// layer count while using [`ParallaxMappingMethod::Occlusion`].
+    ///
+    /// Default is `16.0`.
+    pub max_parallax_layer_count: f32,
 }
 
 impl Default for StandardMaterial {
@@ -260,6 +338,10 @@ impl Default for StandardMaterial {
             fog_enabled: true,
             alpha_mode: AlphaMode::Opaque,
             depth_bias: 0.0,
+            depth_map: None,
+            parallax_depth_scale: 0.1,
+            max_parallax_layer_count: 16.0,
+            parallax_mapping_method: ParallaxMappingMethod::Occlusion,
         }
     }
 }
@@ -302,6 +384,7 @@ bitflags::bitflags! {
         const TWO_COMPONENT_NORMAL_MAP   = (1 << 6);
         const FLIP_NORMAL_MAP_Y          = (1 << 7);
         const FOG_ENABLED                = (1 << 8);
+        const DEPTH_MAP                  = (1 << 9); // Used for parallax mapping
         const ALPHA_MODE_RESERVED_BITS   = (Self::ALPHA_MODE_MASK_BITS << Self::ALPHA_MODE_SHIFT_BITS); // ← Bitmask reserving bits for the `AlphaMode`
         const ALPHA_MODE_OPAQUE          = (0 << Self::ALPHA_MODE_SHIFT_BITS);                          // ← Values are just sequential values bitshifted into
         const ALPHA_MODE_MASK            = (1 << Self::ALPHA_MODE_SHIFT_BITS);                          //   the bitmask, and can range from 0 to 7.
@@ -341,6 +424,16 @@ pub struct StandardMaterialUniform {
     /// When the alpha mode mask flag is set, any base color alpha above this cutoff means fully opaque,
     /// and any below means fully transparent.
     pub alpha_cutoff: f32,
+    /// The depth of the [`StandardMaterial::depth_map`] to apply.
+    pub parallax_depth_scale: f32,
+    /// In how many layers to split the depth maps for Steep parallax mapping.
+    ///
+    /// If your `parallax_depth_scale` is >0.1 and you are seeing jaggy edges,
+    /// increase this value. However, this incurs a performance cost.
+    pub max_parallax_layer_count: f32,
+    /// Using [`ParallaxMappingMethod::Relief`], how many additional
+    /// steps to use at most to find the depth value.
+    pub max_relief_mapping_search_steps: u32,
 }
 
 impl AsBindGroupShaderType<StandardMaterialUniform> for StandardMaterial {
@@ -367,6 +460,9 @@ impl AsBindGroupShaderType<StandardMaterialUniform> for StandardMaterial {
         if self.fog_enabled {
             flags |= StandardMaterialFlags::FOG_ENABLED;
         }
+        if self.depth_map.is_some() {
+            flags |= StandardMaterialFlags::DEPTH_MAP;
+        }
         let has_normal_map = self.normal_map_texture.is_some();
         if has_normal_map {
             if let Some(texture) = images.get(self.normal_map_texture.as_ref().unwrap()) {
@@ -407,15 +503,20 @@ impl AsBindGroupShaderType<StandardMaterialUniform> for StandardMaterial {
             reflectance: self.reflectance,
             flags: flags.bits(),
             alpha_cutoff,
+            parallax_depth_scale: self.parallax_depth_scale,
+            max_parallax_layer_count: self.max_parallax_layer_count,
+            max_relief_mapping_search_steps: self.parallax_mapping_method.max_steps(),
         }
     }
 }
 
+/// The pipeline key for [`StandardMaterial`].
 #[derive(Clone, PartialEq, Eq, Hash)]
 pub struct StandardMaterialKey {
     normal_map: bool,
     cull_mode: Option<Face>,
     depth_bias: i32,
+    relief_mapping: bool,
 }
 
 impl From<&StandardMaterial> for StandardMaterialKey {
@@ -424,6 +525,10 @@ impl From<&StandardMaterial> for StandardMaterialKey {
             normal_map: material.normal_map_texture.is_some(),
             cull_mode: material.cull_mode,
             depth_bias: material.depth_bias as i32,
+            relief_mapping: matches!(
+                material.parallax_mapping_method,
+                ParallaxMappingMethod::Relief { .. }
+            ),
         }
     }
 }
@@ -435,11 +540,14 @@ impl Material for StandardMaterial {
         _layout: &MeshVertexBufferLayout,
         key: MaterialPipelineKey<Self>,
     ) -> Result<(), SpecializedMeshPipelineError> {
-        if key.bind_group_data.normal_map {
         if let Some(fragment) = descriptor.fragment.as_mut() {
-                fragment
-                    .shader_defs
-                    .push("STANDARDMATERIAL_NORMAL_MAP".into());
+            let shader_defs = &mut fragment.shader_defs;
+
+            if key.bind_group_data.normal_map {
+                shader_defs.push("STANDARDMATERIAL_NORMAL_MAP".into());
+            }
+            if key.bind_group_data.relief_mapping {
+                shader_defs.push("RELIEF_MAPPING".into());
             }
         }
         descriptor.primitive.cull_mode = key.bind_group_data.cull_mode;

crates/bevy_pbr/src/render/parallax_mapping.wgsl

@@ -0,0 +1,116 @@
+#define_import_path bevy_pbr::parallax_mapping
+
+fn sample_depth_map(uv: vec2<f32>) -> f32 {
+    // We use `textureSampleLevel` over `textureSample` because the wgpu DX12
+    // backend (Fxc) panics when using "gradient instructions" inside a loop.
+    // It results in the whole loop being unrolled by the shader compiler,
+    // which it can't do because the upper limit of the loop in steep parallax
+    // mapping is a variable set by the user.
+    // The "gradient instructions" comes from `textureSample` computing MIP level
+    // based on UV derivative. With `textureSampleLevel`, we provide ourselves
+    // the MIP level, so no gradient instructions are used, and we can use
+    // sample_depth_map in our loop.
+    // See https://stackoverflow.com/questions/56581141/direct3d11-gradient-instruction-used-in-a-loop-with-varying-iteration-forcing
+    return textureSampleLevel(depth_map_texture, depth_map_sampler, uv, 0.0).r;
+}
+
+// An implementation of parallax mapping, see https://en.wikipedia.org/wiki/Parallax_mapping
+// Code derived from: https://web.archive.org/web/20150419215321/http://sunandblackcat.com/tipFullView.php?l=eng&topicid=28
+fn parallaxed_uv(
+    depth_scale: f32,
+    max_layer_count: f32,
+    max_steps: u32,
+    // The original interpolated uv
+    uv: vec2<f32>,
+    // The vector from the camera to the fragment at the surface in tangent space
+    Vt: vec3<f32>,
+) -> vec2<f32> {
+    if max_layer_count < 1.0 {
+        return uv;
+    }
+    var uv = uv;
+
+    // Steep Parallax Mapping
+    // ======================
+    // Split the depth map into `layer_count` layers.
+    // When Vt hits the surface of the mesh (excluding depth displacement),
+    // if the depth is not below or on surface including depth displacement (textureSample), then
+    // look forward (+= delta_uv) on depth texture according to
+    // Vt and distance between hit surface and depth map surface,
+    // repeat until below the surface.
+    //
+    // Where `layer_count` is interpolated between `1.0` and
+    // `max_layer_count` according to the steepness of Vt.
+
+    let view_steepness = abs(Vt.z);
+    // We mix with minimum value 1.0 because otherwise,
+    // with 0.0, we get a division by zero in surfaces parallel to viewport,
+    // resulting in a singularity.
+    let layer_count = mix(max_layer_count, 1.0, view_steepness);
+    let layer_depth = 1.0 / layer_count;
+    var delta_uv = depth_scale * layer_depth * Vt.xy * vec2(1.0, -1.0) / view_steepness;
+
+    var current_layer_depth = 0.0;
+    var texture_depth = sample_depth_map(uv);
+
+    // texture_depth > current_layer_depth means the depth map depth is deeper
+    // than the depth the ray would be at at this UV offset so the ray has not
+    // intersected the surface
+    for (var i: i32 = 0; texture_depth > current_layer_depth && i <= i32(layer_count); i++) {
+        current_layer_depth += layer_depth;
+        uv += delta_uv;
+        texture_depth = sample_depth_map(uv);
+    }
+
+#ifdef RELIEF_MAPPING
+    // Relief Mapping
+    // ==============
+    // "Refine" the rough result from Steep Parallax Mapping
+    // with a **binary search** between the layer selected by steep parallax
+    // and the next one to find a point closer to the depth map surface.
+    // This reduces the jaggy step artifacts from steep parallax mapping.
+
+    delta_uv *= 0.5;
+    var delta_depth = 0.5 * layer_depth;
+
+    uv -= delta_uv;
+    current_layer_depth -= delta_depth;
+
+    for (var i: u32 = 0u; i < max_steps; i++) {
+        texture_depth = sample_depth_map(uv);
+
+        // Halve the deltas for the next step
+        delta_uv *= 0.5;
+        delta_depth *= 0.5;
+
+        // Step based on whether the current depth is above or below the depth map
+        if (texture_depth > current_layer_depth) {
+            uv += delta_uv;
+            current_layer_depth += delta_depth;
+        } else {
+            uv -= delta_uv;
+            current_layer_depth -= delta_depth;
+        }
+    }
+#else
+    // Parallax Occlusion mapping
+    // ==========================
+    // "Refine" Steep Parallax Mapping by interpolating between the
+    // previous layer's depth and the computed layer depth.
+    // Only requires a single lookup, unlike Relief Mapping, but
+    // may skip small details and result in writhing material artifacts.
+    let previous_uv = uv - delta_uv;
+    let next_depth = texture_depth - current_layer_depth;
+    let previous_depth = sample_depth_map(previous_uv) - current_layer_depth + layer_depth;
+
+    let weight = next_depth / (next_depth - previous_depth);
+
+    uv = mix(uv, previous_uv, weight);
+
+    current_layer_depth += mix(next_depth, previous_depth, weight);
+#endif
+
+    // Note: `current_layer_depth` is not returned, but may be useful
+    // for light computation later on in future improvements of the pbr shader.
+    return uv;
+}

crates/bevy_pbr/src/render/pbr.wgsl

@@ -9,6 +9,7 @@
 #import bevy_pbr::shadows
 #import bevy_pbr::fog
 #import bevy_pbr::pbr_functions
+#import bevy_pbr::parallax_mapping
 
 #import bevy_pbr::prepass_utils
 
@@ -20,13 +21,37 @@ struct FragmentInput {
 
 @fragment
 fn fragment(in: FragmentInput) -> @location(0) vec4<f32> {
+    let is_orthographic = view.projection[3].w == 1.0;
+    let V = calculate_view(in.world_position, is_orthographic);
+    var uv = in.uv;
+#ifdef VERTEX_UVS
+#ifdef VERTEX_TANGENTS
+    if ((material.flags & STANDARD_MATERIAL_FLAGS_DEPTH_MAP_BIT) != 0u) {
+        let N = in.world_normal;
+        let T = in.world_tangent.xyz;
+        let B = in.world_tangent.w * cross(N, T);
+        // Transform V from fragment to camera in world space to tangent space.
+        let Vt = vec3(dot(V, T), dot(V, B), dot(V, N));
+        uv = parallaxed_uv(
+            material.parallax_depth_scale,
+            material.max_parallax_layer_count,
+            material.max_relief_mapping_search_steps,
+            uv,
+            // Flip the direction of Vt to go toward the surface to make the
+            // parallax mapping algorithm easier to understand and reason
+            // about.
+            -Vt,
+        );
+    }
+#endif
+#endif
     var output_color: vec4<f32> = material.base_color;
 #ifdef VERTEX_COLORS
     output_color = output_color * in.color;
 #endif
 #ifdef VERTEX_UVS
     if ((material.flags & STANDARD_MATERIAL_FLAGS_BASE_COLOR_TEXTURE_BIT) != 0u) {
-        output_color = output_color * textureSample(base_color_texture, base_color_sampler, in.uv);
+        output_color = output_color * textureSample(base_color_texture, base_color_sampler, uv);
     }
 #endif
 
@@ -45,7 +70,7 @@ fn fragment(in: FragmentInput) -> @location(0) vec4<f32> {
         var emissive: vec4<f32> = material.emissive;
 #ifdef VERTEX_UVS
         if ((material.flags & STANDARD_MATERIAL_FLAGS_EMISSIVE_TEXTURE_BIT) != 0u) {
-            emissive = vec4<f32>(emissive.rgb * textureSample(emissive_texture, emissive_sampler, in.uv).rgb, 1.0);
+            emissive = vec4<f32>(emissive.rgb * textureSample(emissive_texture, emissive_sampler, uv).rgb, 1.0);
         }
 #endif
         pbr_input.material.emissive = emissive;
@@ -54,7 +79,7 @@ fn fragment(in: FragmentInput) -> @location(0) vec4<f32> {
         var perceptual_roughness: f32 = material.perceptual_roughness;
 #ifdef VERTEX_UVS
         if ((material.flags & STANDARD_MATERIAL_FLAGS_METALLIC_ROUGHNESS_TEXTURE_BIT) != 0u) {
-            let metallic_roughness = textureSample(metallic_roughness_texture, metallic_roughness_sampler, in.uv);
+            let metallic_roughness = textureSample(metallic_roughness_texture, metallic_roughness_sampler, uv);
             // Sampling from GLTF standard channels for now
             metallic = metallic * metallic_roughness.b;
             perceptual_roughness = perceptual_roughness * metallic_roughness.g;
@@ -66,7 +91,7 @@ fn fragment(in: FragmentInput) -> @location(0) vec4<f32> {
         var occlusion: f32 = 1.0;
 #ifdef VERTEX_UVS
         if ((material.flags & STANDARD_MATERIAL_FLAGS_OCCLUSION_TEXTURE_BIT) != 0u) {
-            occlusion = textureSample(occlusion_texture, occlusion_sampler, in.uv).r;
+            occlusion = textureSample(occlusion_texture, occlusion_sampler, uv).r;
         }
 #endif
         pbr_input.frag_coord = in.frag_coord;
@@ -82,7 +107,7 @@ fn fragment(in: FragmentInput) -> @location(0) vec4<f32> {
         );
 #endif // LOAD_PREPASS_NORMALS
 
-        pbr_input.is_orthographic = view.projection[3].w == 1.0;
+        pbr_input.is_orthographic = is_orthographic;
 
         pbr_input.N = apply_normal_mapping(
             material.flags,
@@ -93,10 +118,10 @@ fn fragment(in: FragmentInput) -> @location(0) vec4<f32> {
 #endif
 #endif
 #ifdef VERTEX_UVS
-            in.uv,
+            uv,
 #endif
         );
-        pbr_input.V = calculate_view(in.world_position, pbr_input.is_orthographic);
+        pbr_input.V = V;
         pbr_input.occlusion = occlusion;
 
         pbr_input.flags = mesh.flags;

crates/bevy_pbr/src/render/pbr_bindings.wgsl

@@ -24,3 +24,7 @@ var occlusion_sampler: sampler;
 var normal_map_texture: texture_2d<f32>;
 @group(1) @binding(10)
 var normal_map_sampler: sampler;
+@group(1) @binding(11)
+var depth_map_texture: texture_2d<f32>;
+@group(1) @binding(12)
+var depth_map_sampler: sampler;

crates/bevy_pbr/src/render/pbr_types.wgsl

@@ -9,6 +9,9 @@ struct StandardMaterial {
     // 'flags' is a bit field indicating various options. u32 is 32 bits so we have up to 32 options.
     flags: u32,
     alpha_cutoff: f32,
+    parallax_depth_scale: f32,
+    max_parallax_layer_count: f32,
+    max_relief_mapping_search_steps: u32,
 };
 
 const STANDARD_MATERIAL_FLAGS_BASE_COLOR_TEXTURE_BIT: u32         = 1u;
@@ -20,6 +23,7 @@ const STANDARD_MATERIAL_FLAGS_UNLIT_BIT: u32                      = 32u;
 const STANDARD_MATERIAL_FLAGS_TWO_COMPONENT_NORMAL_MAP: u32       = 64u;
 const STANDARD_MATERIAL_FLAGS_FLIP_NORMAL_MAP_Y: u32              = 128u;
 const STANDARD_MATERIAL_FLAGS_FOG_ENABLED_BIT: u32                = 256u;
+const STANDARD_MATERIAL_FLAGS_DEPTH_MAP_BIT: u32                  = 512u;
 const STANDARD_MATERIAL_FLAGS_ALPHA_MODE_RESERVED_BITS: u32       = 3758096384u; // (0b111u32 << 29)
 const STANDARD_MATERIAL_FLAGS_ALPHA_MODE_OPAQUE: u32              = 0u;          // (0u32 << 29)
 const STANDARD_MATERIAL_FLAGS_ALPHA_MODE_MASK: u32                = 536870912u;  // (1u32 << 29)
@@ -42,6 +46,9 @@ fn standard_material_new() -> StandardMaterial {
     material.reflectance = 0.5;
     material.flags = STANDARD_MATERIAL_FLAGS_ALPHA_MODE_OPAQUE;
     material.alpha_cutoff = 0.5;
+    material.parallax_depth_scale = 0.1;
+    material.max_parallax_layer_count = 16.0;
+    material.max_relief_mapping_search_steps = 5u;
 
     return material;
 }

examples/3d/parallax_mapping.rs

@@ -0,0 +1,387 @@
+//! A simple 3D scene with a spinning cube with a normal map and depth map to demonstrate parallax mapping.
+//! Press left mouse button to cycle through different views.
+
+use std::fmt;
+
+use bevy::{prelude::*, render::render_resource::TextureFormat, window::close_on_esc};
+
+fn main() {
+    App::new()
+        .add_plugins(DefaultPlugins)
+        .insert_resource(Normal(None))
+        .add_systems(Startup, setup)
+        .add_systems(
+            Update,
+            (
+                spin,
+                update_normal,
+                move_camera,
+                update_parallax_depth_scale,
+                update_parallax_layers,
+                switch_method,
+                close_on_esc,
+            ),
+        )
+        .run();
+}
+
+#[derive(Component)]
+struct Spin {
+    speed: f32,
+}
+
+/// The camera, used to move camera on click.
+#[derive(Component)]
+struct CameraController;
+
+const DEPTH_CHANGE_RATE: f32 = 0.1;
+const DEPTH_UPDATE_STEP: f32 = 0.03;
+const MAX_DEPTH: f32 = 0.3;
+
+struct TargetDepth(f32);
+impl Default for TargetDepth {
+    fn default() -> Self {
+        TargetDepth(0.09)
+    }
+}
+struct TargetLayers(f32);
+impl Default for TargetLayers {
+    fn default() -> Self {
+        TargetLayers(5.0)
+    }
+}
+struct CurrentMethod(ParallaxMappingMethod);
+impl Default for CurrentMethod {
+    fn default() -> Self {
+        CurrentMethod(ParallaxMappingMethod::Relief { max_steps: 4 })
+    }
+}
+impl fmt::Display for CurrentMethod {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        match self.0 {
+            ParallaxMappingMethod::Occlusion => write!(f, "Parallax Occlusion Mapping"),
+            ParallaxMappingMethod::Relief { max_steps } => {
+                write!(f, "Relief Mapping with {max_steps} steps")
+            }
+        }
+    }
+}
+impl CurrentMethod {
+    fn next_method(&mut self) {
+        use ParallaxMappingMethod::*;
+        self.0 = match self.0 {
+            Occlusion => Relief { max_steps: 2 },
+            Relief { max_steps } if max_steps < 3 => Relief { max_steps: 4 },
+            Relief { max_steps } if max_steps < 5 => Relief { max_steps: 8 },
+            Relief { .. } => Occlusion,
+        }
+    }
+}
+
+fn update_parallax_depth_scale(
+    input: Res<Input<KeyCode>>,
+    mut materials: ResMut<Assets<StandardMaterial>>,
+    mut target_depth: Local<TargetDepth>,
+    mut depth_update: Local<bool>,
+    mut text: Query<&mut Text>,
+) {
+    if input.just_pressed(KeyCode::Key1) {
+        target_depth.0 -= DEPTH_UPDATE_STEP;
+        target_depth.0 = target_depth.0.max(0.0);
+        *depth_update = true;
+    }
+    if input.just_pressed(KeyCode::Key2) {
+        target_depth.0 += DEPTH_UPDATE_STEP;
+        target_depth.0 = target_depth.0.min(MAX_DEPTH);
+        *depth_update = true;
+    }
+    if *depth_update {
+        let mut text = text.single_mut();
+        for (_, mat) in materials.iter_mut() {
+            let current_depth = mat.parallax_depth_scale;
+            let new_depth =
+                current_depth * (1.0 - DEPTH_CHANGE_RATE) + (target_depth.0 * DEPTH_CHANGE_RATE);
+            mat.parallax_depth_scale = new_depth;
+            text.sections[0].value = format!("Parallax depth scale: {new_depth:.5}\n");
+            if (new_depth - current_depth).abs() <= 0.000000001 {
+                *depth_update = false;
+            }
+        }
+    }
+}
+
+fn switch_method(
+    input: Res<Input<KeyCode>>,
+    mut materials: ResMut<Assets<StandardMaterial>>,
+    mut text: Query<&mut Text>,
+    mut current: Local<CurrentMethod>,
+) {
+    if input.just_pressed(KeyCode::Space) {
+        current.next_method();
+    } else {
+        return;
+    }
+    let mut text = text.single_mut();
+    text.sections[2].value = format!("Method: {}\n", *current);
+
+    for (_, mat) in materials.iter_mut() {
+        mat.parallax_mapping_method = current.0;
+    }
+}
+
+fn update_parallax_layers(
+    input: Res<Input<KeyCode>>,
+    mut materials: ResMut<Assets<StandardMaterial>>,
+    mut target_layers: Local<TargetLayers>,
+    mut text: Query<&mut Text>,
+) {
+    if input.just_pressed(KeyCode::Key3) {
+        target_layers.0 -= 1.0;
+        target_layers.0 = target_layers.0.max(0.0);
+    } else if input.just_pressed(KeyCode::Key4) {
+        target_layers.0 += 1.0;
+    } else {
+        return;
+    }
+    let layer_count = target_layers.0.exp2();
+    let mut text = text.single_mut();
+    text.sections[1].value = format!("Layers: {layer_count:.0}\n");
+
+    for (_, mat) in materials.iter_mut() {
+        mat.max_parallax_layer_count = layer_count;
+    }
+}
+
+fn spin(time: Res<Time>, mut query: Query<(&mut Transform, &Spin)>) {
+    for (mut transform, spin) in query.iter_mut() {
+        transform.rotate_local_y(spin.speed * time.delta_seconds());
+        transform.rotate_local_x(spin.speed * time.delta_seconds());
+        transform.rotate_local_z(-spin.speed * time.delta_seconds());
+    }
+}
+
+// Camera positions to cycle through when left-clickig.
+const CAMERA_POSITIONS: &[Transform] = &[
+    Transform {
+        translation: Vec3::new(1.5, 1.5, 1.5),
+        rotation: Quat::from_xyzw(-0.279, 0.364, 0.115, 0.880),
+        scale: Vec3::ONE,
+    },
+    Transform {
+        translation: Vec3::new(2.4, 0.0, 0.2),
+        rotation: Quat::from_xyzw(0.094, 0.676, 0.116, 0.721),
+        scale: Vec3::ONE,
+    },
+    Transform {
+        translation: Vec3::new(2.4, 2.6, -4.3),
+        rotation: Quat::from_xyzw(0.170, 0.908, 0.308, 0.225),
+        scale: Vec3::ONE,
+    },
+    Transform {
+        translation: Vec3::new(-1.0, 0.8, -1.2),
+        rotation: Quat::from_xyzw(-0.004, 0.909, 0.247, -0.335),
+        scale: Vec3::ONE,
+    },
+];
+
+fn move_camera(
+    mut camera: Query<&mut Transform, With<CameraController>>,
+    mut current_view: Local<usize>,
+    button: Res<Input<MouseButton>>,
+) {
+    let mut camera = camera.single_mut();
+    if button.just_pressed(MouseButton::Left) {
+        *current_view = (*current_view + 1) % CAMERA_POSITIONS.len();
+    }
+    let target = CAMERA_POSITIONS[*current_view];
+    camera.translation = camera.translation.lerp(target.translation, 0.2);
+    camera.rotation = camera.rotation.slerp(target.rotation, 0.2);
+}
+
+fn setup(
+    mut commands: Commands,
+    mut materials: ResMut<Assets<StandardMaterial>>,
+    mut meshes: ResMut<Assets<Mesh>>,
+    mut normal: ResMut<Normal>,
+    asset_server: Res<AssetServer>,
+) {
+    // The normal map. Note that to generate it in the GIMP image editor, you should
+    // open the depth map, and do Filters → Generic → Normal Map
+    // You should enable the "flip X" checkbox.
+    let normal_handle = asset_server.load("textures/parallax_example/cube_normal.png");
+    normal.0 = Some(normal_handle);
+
+    // Camera
+    commands.spawn((
+        Camera3dBundle {
+            transform: Transform::from_xyz(1.5, 1.5, 1.5).looking_at(Vec3::ZERO, Vec3::Y),
+            ..default()
+        },
+        CameraController,
+    ));
+
+    // light
+    commands
+        .spawn(PointLightBundle {
+            transform: Transform::from_xyz(1.8, 0.7, -1.1),
+            point_light: PointLight {
+                intensity: 226.0,
+                shadows_enabled: true,
+                ..default()
+            },
+            ..default()
+        })
+        .with_children(|commands| {
+            // represent the light source as a sphere
+            let mesh = meshes.add(
+                shape::Icosphere {
+                    radius: 0.05,
+                    subdivisions: 3,
+                }
+                .try_into()
+                .unwrap(),
+            );
+            commands.spawn(PbrBundle { mesh, ..default() });
+        });
+
+    // Plane
+    commands.spawn(PbrBundle {
+        mesh: meshes.add(
+            shape::Plane {
+                size: 10.0,
+                subdivisions: 0,
+            }
+            .into(),
+        ),
+        material: materials.add(StandardMaterial {
+            // standard material derived from dark green, but
+            // with roughness and reflectance set.
+            perceptual_roughness: 0.45,
+            reflectance: 0.18,
+            ..Color::rgb_u8(0, 80, 0).into()
+        }),
+        transform: Transform::from_xyz(0.0, -1.0, 0.0),
+        ..default()
+    });
+
+    let mut cube: Mesh = shape::Cube { size: 1.0 }.into();
+
+    // NOTE: for normal maps and depth maps to work, the mesh
+    // needs tangents generated.
+    cube.generate_tangents().unwrap();
+
+    let parallax_depth_scale = TargetDepth::default().0;
+    let max_parallax_layer_count = TargetLayers::default().0.exp2();
+    let parallax_mapping_method = CurrentMethod::default();
+    let parallax_material = materials.add(StandardMaterial {
+        perceptual_roughness: 0.4,
+        base_color_texture: Some(asset_server.load("textures/parallax_example/cube_color.png")),
+        normal_map_texture: normal.0.clone(),
+        // The depth map is a greyscale texture where black is the highest level and
+        // white the lowest.
+        depth_map: Some(asset_server.load("textures/parallax_example/cube_depth.png")),
+        parallax_depth_scale,
+        parallax_mapping_method: parallax_mapping_method.0,
+        max_parallax_layer_count,
+        ..default()
+    });
+    commands.spawn((
+        PbrBundle {
+            mesh: meshes.add(cube),
+            material: parallax_material.clone_weak(),
+            ..default()
+        },
+        Spin { speed: 0.3 },
+    ));
+
+    let mut background_cube: Mesh = shape::Cube { size: 40.0 }.into();
+    background_cube.generate_tangents().unwrap();
+    let background_cube = meshes.add(background_cube);
+
+    let background_cube_bundle = |translation| {
+        (
+            PbrBundle {
+                transform: Transform::from_translation(translation),
+                mesh: background_cube.clone(),
+                material: parallax_material.clone(),
+                ..default()
+            },
+            Spin { speed: -0.1 },
+        )
+    };
+    commands.spawn(background_cube_bundle(Vec3::new(45., 0., 0.)));
+    commands.spawn(background_cube_bundle(Vec3::new(-45., 0., 0.)));
+    commands.spawn(background_cube_bundle(Vec3::new(0., 0., 45.)));
+    commands.spawn(background_cube_bundle(Vec3::new(0., 0., -45.)));
+
+    let style = TextStyle {
+        font: asset_server.load("fonts/FiraMono-Medium.ttf"),
+        font_size: 18.0,
+        color: Color::WHITE,
+    };
+
+    commands.spawn(
+        TextBundle::from_sections(vec![
+            TextSection::new(
+                format!("Parallax depth scale: {parallax_depth_scale:.5}\n"),
+                style.clone(),
+            ),
+            TextSection::new(
+                format!("Layers: {max_parallax_layer_count:.0}\n"),
+                style.clone(),
+            ),
+            TextSection::new(format!("{parallax_mapping_method}\n"), style.clone()),
+            TextSection::new("\n\n", style.clone()),
+            TextSection::new("Controls\n", style.clone()),
+            TextSection::new("---------------\n", style.clone()),
+            TextSection::new("Left click - Change view angle\n", style.clone()),
+            TextSection::new(
+                "1/2 - Decrease/Increase parallax depth scale\n",
+                style.clone(),
+            ),
+            TextSection::new("3/4 - Decrease/Increase layer count\n", style.clone()),
+            TextSection::new("Space - Switch parallaxing algorithm\n", style),
+        ])
+        .with_style(Style {
+            position_type: PositionType::Absolute,
+            top: Val::Px(10.0),
+            left: Val::Px(10.0),
+            ..default()
+        }),
+    );
+}
+
+/// Store handle of the normal to later modify its format in [`update_normal`].
+#[derive(Resource)]
+struct Normal(Option<Handle<Image>>);
+
+/// Work around the default bevy image loader.
+///
+/// The bevy image loader used by `AssetServer` always loads images in
+/// `Srgb` mode, which is usually what it should do,
+/// but is incompatible with normal maps.
+///
+/// Normal maps require a texture in linear color space,
+/// so we overwrite the format of the normal map we loaded through `AssetServer`
+/// in this system.
+///
+/// Note that this method of conversion is a last resort workaround. You should
+/// get your normal maps from a 3d model file, like gltf.
+///
+/// In this system, we wait until the image is loaded, immediately
+/// change its format and never run the logic afterward.
+fn update_normal(
+    mut already_ran: Local<bool>,
+    mut images: ResMut<Assets<Image>>,
+    normal: Res<Normal>,
+) {
+    if *already_ran {
+        return;
+    }
+    if let Some(normal) = normal.0.as_ref() {
+        if let Some(mut image) = images.get_mut(normal) {
+            image.texture_descriptor.format = TextureFormat::Rgba8Unorm;
+            *already_ran = true;
+        }
+    }
+}

examples/README.md

@@ -119,6 +119,7 @@ Example | Description
 [Lines](../examples/3d/lines.rs) | Create a custom material to draw 3d lines
 [Load glTF](../examples/3d/load_gltf.rs) | Loads and renders a glTF file as a scene
 [Orthographic View](../examples/3d/orthographic.rs) | Shows how to create a 3D orthographic view (for isometric-look in games or CAD applications)
+[Parallax Mapping](../examples/3d/parallax_mapping.rs) | Demonstrates use of a normal map and depth map for parallax mapping
 [Parenting](../examples/3d/parenting.rs) | Demonstrates parent->child relationships and relative transformations
 [Physically Based Rendering](../examples/3d/pbr.rs) | Demonstrates use of Physically Based Rendering (PBR) properties
 [Render to Texture](../examples/3d/render_to_texture.rs) | Shows how to render to a texture, useful for mirrors, UI, or exporting images