Masks, the triangle filter and materials

Each instance has an 8-bit mask, 0xff by default, as in DXR. A masked trace skips an instance when (instanceMask & rayMask) == 0, at the top level, before it enters the instance's tree.

// 8-bit instance masks (default 0xff) and a material index per instance (0-65534).
trees.addInstance({ mesh, objectToWorld: matrix, mask: 0b0000_0011, material: 12 });
// A masked trace skips an instance when (instanceMask & rayMask) == 0.
// hit.material is 12 for this instance's hits, BVH_NO_MATERIAL where none was given.

// No *Masked functions, no mask test in the walk, every instance visible.
const code = traceModuleWgsl({ rayMasks: false });

In three, layers 0-7 become mask bits 0-7 (instanceMaskFromLayers), and rayMaskFromLayers(camera.layers) gives a ray mask. Layers 8-31 are dropped.

A triangle whose flags bit 0 (TRIANGLE_FLAG_FILTER) is set asks a WGSL function before its hit is taken:

fn bvhTrianglePasses(candidate: BvhCandidate) -> bool

true means the ray passes through: nothing is recorded and the closest distance stays, so what lies behind is still found. Unflagged triangles never ask, so a scene pays only for the triangles that need it. Use it for alpha-tested foliage, glass that casts no shadow, or one-sided surfaces.

The candidate holds instance, triangle (the original index), t, barycentrics, frontFace, query (BVH_QUERY_CLOSEST or BVH_QUERY_ANY) and rayMask. It is a struct, so fields can be added without breaking filters.

The filter must not trace or query (bvhTraceAny, bvhClosestPoint, bvhIntersects*, bvhShapecast): a nested walk overwrites state the outer walk keeps in private variables (the node frame of quantized nodes), and the outer walk would read the rest of its instance wrongly. A shapecast's triangle hook may nest; see Spatial queries on the GPU.

    // Asked only for triangles with TRIANGLE_FLAG_FILTER set. true: the ray passes through.
    const triangleFilter = /* wgsl */ `
fn bvhTrianglePasses(candidate: BvhCandidate) -> bool {
    // Shadow rays pass through flagged triangles (foliage cards, glass); closest-hit rays still see them.
    return candidate.query == BVH_QUERY_ANY;
}`;
    const code = traceModuleWgsl({ triangleFilter });

The default filter, DEFAULT_TRIANGLE_FILTER_WGSL, lets nothing through. triangleFilter: null leaves the function out, for a kernel that declares bvhTrianglePasses itself (to read a texture, for example).

Flag the triangles when you add the mesh:

import { TRIANGLE_FLAG_FILTER, type MeshGeometry } from '@pmndrs/bvh';

const triangleCount = indices.length / 3;
const triangleFlags = new Uint32Array(triangleCount);
for (let triangle = 0; triangle < triangleCount; triangle++) if (isGlass(triangle)) triangleFlags[triangle] = TRIANGLE_FLAG_FILTER;

const window: MeshGeometry = { positions, indices, triangleFlags }; // only bit 0 may be set
const mesh = trees.addMesh(window);

For a scene assembled with buildTraceScene, writeTriangleFilterFlags(scene, (originalIndex, sourceInstance) => ..., scene.meshes[i]) writes the flags into the scene's own buffer after assembly. The CPU walk and the reference tracer take the same filter as a TypeScript function (TrianglePasses).

Each instance can carry a material index, 0 to 65534. It comes back as hit.material (and as word 6 of the batch kernel's 40-byte hit record). BVH_NO_MATERIAL (0xffffffff, NO_MATERIAL in TypeScript) means none was given, and is what a miss reports.

A mesh can carry a material per triangle instead: MeshInput.triangleMaterials, one value per original triangle, 0 to 65534, or TRIANGLE_MATERIAL_NONE (0xffff) for none. A hit's material is its triangle's when it has one, else its instance's, else BVH_NO_MATERIAL.

import { TRIANGLE_MATERIAL_NONE } from '@pmndrs/bvh';

// One material per triangle (0-65534), by original triangle; TRIANGLE_MATERIAL_NONE (0xffff) takes the instance's.
const triangleMaterials = new Uint16Array(indices.length / 3).fill(TRIANGLE_MATERIAL_NONE);
triangleMaterials.fill(3, 0, 100); // triangles 0-99 are material 3
const mesh = trees.addMesh({ positions, indices, triangleMaterials });
trees.addInstance({ mesh, objectToWorld: matrix, material: 7 });
// hit.material: 3 on triangles 0-99, 7 (the instance's) on the others.

The hit also has normal: the world-space geometric normal of the authored front side, unit length. Interpolated normals need your own attributes, through hit.triangle and hit.barycentrics.