xr Full documentation content. setRed(!red)} position={[0, 1, -1]}> } ``` ### Turn any @react-three/fiber app into an XR experience 1. `const store = createXRStore()` create a xr store 2. `store.enterAR()` call enter AR when clicking on a button 3. `...` wrap your content with the XR component ... or read this guide for [converting a react-three/fiber app to XR](../getting-started/convert-to-xr.md). ## Tutorials - ๐Ÿ’พ [Store](../tutorials/store.md) - ๐Ÿ‘† [Interactions](../tutorials/interactions.md) - ๐Ÿ‘Œ [Handles](../handles/introduction.md) - ๐ŸงŠ [Object Detection](../tutorials/object-detection.md) - โœด [Origin](../tutorials/origin.md) - ๐Ÿช„ [Teleport](../tutorials/teleport.md) - ๐Ÿ•น๏ธ [Gamepad](../tutorials/gamepad.md) - โž• [Secondary Input Sources](../tutorials/secondary-input-sources.md) - ๐Ÿ“บ [Layers](../tutorials/layers.md) - ๐ŸŽฎ [Custom Controller/Hands/...](../tutorials/custom-inputs.md) - โš“๏ธ [Anchors](../tutorials/anchors.md) - ๐Ÿ“ฑ [Dom Overlay](../tutorials/dom-overlay.md) - ๐ŸŽฏ [Hit Test](../tutorials/hit-test.md) - โ›จ [Guards](../tutorials/guards.md) - ๐Ÿ–ผ๏ธ [Render Targets](../advanced/render-targets.md) ## External Tutorials - ๐Ÿฅ‡ [**WebXR First Steps React** by Meta Quest](https://github.com/meta-quest/webxr-first-steps-react) ## Roadmap - ๐Ÿคณ XR Gestures - ๐Ÿ•บ Tracked Body ## Migration guides - from [@react-three/xr v5](../migration/from-react-three-xr-5.md) - from [natuerlich](../migration/from-natuerlich.md) ## Sponsors This project is supported by a few companies and individuals building cutting-edge 3D Web & XR experiences. Check them out! ![Sponsors Overview](https://bbohlender.github.io/sponsors/screenshot.png) ]]> ...your scene ``` Lastly, use the `store` to setup the `XR` component to wrap your scene. ```tsx <> ...your scene ``` **Your application is now useable with an AR or VR headset.** If something did not work as expected, check out the [FAQ](../getting-started/faq.md), create an [issue on github](https://github.com/pmndrs/react-xr/issues), or message us on [Discord](https://discord.gg/poimandres). With this basic XR setup, you can start expanding the features of your XR application. The following questions might help you in integrating those features. > How do I move around in my scene? **โ†ณ Checkout out the tutorial about [XROrigin](../tutorials/origin.md) or [Teleportation](../tutorials/teleport.md).** > How can I customize the way my hands/controllers/... feel, look, or interact with the scene? **โ†ณ Check out the tutorial about [Custom Hands/Controllers/...](../tutorials/custom-inputs.md).** > How do interactions work in XR, and how can I build more advanced interactions? **โ†ณ Check out the tutorial about [Interactions](../tutorials/interactions.md)** > How can I leverage the mixed reality features of my headset, such as Plane Detection? **โ†ณ Check out the tutorial about [Object Detection](../tutorials/object-detection.md)** > What else can I do? I need inspiration. **โ†ณ Check out the [examples](./examples.md).** ]]>
  • [![Screenshot from Room demo](./room-demo.gif)](https://pmndrs.github.io/xr/examples/room-with-shadows/)
  • [![Screenshot from Stage demo](./stage-demo.gif)](https://pmndrs.github.io/xr/examples/stage/)
  • [![Screenshot from Ragdoll demo](./ragdoll-demo.gif)](https://pmndrs.github.io/xr/examples/rag-doll/)
  • [![Screenshot from Watch demo](./watch-demo.gif)](https://pmndrs.github.io/xr/examples/watch/)
  • [![Screenshot from Minecraft demo](./minecraft-demo.gif)](https://pmndrs.github.io/xr/examples/minecraft/)
  • [![Screenshot from Pingpong demo](./pingpong-demo.gif)](https://pmndrs.github.io/xr/examples/pingpong/)
  • [![Screenshot from Layers demo](./layers.gif)](https://pmndrs.github.io/xr/examples/layers/)
  • [![Screenshot from Secondary Input Sources demo](./secondary-input-sources.gif)](https://pmndrs.github.io/xr/examples/secondary-input-sources/)
  • [![Screenshot from the React-three-handle Editor demo](./editor.gif)](https://pmndrs.github.io/xr/examples/editor/)
  • [![Screenshot from the hit testing demo](./hit-testing.gif)](https://pmndrs.github.io/xr/examples/hit-testing/) by [Sung Powley](https://bsky.app/profile/sung-powley.bsky.social)
  • [![Screenshot from the uikit + handle demo](./uikit.gif)](https://pmndrs.github.io/xr/examples/uikit/)
  • [![Screenshot from the portal demo](./portal.gif)](https://pmndrs.github.io/xr/examples/portal/)
  • ]]>
  • ![volu.dev](./showcases/volu-dev.gif) [Spatial development hub](https://volu.dev)
  • Are we missing your public product? Please message us via [Twitter](https://x.com/BelaBohlender) or [Discord](https://discord.gg/poimandres).]]>
    console.log(state.camera.getWorldPosition(new Vector3()))) ``` ## How can I change the camera position in XR? In contrast to non-immersive 3D applications, the camera transformation in MR/VR/AR applications should never be directly controlled by the developer since the user's head movement must control the camera's transformation. Therefore, pmndrs/xr provides the XROrigin component, which allows to control where the session's origin is placed inside the 3D scene. The session origin is at the users' feet once they recenter their session. This allows to implicitly control the camera position but prevents the user from getting motion sick when their movement is not reflected in the camera's movement. ## ## Having problems accessing the camera position or rotation. Check if you have OrbitControls, CameraControls from `@react-three/drei`, or other controls in your scene and make sure to place an `` guard around them when in XR or replace them with `OrbitHandles` or `MapHandles` from `@react-three/handle`. This prevents overwriting the camera transformation which is controlled through WebXR when inside an immersive session and allows to access the correct transformation. ```tsx import { OrbitHandles } from '@react-three/handle' import { noEvents, PointerEvents } from '@react-three/xr' ``` ## I cannot enter the XR session! 1. **Missing Https** If you are trying to enter the AR or VR modus and nothing is happening, make sure that you are accessing the website using `https://`. In case you are using vite, we recommend using the `@vitejs/plugin-basic-ssl` to try out your vite application on your device while developing. 2. **Missing XR component** If you made sure that the website is accessed using `https://` and still nothing happens when executing `enterAR` or `enterVR`, it is likely that the `` component is missing. Be sure to add the `` component directly into the `` and make sure both the `` and the `` component are present when the button is pressed. 3. **Entering while loading content** If you cannot enter the VR or AR experience, there might be assets in your scene that are loading. Make sure to place a suspense boundary around your scene. With this setup, the `` component stays mounted while your scene loads. ```tsx ... your scene ``` ## How can I exit an XR session? ```ts store.getState().session?.end() ``` ## Is WebGPU supported? WebGPU is finding its way to more and more devices. However, AR and VR devices do not yet implement WebGPU for WebXR, which requires the [WebXR-WebGPU-Binding](https://github.com/immersive-web/WebXR-WebGPU-Binding/blob/main/explainer.md). Therefore, WebGPU is not yet usable for WebXR in general. ## How can I put HTML in my XR scene? If you are targeting only handheld AR experiences (e.g., for smartphones), you can use dom overlay. Here's a [tutorial for using XRDomOverlays](../tutorials/dom-overlay.md) in your `react-three/xr` experience. For non-handheld VR and AR experiences, you can use [react-three/uikit](https://github.com/pmndrs/uikit), which renders user interfaces directly inside the 3D scene and is aligned with HTML and CSS concepts. ## Does it work on iOS? WebXR for VR experiences is supported on Safari for Apple Vision Pro. WebXR is not supported on iOS Safari yet. The alternative is to use products such as [Variant Launch](https://launch.variant3d.com/), which allow to build WebXR experiences for iOS. ## XRSpace If you are placing `` components outside of the `` while changing the transformation of the `` (e.g. by setting ``), the elements rendered inside of the `` will not be transformed with the origin. If the transformations of the origin should be applied to the ``, make sure to place those components inside the ``. Not placing `` components into the `` can be useful in scenarios where you want to move the `` independently from the ``. For instance, building a virtual elevator where your actual room is duplicated into the x-axis so that you can use the elevator to travel between multiple instances of your room. ## `onClick` does not play video or allow file uploading (in certain browsers) As a performance optimization the react-three/xr event system batches html user events per frame. This only applies if you are using `PointerEvents`, `forwardHtmlEvents`, or `forwardObjectEvents`. This can cause issue when executing functions that require a user action. For instance, uploading a file through a input element in a safari can only be triggered manually when immediately caused by a user input. For these use cases, please disable the event batching performance optimization through the options by setting `batchEvents` to `false`. ]]> ` or allows providing a custom implementation. You can set this for each handedness (e.g., left or right hand) individually. Setting this to `false` prevents the controllers from being used. | `true` | | `transientPointer` | Configures the `` or allows providing a custom implementation. This can be set individually for each handedness. Setting this to `false` prevents transient pointers from being used. | N/A | | `hand` | Configures the `` or allows providing a custom implementation. You can set this individually for each handedness. Setting this to `false` prevents hand tracking from being used. | N/A | | `gaze` | Configures the `` or allows providing a custom gaze implementation. Setting this to `false` prevents gaze-based interaction from being used. | `true` | | `screenInput` | Configures the `` or allows providing a custom screen input implementation. Setting this to `false` prevents screen input from being used. | `true` | | `emulate` | Emulates a specific device (e.g., "metaQuest3") using [IWER](https://github.com/meta-quest/immersive-web-emulation-runtime/) if WebXR is not supported and running on `"localhost"` or pressing `Window/Command + Alt/Option + E`. It can also be set to `false` to disable emulation. | `"metaQuest3"` | | `frameRate` | Sets the session's framerate, with options such as `"high"` for smoother performance. | `"high"` | | `foveation` | Sets the WebXR foveation level between `0` (no foveation) and `1` (maximum foveation). If `undefined`, the device/browser's default foveation setting is used. | `undefined` | | `frameBufferScaling` | Adjusts the framebuffer scaling of the session. If undefined, the device/browser's default scaling is used (typically `1`). | `undefined` | | `enterGrantedSession` | Automatically enters session modes when granted by the system without manually requesting a session. It can be an array of session modes or a boolean value to enable/disable this feature. | `true` | | `baseAssetPath` | Specifies the path to load the controller and hand models, and controller profiles from a CDN or local source. | `'https://cdn.jsdelivr.net/npm/@webxr-input-profiles/assets@1.0/dist/profiles/'` | | `defaultControllerProfileId` | Specifies the fallback profile ID for the controller if no matching profile is found. It is useful for ensuring basic functionality when a specific controller profile isn't available. | `'generic-trigger'` | | `defaultXRHandProfileId` | Specifies the fallback profile ID for hand tracking if no matching profile is found. It ensures basic hand-tracking functionality. | `'generic-hand'` | | `originReferenceSpace` | Defines the reference space type for the origin, such as `'local-floor'` or `'bounded-floor'`. Determines how the user's position is tracked in the XR environment. | N/A | | `bounded` | Enables or disables the session bounds. `false` means unbounded (only available in AR). `true` means bounded (allows to reference the bounding space). `undefined` means bounded but no access to bounding space. | `undefined` | | `anchors` | Enables or disables anchors, which are fixed points in the XR environment that can be used to attach virtual objects. | `true` | | `handTracking` | Enables or turns off hand-tracking in the session, allowing users to interact with the XR environment using their hands. | `true` (`false` for Apple Vision Pro) | | `layers` | Enables or turns off the use of layers in the session, which can enhance rendering performance by stacking visual content. | `true` | | `meshDetection` | Enables or turns off mesh detection, allowing the system to recognize and interact with real-world objects by detecting their mesh. | `true` | | `planeDetection` | Enables or turns off plane detection, allowing the system to recognize flat surfaces like floors and tables. | `true` | | `depthSensing` | Enables or disables depth sensing in the session, which can enhance realism by occluding virtual objects from real-world objects. | `false` | | `customSessionInit` | Overrides the session initialization object with custom settings. Use with caution, as it can significantly alter the behavior of the XR session. | `undefined` | | `hitTest` | Enables or turns off hit testing, which allows the system to detect where the user's input (e.g., a tap or gaze) intersects with objects in the XR environment. | `true` | | `domOverlay` | Enables or turns off DOM overlay in the session or provides a custom DOM element for the overlay, allowing HTML content to be rendered within the XR environment. | `true` | | `secondaryInputSources` | Enables non-primary (secondary input / tracked) sources. For example, when the device supports hands and controllers, the controllers can be used while the hands are tracked as primary input sources. This can allow to use the tracked controllers for other inputs. | `false` | | `offerSession` | If not set to `false`, a session request is automatically send to the browser which can provide a custom ui for the user to start the XR experience. If set to `true` the system will request an `"immersive-ar"` session if supported, else an `"immersive-vr"` session. Alterantively offer session can be directly configured to only enter a `"immersive-vr"` or `"immersive-vr"` session. | `true` | ## Functions The xr store provides a large set of functions to modify and control the xr store. For instance, key functions are the `store.enterAR` and `store.enterVR` functions. The following table gives an overview of the complete set of functions that the xr store provides. | **Function** | **Description** | | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `destroy()` | Irreversibly destroys the XR store. | | `enterXR(mode)` | Initiates an XR session in the specified mode (such as `immersive-ar` or `immersive-vr`). The function returns a promise that eventually resolves with the XR session or `undefined` if the session fails to start. | | `enterAR()` | Starts an Augmented Reality (AR) session. The function returns a promise that eventually resolves with the AR session or `undefined` if AR is not supported. | | `enterVR()` | Starts a Virtual Reality (VR) session. The function returns a promise that eventually resolves with the VR session or `undefined` if VR is not supported. | | `onSessionEnd(callback)` | Subscribes to the session ending, whether through calling `session.end()` or through hardware/OS triggered termination (e.g. the user removing the headset, the browser closing the session, ...). Returns a function to unsubscribe. Prefer this over `session.addEventListener('end', ...)`, since the session reference can go stale across re-renders. | | `setHand(implementation, handedness?)` | Updates the hand tracking configuration or implementation. You can target both hands or specify the handedness (`left` or `right`). This allows customization or updates to the hand implementation or configuration during runtime. | | `setController(implementation, handedness?)` | Updates the controller configuration or implementation. You can target both hands or specify the handedness (`left` or `right`). This enables dynamic updates to the controller setup. | | `setGaze(implementation)` | Updates the gaze-based interaction configuration or implementation. This function is used to modify the gaze implementation or configuration. | | `setScreenInput(implementation)` | Updates the screen input configuration or implementation. This function modifies how screen inputs are handled within the XR session. | | `setTransientPointer(implementation, handedness?)` | Updates the transient pointer configuration or implementation. You can target both hands or specify the handedness. | | `setFrameRate(value)` | Sets the framerate of the XR session, adjusting the session's performance and visual smoothness. Higher framerates can improve user experience but may require more processing power. | | `requestFrame()` | Returns a promise that resolves with the XR frame on the next render. This function is useful for synchronizing actions or processing data in the next render cycle, especially for tasks that need to be aligned with the rendering loop. | ## State Alongside a set of functions, the xr store also provides the state of the current experience. For instance, the state of the xr store contains the current `XRSession` inside `state.session`. The following table provides a list of properties available in the state of the xr store. | **State Property** | **Description** | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `session` | Represents the current `XRSession`. This object contains all the details and state information about the active XR session. | | `originReferenceSpace` | Refers to the `XRReferenceSpace` of the origin in the current session. This space is typically set at the floor level and serves as the reference point for the user's position in the XR environment. | | `origin` | Represents the 3D object that defines the session's origin. If this object is `undefined`, the origin is implicitly set at the world position `(0,0,0)`. | | `domOverlayRoot` | The HTML element used for DOM overlays in handheld AR experiences. This is where any content will be overlayed over the handheld AR session. | | `visibilityState` | Indicates the session's visibility state, such as `"visible-blurred"` when the user sees an Operating System overlay. This property helps manage how the XR experience adjusts to different visibility conditions. | | `frameRate` | Represents the configured framerate for the XR session. Note that the actual framerate might be lower if the system cannot maintain the desired performance level. | | `mode` | Specifies the current XR session mode, such as `immersive-vr`, `immersive-ar`, or `inline`. If no session is active, this will be `null`. | | `detectedPlanes` | A read-only array of `XRPlane` objects representing the planes detected in the XR environment. These could include surfaces like floors, walls, and tables. | | `detectedMeshes` | A read-only array of `XRMesh` objects representing the meshes detected in the XR environment. These are typically 3D objects that have been identified and tracked during the session. | ## useXR The `useXR` hook allows to retrieve the state from the xr store. For instance, `const session = useXR(xr => xr.session)` allows us to always get the current session from any component that is placed inside the `` component. ]]> console.log("I've been clicked", event)}> ``` The `event` object provided to the onClick handler contains useful information, such as the intersection `point` in world space. The way pointer events are handled can be configured using the `pointerEvents`, `pointerEventsType`, and `pointerEventsOrder` properties, which are available on all threejs objects. The `pointerEvents` property corresponds to the `pointerEvents` property of CSS, which allows to completely disable pointer events for an element and its children. However, children can also re-enable pointer events by setting `pointerEvents="auto"`. The `pointerEventsType` property allows to blacklist or whitelist pointer events for specific pointer types. For instance, setting `pointerEventsType={{ deny: "grab" }}` prevents triggering pointer events from grabbing the object or any of its children. The `pointerEventsOrder` allows to overwrite the sorting order, similar to how `renderOrder` allows to overwrite the rendering order in threejs. The default pointer events order is `0`. Setting it to a value greater than `0` will ensure it is intersected before anything with a lower pointer events order. Setting `pointerEventsOrder` is helpful for building an interactive x-ray object that is always rendered above anything else and should, therefore, always be interacted with first. For instance, this can be used to build controls that are overlayed over the object that they control. ## Pointer Capture Another concept that @react-three/fiber leverages from the web is pointer captures. Pointer captures allow to force all consecutive events to be emitted to a specific object, even if that object is not intersected. This is useful for building dragging interactions without a complex global state. Typically, a pointer capture is set using `object.setPointerCapture` in the event handler of `onPointerDown` with the `pointerId` of the pointer that pressed on the object. .The following example illustrates how pointer events can be built to create a simple dragging implementation (that only works if the mesh is not inside a transformed group). ```tsx function DraggableCube() { const isDraggingRef = useRef(false) const meshRef = useRef(null) return ( { if (isDraggingRef.current) { return } isDraggingRef.current = true meshRef.position.copy(e.point) }} onPointerMove={(e) => { if (!isDraggingRef.current) { return } meshRef.position.copy(e.point) }} onPointerUp={(e) => (isDraggingRef.current = false)} > ) } ``` ]]> xr.detectPlanes)` and manually go through the returned array. To render the planes in the correct place, the planes' space must provided to the `XRSpace` component. The following example shows how to render the red planes for all detected walls. ```tsx function RedWalls() { const wallPlanes = useXRPlanes('wall') return ( <> {wallPlanes.map((plane) => ( ))} ) } ``` ## Detected Meshes Mesh detection provides access to the geometry of the environment. Similarly to xr planes, @react-three/fiber allows to retrieve detected meshes using `useXRMeshes` and offers the `XRMeshModel` to render the individual meshes. ]]> ) } function RollerCoaster() { const gltf = useGLTF('rollercoaster.glb') const mixer = useMemo(() => new AnimationMixer(gltf.scene), []) useEffect(() => { for (const animation of gltf.animations) { mixer.clipAction(animation).play() } }, [gltf, mixer]) useFrame((state, delta) => mixer.update(delta)) return ( <> {createPortal( , gltf.scene.getObjectByName('Sessel')!, )} ) } ``` ## Resizing example Transforming the `XROrigin` is not limited to position but can also include rotation and scale. The following example shows how the `XROrigin` can be used to achieve a resizing interaction. ```tsx const store = createXRStore() function App() { const [miniature, setMinitature] = useState(false) return ( <> ) } ``` ]]> ``` Lastly, we need to add a teleport target to our scene. In this case, we're using a simple 10x10 meter green box. We need to bind our `setPosition` function to the `onTeleport` handler of the `TeleportTarget` to update the user's position whenever the user teleports. ```tsx ``` Combined, this example looks like this. ```tsx const store = createXRStore({ hand: { teleportPointer: true }, controller: { teleportPointer: true }, }) export function App() { const [position, setPosition] = useState(new Vector3()) return ( <> ) } ``` ![Recording of teleport example](./teleport-example.gif)]]> ) } function Locomotion() { const controller = useXRInputSourceState('controller', 'right') const ref = useRef(null) useFrame((_, delta) => { if (ref.current == null || controller == null) { return } const thumstickState = controller.gamepad['xr-standard-thumbstick'] if (thumstickState == null) { return } ref.current.position.x += (thumstickState.xAxis ?? 0) * delta ref.current.position.z += (thumstickState.yAxis ?? 0) * delta }) return } ``` ]]> { const { isPrimary } = useXRInputSourceStateContext('controller') if (isPrimary) { return } return ( ) }, }) ``` ]]> video.play()} scale={0.5} src={video} /> ``` The assigned video is an HTML video element that is loaded from `test.mp4`. ```tsx const video = useMemo(() => { const result = document.createElement('video') result.src = 'test.mp4' return result }, []) ``` Combined, the final app looks like this ```tsx export function App() { const video = useMemo(() => { const result = document.createElement('video') result.src = 'test.mp4' return result }, []) return ( video.play()} scale={0.5} src={video} /> ) } ``` Instead of images and videos, Layers can also be used to display dynamically rendered content. The following example illustrates how to render a red cube onto the layer. This scene will be re-rendered every frame, allowing for fully dynamic content. ```tsx ``` ]]> (null) const pointer = useTouchPointer(middleFingerRef, state) ``` Next, we use the `state` to place an `XRSpace` for setting up the `middleFingerRef` and add an `XRHandModel` and `PointerCursorModel` to render the hand and a cursor visualization. ```tsx ```
    Full Code ```tsx export function CustomHand() { const state = useXRInputSourceStateContext('hand') const middleFingerRef = useRef(null) const pointer = useTouchPointer(middleFingerRef, state) return ( <> ) } ```
    This tutorial also applies to building custom controllers, transient pointers, gaze, and screen input implementations. ]]>
    ` component. ```tsx ...your content ``` The following example shows a `Anchor` component that uses the `useXRAnchor` hook and the `XRSpace` component to anchor a Box to the position of the right hand or controller when the respective hand or controller is selected (pinch/trigger). ```tsx export function Anchor() { const [anchor, requestAnchor] = useXRAnchor() const controllerState = useXRInputSourceState('controller', 'right') const handState = useXRInputSourceState('hand', 'right') const inputSource = controllerState?.inputSource ?? handState?.inputSource useXRInputSourceEvent( inputSource, 'select', async () => { if (inputSource == null) { return } requestAnchor({ relativeTo: 'space', space: inputSource.targetRaySpace }) }, [requestAnchor, inputSource], ) if (anchor == null) { return null } return ( ) } ``` ]]>
    Hello World
    ``` The following shows the complete code for a simple AR experience with a `Hello World` button that can toggle its color when clicked on. ```tsx const store = createXRStore() export function App() { const [bool, setBool] = useState(false) return ( <>
    setBool((b) => !b)} > Hello World
    ) } ``` ]]>
    (null) useXRHitTest( (results, getWorldMatrix) => { if (results.length === 0) return getWorldMatrix(matrixHelper, results[0]) hitTestPosition.setFromMatrixPosition(matrixHelper) }, 'viewer', // Cast rays from the viewer reference space. This will typically be either the camera or where the user is looking 'plane' // Only hit test against detected planes ) useFrame(() => { if (hitTestPosition && previewRef.current) { previewRef.current.position.copy(hitTestPosition) } }) return ( {/* Renders a sphere where the hit test intersects with the plane */} ) } ``` ## XRHitTest `XRHitTest` is a component that wraps the `useXRHitTest` hook. This makes it easier to add hit testing anywhere within your component tree. ```tsx const matrixHelper = new Matrix4() const hitTestPosition = new Vector3() const store = createXRStore({ hand: () => { const inputSourceState = useXRInputSourceStateContext() return ( <> { if (results.length === 0) return getWorldMatrix(matrixHelper, results[0]) hitTestPosition.setFromMatrixPosition(matrixHelper) }} /> ) }, }) ``` `XRHitTest` has all of the same functionality as the `useXRHitTest` hook, just that it's built as a component. ## useXRHitTestSource Hook The `useXRHitTestSource` hook provides lower-level access to hit test sources, giving you more control over when and how hit tests are performed. It is the same as the `useXRHitTest` hook, the only difference being that you have to manually check for hit test results; typically every frame, or every few frames. **What it does:** Does the same thing as the `useXRHitTest` hook, but does not automatically hit test every frame. **When to use it:** In most cases you should use either `useXRHitTest` or `useXRRequestHitTest`, but you can use this hook when you have a static hit test source that you only want to occasionally perform constant hit tests from. Or if you want to recreate the `useXRHitTest` behavior manually. **Parameters:** - `relativeTo` - The object, XR space, or reference space to cast rays from - `trackableType` - Optional parameter specifying what types of surfaces to hit test against **Returns:** A hit test source object that you can use with `frame.getHitTestResults()` ```tsx function ManualHitTest() { const meshRef = useRef(null) const hitTestSource = useXRHitTestSource(meshRef) const [someCondition, setSomeCondition] = useState(false) const [hitResults, setHitResults] = useState([]) useFrame((_, __, frame: XRFrame | undefined) => { // Only perform hit testing when certain conditions are met if (frame && hitTestSource && someCondition) { const results = frame.getHitTestResults(hitTestSource.source) setHitResults(results) } }) return ( {/* Render hit test results. This will put spheres everywhere the hit test succeeds. In a real app don't use index as the key */} {hitResults.map((result, index) => { const matrix = new Matrix4() hitTestSource?.getWorldMatrix(matrix, result) const position = new Vector3().setFromMatrixPosition(matrix) return ( ) })} ) } ``` ## useXRRequestHitTest Hook The `useXRRequestHitTest` hook provides a function for one-time hit test requests. Useful for event-driven hit testing. Cannot be called in the `useFrame` hook. **What it does:** Returns a function that can perform a single hit test request when called. **When to use it:** Use this for event-driven hit testing, such as when a user taps the screen, clicks a button, or performs a gesture. It's ideal for placing objects or checking intersections at specific moments. **Returns:** A function that takes the same parameters as other hit test hooks and returns a promise with hit test results ```tsx const matrixHelper = new Matrix4() function EventDrivenHitTest() { const requestHitTest = useXRRequestHitTest() const [placedObjects, setPlacedObjects] = useState([]) const handleTap = async () => { const hitTestResult = await requestHitTest('viewer', ['plane', 'mesh']) const { results, getWorldMatrix } = hitTestResult if (results?.length > 0) { getWorldMatrix(matrixHelper, results[0]) const position = new Vector3().setFromMatrixPosition(matrixHelper) setPlacedObjects((prev) => [...prev, position]) } } return ( <> {/* Render placed objects */} {placedObjects.map((position, index) => ( ))} ) } ``` ## Trackable Types All hit testing hooks support specifying trackable types to control what surfaces the hit tests should target: - `'plane'` - Hit test against detected planes (floors, walls, tables) - `'point'` - Hit test against feature points in the environment - `'mesh'` - Hit test against detected meshes (requires mesh detection support) You can specify a single type or an array of types: ```tsx // Single type useXRHitTest(callback, spaceRef, 'plane') // Multiple types useXRHitTest(callback, spaceRef, ['plane', 'mesh']) ``` ## Practical Example: Object Placement Here's a complete example combining multiple hooks for a robust object placement system: ```tsx const matrixHelper = new Matrix4() const hitTestPositionHelper = new Vector3() function ObjectPlacement() { const [placedObjects, setPlacedObjects] = useState([]) const [previewPosition, setPreviewPosition] = useState(null) const controllerRef = useRef(null) // Continuous hit testing for preview useXRHitTest( (results, getWorldMatrix) => { if (results.length > 0) { getWorldMatrix(matrixHelper, results[0]) const position = hitTestPositionHelper.setFromMatrixPosition(matrixHelper) setPreviewPosition(position) } else { setPreviewPosition(null) } }, 'viewer', // Use viewer space for screen-based hit testing ) const placeObject = async () => { if (previewPosition) { setPlacedObjects((prev) => [...prev, previewPosition.clone()]) } } return ( <> {/* Preview object at hit test position */} {previewPosition && ( )} {/* Placed objects */} {placedObjects.map((position, index) => ( ))} {/* Placement trigger */} ) } ``` Alternatively, for devices that provide mesh detection -- such as newer Meta Quest devices -- you can also add normal pointer event listeners to an XR Mesh to achieve the same behavior. Check out [this tutorial](./object-detection.md) for more information about mesh detection. ]]> ) } ``` ]]> new WebGLRenderTarget(1024, 1024), []) const scene = useMemo(() => new Scene(), []) const camera = useMemo(() => new PerspectiveCamera(50, 1, 0.1, 100), []) useFrame(() => { const previousTarget = gl.getRenderTarget() const previousXrEnabled = gl.xr.enabled const previousIsPresenting = gl.xr.isPresenting gl.getViewport(viewport) gl.xr.enabled = false gl.xr.isPresenting = false gl.setRenderTarget(target) gl.setViewport(0, 0, target.width, target.height) gl.render(scene, camera) gl.setRenderTarget(previousTarget) gl.setViewport(viewport) gl.xr.enabled = previousXrEnabled gl.xr.isPresenting = previousIsPresenting }) return null } ``` If you are rendering content for an `XRLayer`, prefer the layer API instead of wiring this manually. `XRLayer` accepts children for dynamic content and uses the same state-preserving offscreen render pattern internally. ```tsx ``` Use a manual render target when you need the resulting texture for a material, post-processing pass, portal, minimap, or another non-layer effect. Use `XRLayer` when the goal is to display high-quality flat, cylinder, or equirect content in XR. ]]> ` component and the door body with the `` component. ```tsx {6,10,14-15} export function Door() { const { nodes, materials } = useGLTF('/door.glb') return ( ) } ``` Next, we need to configure the handle to rotate the door when grabbed. We instruct it to use the target from the context using `targetRef="from-context"`, making sure the transformations are applied to the door body. Additionally, we ensure that moving the handle is translated into a rotation using the `translate="as-rotate"` property. Lastly, we disable rotations on all axes except for the `z` axis and limit the rotation between -180ยฐ and 0ยฐ. *Learn more about all the available properties for the handle component [here](./handle-component.md).* The final result looks like this (I added an additional handle around the door handle that allows it to rotate on its own y-axis). ![](./door-handle.gif) ### Editor Example Handles are made for all kinds of use cases, from games to professional applications, which we emphasized by building the following editor demo that uses over 40 handles for moving the elements on the screen, resizing the virtual screen, and moving the virtual camera. The nice part is that it works on all devices, ranging from mouse-driven PCs to eye-driven mixed-reality headsets (Apple Vision Pro). ![](./editor.gif) You can check it out [here](https://pmndrs.github.io/xr/examples/editor/) and read the source code (700 LOC) [here](https://github.com/pmndrs/xr/tree/main/examples/editor/app.tsx). ### Screen Handles A specific type of handle is the screen handle, where not an individual object is the handle, but the whole screen. Therefore, we build screen handles, which allow panning, zooming, and rotating the camera using swipe, drag, and scroll interactions. These are built on the ideas of `OrbitControls` and `MapControls` from Three.js, but respect the event system (e.g., you don't need to disable them when you drag an object) and are automatically forwarded to virtual screens, as shown in the editor demo. Learn more about the available screen handles [here](./screen-handle-components.md). ### Prebuilt Handles For many use cases, such as 3D editors, handles often come in specific forms, like the `TransformControls` available in Three.js. These opinionated pre-built handles have proven to be very useful, which is why `@react-three/handle` ships with implementations for `TransformHandles` and `PivotHandles`. ![](./prebuild-handles.gif) Learn more about the available prebuilt handles [here](./screen-handle-components.md). ## Sponsors This project is supported by a few companies and individuals building cutting-edge 3D Web & XR experiences. Check them out! ![Sponsors Overview](https://bbohlender.github.io/sponsors/screenshot.png)]]> ``` *Allows scaling of the cube by dragging it outward from the cube's center.* ### Properties **handleRef** Allows overriding to pass a custom handle object. **targetRef** Allows to pass in a ref to an object that should be the target of the handle. Alternatively `targetRef` can be set to `"from-context"` use the target provided by a surrounding `HandleTarget` component. **getHandleOptions** Allows passing a function that dynamically generates options to override the current handle options. **bind** Allows disabling automatic binding of the event listeners to the provided handle, which can be necessary when capturing pointers manually. **apply** The `apply` function is used to apply a state modification that originates from a user interaction to the state. This property allows overriding the default apply function, giving the developer complete control over how modifications affect the state. For instance, instead of applying the modification directly, the developer can apply it to their own state management solution. The state management solution can then apply the modification to the handle target. **projectRays** Allows to configure whether rays from input devices should be projected onto the interaction space (3D plane or 3D Line). **alwaysUpdate** In situations where the handle target is placed inside a constantly changing group, the `alwaysUpdate` flag ensures that the handle target's transformation is updated every frame to reflect the current state of the handle. **multitouch** By default, handles can be interacted with using multiple input devices. By setting `multitouch` to `false`, only the first input device will be used. **filter** Allows to filter interactions based on the event. Return `false` to ignore the event. **stopPropagation** By default, events that occur on handles are not propagated upwards and therefore do not reach their ancestors. Setting `stopPropagation` to `false` will re-enable event propagation for events that occur on the handle. **rotate** The `rotate` property allows configuring if and how the user can rotate the target. Setting `rotate` to `false` disables rotation. Setting `rotate` to `x` restricts rotation to the x-axis. Setting `rotate` to `{ x: false, y: [0, Math.PI] }` disables rotation on the x-axis and restricts rotation on the y-axis to be between 0 and 180ยฐ, while rotation on the z-axis is enabled. **scale** The `scale` property allows configuring if and how the user can scale the target. Setting `scale` to `false` disables scaling. Setting `scale` to `x` restricts scaling to the x-axis. Setting `scale` to `{ x: false, y: [1, 2] }` disables scaling on the x-axis and restricts the scaling factor on the y-axis to be between 1 and 2, while scaling on the z-axis is enabled. **translate** The `translate` property allows configuring if and how the user can translate the target. Setting `translate` to `false` disables translation. Setting `translate` to `x` restricts translation to the x-axis. Setting `translate` to `{ x: false, y: [-1, 1] }` disables translation on the x-axis and restricts translation on the y-axis to be between -1 and 1, while translation on the z-axis is enabled. Furthermore, the `translate` property can be configured to transform translations into rotations and/or scalings using `translate="as-scale"`, allowing the user to scale the target by grabbing and moving the handle. **ref** Allows retrieval of a reference to the internal handle store (``). ## Handle Target Component The `HandleTarget` component allows declaratively specifying a handle target that is hierarchically above the `Handle` component. To prevent accidentally providing a different target to a handle, using the target from the context requires setting `targetRef="from-context"` on the `Handle` component. **Example** ```tsx ```]]> ...` instead of `XRCanvas` - configure settings such as `foveation` through `createXRStore` - use `store.enterXR` instead of `useEnterXR` - use `DragControls` **TBD** instead of `Grabbale` - don't add hands and controllers yourself, and configure them through the `createXRStore` options. Click [here](../tutorials/custom-inputs.md) for more info regarding controller/hand/... customization. - use teleport as described [here](../tutorials/teleport.md) ]]> ``` ## Transform Handles *Alias for `TransformControls`* ![](./transform.gif) ### Properties **alwaysUpdate** In situations where the transform handles are placed inside a constantly changing group, the `alwaysUpdate` flag ensures that the transform handles' transformation is updated every frame to reflect the current state of the handle. **apply** Allows overriding the default apply function, giving the developer complete control over how modifications affect the state. For instance, instead of applying the modification directly, the developer can apply it to their own state management solution. The state management solution can then apply the modification to the handle target. **stopPropagation** Setting `stopPropagation` to `false` will re-enable event propagation for events that occur on the handles. **space** Allows configuring whether the transformations should happen in `"local"` or `"world"` space. This property has no effect when the `mode` property is set to `scale`, as scaling must occur on the local axis. **mode** Allows configuring whether the transformation should be `"translate"`, `"rotate"`, or `"scale"`, which also changes the visualization of the transform handles. **x** Allows configuring the transformation on the x-axis. Setting `x` to `false` disables transformations on the x-axis and also hides the respective user interface. **y** Allows configuring the transformation on the y-axis. Setting `y` to `false` disables transformations on the y-axis and also hides the respective user interface. **z** Allows configuring the transformation on the z-axis. Setting `z` to `false` disables transformations on the z-axis and also hides the respective user interface. **e** The `e` axis represents the axis for rotating the transform handles in screen space, which is only available when `mode` is set to `rotation`. Setting `e` to `false` disables rotation in screen space and also hides the respective user interface. **enabled** Setting `enabled` to `false` momentarily disables the transform handles. **fixed** By default, the transform handles have a fixed size independent of their distance from the camera, which means they scale up when they move away from the camera. Setting `fixed` to `false` will make them appear smaller when further away from the camera. **size** The `size` property allows configuring the size of the transform handles, which has no effect on their contents. ## Pivot Handles *Alias for `PivotControls`* In contrast to the transform handles, the pivot handles only operate in local space but allow rotation, scaling, and translation transformations simultaneously. ![](./pivot.gif) ### Properties **scale** The `scale` property allows configuring if and how the user can scale the pivot handles. Setting `scale` to `false` disables scaling. Setting `scale` to `x` restricts scaling to the x-axis and only shows the user interface elements for scaling on the x-axis. Similarly, setting `scale` to `{ x: false }` hides the user interface elements for scaling on the x-axis and only allows scaling on the y- and z-axes. When scaling is allowed on all axes, a uniform scale handle is shown in addition to the per-axis handles. Scale limits such as `{ x: [0.5, 2] }` are also respected by the uniform scale handle. **translation** The `translation` property allows configuring if and how the user can translate the pivot handles. Setting `translation` to `false` disables translation. Setting `translation` to `x` restricts translation to the x-axis and only shows the user interface elements for translation on the x-axis. Similarly, setting `translation` to `{ x: false }` hides the user interface elements for translation on the x-axis and only allows translation on the y- and z-axes. **rotation** The `rotation` property allows configuring if and how the user can rotate the pivot handles. Setting `rotation` to `false` disables rotation. Setting `rotation` to `x` restricts rotation to the x-axis and only shows the user interface elements for rotating on the x-axis. Similarly, setting `rotation` to `{ x: false }` hides the user interface elements for rotating on the x-axis and only allows rotation on the y- and z-axes. **alwaysUpdate** In situations where the pivot handles are placed inside a constantly changing group, the `alwaysUpdate` flag ensures that the pivot handles' transformation is updated every frame to reflect the current state of the handle. **apply** Allows overriding the default apply function, giving the developer complete control over how modifications affect the state. For instance, instead of applying the modification directly, the developer can apply it to their own state management solution. The state management solution can then apply the modification to the handle target. **stopPropagation** Setting `stopPropagation` to `false` will re-enable event propagation for events that occur on the handles. **enabled** Setting `enabled` to `false` momentarily disables the pivot handles. **fixed** By default, the pivot handles have a fixed size independent of their distance from the camera, which means they scale up when they move away from the camera. Setting `fixed` to `false` will make them appear smaller when further away from the camera. **size** The `size` property allows configuring the size of the pivot handles, which has no effect on their contents.]]> [!CAUTION] > Deprecated: use ` ) } ``` ## See - [Hit Test Tutorial](https://pmndrs.github.io/xr/docs/tutorials/hit-test) - [Hit Test Example](https://pmndrs.github.io/xr/examples/hit-testing/) ]]> **useXRInputSourceEvent**(`inputSource`, `event`, `fn`, `deps`): `void` Hook for listening to xr input source events ## Parameters **inputSource** The input source to listen to, or 'all' to listen to all input sources `undefined` | `XRInputSource` | `"all"` **event** The event to listen to. ([List of events](https://developer.mozilla.org/en-US/docs/Web/API/XRInputSourceEvent)) `"squeeze"` | `"selectstart"` | `"select"` | `"selectend"` | `"squeezestart"` | `"squeezeend"` **fn** (`event`) => `void` Callback function called when the event is triggered. **deps** `any`[] Retriggers the binding of the event when the dependencies change. ## Returns --- `void` ]]> **useXRMeshes**(`semanticLabel?`): readonly `XRMesh`[] Hook for getting all detected meshes with the provided semantic label ## Parameters **semanticLabel?** `string` ## Returns --- readonly `XRMesh`[] ]]> **useXRMeshGeometry**(`mesh`, `disposeBuffer`): `BufferGeometry` Hook for getting the geometry from the detected mesh ## Parameters **mesh** `XRMesh` the detected mesh **disposeBuffer** `boolean` = `true` allows to disable auto disposing the geometry buffer ## Returns --- `BufferGeometry` ]]> **useXRPlaneGeometry**(`plane`, `disposeBuffer`): `BufferGeometry` Hook for getting the geometry from the detected plane ## Parameters **plane** `XRPlane` the detected plane **disposeBuffer** `boolean` = `true` allows to disable auto disposing the geometry buffer ## Returns --- `BufferGeometry` ]]> **useXRPlanes**(`semanticLabel?`): readonly `XRPlane`[] Hook for getting all dected planes with the provided semantic label ## Parameters **semanticLabel?** `string` ## Returns --- readonly `XRPlane`[] ]]> [!CAUTION] > Deprecated: use `useXRSpace` instead > `const` **useXRReferenceSpace**: \{(): `XRSpace`; (`type`): `undefined` \| `XRReferenceSpace`; (`type`): `undefined` \| `XRSpace`; \} = `useXRSpace` ## Call Signature > (): `XRSpace` Hook for retrieving XR space from the context **Returns** `XRSpace` ## Call Signature > (`type`): `undefined` \| `XRReferenceSpace` Hook for retrieving XR space from the context **Parameters** **type** `XRReferenceSpaceType` **Returns** `undefined` \| `XRReferenceSpace` ## Call Signature > (`type`): `undefined` \| `XRSpace` Hook for retrieving XR space from the context **Parameters** **type** `XRSpaceType` **Returns** `undefined` \| `XRSpace` ]]> **useXRRequestHitTest**(): (`relativeTo`, `trackableType?`) => `undefined` \| `Promise`\<`undefined` \| \{ `getWorldMatrix`: (...`args`) => `boolean`; `results`: `XRHitTestResult`[]; \}\> Hook that returns a function to request a single hit test. Cannot be called in the useFrame hook. ## Returns --- > (`relativeTo`, `trackableType?`): `undefined` \| `Promise`\<`undefined` \| \{ `getWorldMatrix`: (...`args`) => `boolean`; `results`: `XRHitTestResult`[]; \}\> **Parameters** **relativeTo** `XRSpace` | `XRReferenceSpaceType` | `RefObject`\<`null` \| `Object3D`\<`Object3DEventMap`\>\> **trackableType?** `XRHitTestTrackableType` | `XRHitTestTrackableType`[] **Returns** `undefined` \| `Promise`\<`undefined` \| \{ `getWorldMatrix`: (...`args`) => `boolean`; `results`: `XRHitTestResult`[]; \}\> ## Example ```ts const matrixHelper = new Matrix4() function EventDrivenHitTest() { const requestHitTest = useXRRequestHitTest() const [placedObjects, setPlacedObjects] = useState([]) const handleTap = async () => { const hitTestResult = await requestHitTest('viewer', ['plane', 'mesh']) const { results, getWorldMatrix } = hitTestResult if (results?.length > 0) { getWorldMatrix(matrixHelper, results[0]) const position = new Vector3().setFromMatrixPosition(matrixHelper) setPlacedObjects((prev) => [...prev, position]) } } return ( <> {placedObjects.map((position, index) => ( ))} ) } ``` ## See - [Hit Test Tutorial](https://pmndrs.github.io/xr/docs/tutorials/hit-test) - [Hit Test Example](https://pmndrs.github.io/xr/examples/hit-testing/) ]]> [!CAUTION] > Deprecated: `useXRInputSourceStateContext("screenInput")` instead > **useXRScreenInputState**(): `XRScreenInputState` Hook for getting the screen-input state ## Returns --- `XRScreenInputState` ]]> **useXRSessionFeatureEnabled**(`feature`): `boolean` Checks if a specific XR session feature is enabled. ## Parameters **feature** `string` The XR session feature to check against. ## Returns --- `boolean` Whether the feature is enabled. ]]> **useXRSessionModeSupported**(`mode`, `onError?`): `undefined` \| `boolean` Checks whether a specific XRSessionMode is supported or not ## Parameters **mode** `XRSessionMode` The `XRSessionMode` to check against. **onError?** (`error`) => `void` Callback executed when an error occurs. ## Returns --- `undefined` \| `boolean` ]]> **useXRSessionVisibilityState**(): `undefined` \| `XRVisibilityState` Gets the visibility state of the XR session. ## Returns --- `undefined` \| `XRVisibilityState` The visibility state of the XR session. ]]> **useXRSpace**(): `XRSpace` Hook for retrieving XR space from the context **Returns** `XRSpace` ## Call Signature > **useXRSpace**(`type`): `undefined` \| `XRReferenceSpace` Hook for retrieving XR space from the context **Parameters** **type** `XRReferenceSpaceType` **Returns** `undefined` \| `XRReferenceSpace` ## Call Signature > **useXRSpace**(`type`): `undefined` \| `XRSpace` Hook for retrieving XR space from the context **Parameters** **type** `XRSpaceType` **Returns** `undefined` \| `XRSpace` ]]> **useXRStore**(): `XRStore` Hook for getting the xr store from the context ## Returns --- `XRStore` ]]> [!CAUTION] > Deprecated: use `useXRInputSourceState("transientPointer", "left")` instead ## Call Signature > **useXRTransientPointerState**(`handedness`): `undefined` \| `XRTransientPointerState` Hook for getting the transient-pointer state **Parameters** **handedness** `XRHandedness` the handedness that the XRHandState should have **Returns** `undefined` \| `XRTransientPointerState` # ## Call Signature > **useXRTransientPointerState**(): `XRTransientPointerState` Hook for getting the transient-pointer state inside the xr store config **Returns** `XRTransientPointerState` #]]> [!CAUTION] > Deprecated: use `