Augmented and Virtual Reality

Learn how to add AR and VR support to your games.

This tutorial shows how to add Augmented Reality (AR) and Virtual Reality (VR) support to your @react-three/viverse games. We'll start with AR and then show the additional changes needed for VR.

Prerequisites

Make sure you have the @react-three/xr package installed:

npm install @react-three/xr

Augmented Reality (AR)

Let's start by adding AR support to a basic @react-three/viverse game.

Here's the AR app we'll build now:

import { Canvas } from '@react-three/fiber'
import { Viverse } from '@react-three/viverse'
import { XR, XROrigin, createXRStore } from '@react-three/xr'
import { Scene } from './Scene'

const store = createXRStore({ offerSession: 'immersive-ar' })

export default function App() {
return (
  <Viverse>
    <Canvas
      style={{ position: "absolute", inset: "0", touchAction: "none" }}
      camera={{ fov: 90, position: [0, 2, 2] }}
      shadows
      gl={{ antialias: true, localClippingEnabled: true }}
    >
      <XR store={store}>
          <XROrigin scale={10} position-y={-8} position-z={10} />
          <Scene />
      </XR>
    </Canvas>
  </Viverse>
)
}

Open Sandbox

Step 1: Set Up the XR Store

Create an XR store configured for AR. Add these imports and create the store:

import { XR, XROrigin, createXRStore } from '@react-three/xr'

const store = createXRStore({ offerSession: 'immersive-ar' })

The offerSession: 'immersive-ar' option tells the XR system that we want to create an AR experience.

Step 2: Wrap Your Scene with XR Components

Wrap your scene content with the XR component and add an XROrigin:

export function App() {
  return (
    <Viverse>
      <Canvas
        style={{ width: '100%', flexGrow: 1 }}
        camera={{ fov: 90, position: [0, 2, 2] }}
        shadows
        gl={{ antialias: true, localClippingEnabled: true }}
      >
        <Suspense fallback={<Text>Loading...</Text>}>
          <XR store={store}>
            <XROrigin scale={10} position-y={-8} position-z={10} />
            <Scene />
          </XR>
        </Suspense>
      </Canvas>
    </Viverse>
  )
}

Key points:

XROrigin defines the coordinate system origin for AR tracking
scale={10} makes the scene 10x larger relative to the real world
position-y={-8} position-z={10} adjusts the initial positioning

Step 3: Remove Sky Component

For AR, you don't want a sky background since the camera feed should show through. Remove any Sky components from your scene:

export function Scene() {
  return (
    <>
      {/* Remove <Sky /> for AR */}
      <directionalLight intensity={1.2} position={[5, 10, 10]} castShadow />
      <ambientLight intensity={1} />
      {/* ... rest of your scene */}
    </>
  )
}

Step 4: Use XR Controller Input

Replace keyboard/mouse input with XR controller input using the useXRControllerInput hook:

import { useXRControllerInput } from '@react-three/viverse'

export function Scene() {
  const input = useXRControllerInput()
  
  return (
    <>
      <SimpleCharacter 
        input={[input]} 
        cameraBehavior={false} 
        ref={characterRef}
      >
        <PlayerTag />
      </SimpleCharacter>
      {/* ... rest of scene */}
    </>
  )
}

Key changes:

input={[input]} - Uses XR controller input instead of keyboard/mouse
cameraBehavior={false} - Disables automatic camera control (AR handles this)
useXRControllerInput() - Hook that provides controller input for movement

The useXRControllerInput hook maps XR controller inputs to character movement:

Left thumbstick controls movement (forward/backward/left/right)
Right controller A button triggers jumping
Left trigger enables running

Virtual Reality (VR)

Now let's look at how to add VR support to a game building on the knowledge from adding AR support.

Here's the VR app we'll build now:

import { Canvas } from '@react-three/fiber'
import { Viverse } from '@react-three/viverse'
import { XR, createXRStore } from '@react-three/xr'
import { Scene } from './Scene'

const store = createXRStore({ offerSession: 'immersive-vr' })

export default function App() {
return (
  <Viverse>
    <Canvas
      style={{ position: "absolute", inset: "0", touchAction: "none" }}
      camera={{ fov: 90, position: [0, 2, 2] }}
      shadows
      gl={{ antialias: true, localClippingEnabled: true }}
    >
      <XR store={store}>
          <Scene />
      </XR>
    </Canvas>
  </Viverse>
)
}

Open Sandbox

Step 1: Change Session Type to VR

We can update the offer session to show the user a native "VR" enter button.

const store = createXRStore({ 
  offerSession: 'immersive-vr',
})

Step 2: Re-add the Sky for VR

Unlike AR, VR needs a sky background since there's no camera feed:

import { Sky } from '@react-three/drei'

export function Scene() {
  return (
    <>
      <Sky />
      <directionalLight intensity={1.2} position={[5, 10, 10]} castShadow />
      <ambientLight intensity={1} />
      {/* ... rest of scene */}
    </>
  )
}

Step 3: Hide the Character Model

In VR, you typically don't want to see your own character model:

<SimpleCharacter 
  input={[input]} 
  cameraBehavior={false} 
  model={false}
  ref={characterRef}
>

Key change:

model={false} - Hides the character model in VR

Step 4: Place the XROrigin into the Simple Character and Optionally Add Snap Rotation

As the XROrigin defines the player's position, we need to remove it from outside the Scene and add it into the SimpleCharacter.

import { useXRInputSourceState } from '@react-three/xr'

<SimpleCharacter ... >
    <XROrigin />
</SimpleCharacter>

For comfort in VR, you can add snap rotation using the right thumbstick by building a SnapRotateXROrigin which replaces the XROrigin component.

import { useXRInputSourceState } from '@react-three/xr'

<SimpleCharacter ... >
    <SnapRotateXROrigin />
</SimpleCharacter>

function SnapRotateXROrigin() {
  const ref = useRef<Group>(null)
  const rightController = useXRInputSourceState('controller', 'right')
  const prev = useRef(0)
  
  useFrame(() => {
    if (ref.current == null) return
    
    const current = Math.round(rightController?.gamepad?.['xr-standard-thumbstick']?.xAxis ?? 0)
    if (current < 0 && prev.current >= 0) {
      // Rotate left
      ref.current.rotation.y += Math.PI / 2
    }
    if (current > 0 && prev.current <= 0) {
      // Rotate right
      ref.current.rotation.y -= Math.PI / 2
    }
    prev.current = current
  })
  
  return <XROrigin ref={ref} />
}

Summary

For AR:

Use offerSession: 'immersive-ar'
Remove Sky component
Use useXRControllerInput() for input
Set cameraBehavior={false} on SimpleCharacter
Add XROrigin with appropriate scaling/positioning

Additional changes for VR:

Change to offerSession: 'immersive-vr'
Re-add Sky component
Set model={false} to hide character
Place the XROrigin into the SimpleCharacter and optionally add snap rotation for comfort

Edit this page