Grab, hold and throw in VR

VR grab interaction — pick up, hold, throw

Grabbing is the core verb of most VR games, and getting it to feel right comes down to three things: select the nearest object within hand reach, reparent it to the controller so it tracks the hand without jitter, and on release hand it back to the scene with the velocity the hand was moving — so throwing works. A short haptic pulse on grab confirms the action in the player's hand.

Core principle: reparent to the controller while held (controller.attach), restore to the scene on release with measured velocity.

When to use

• Picking up, holding, and throwing objects with motion controllers.
• Symptoms: grabbed object jumps to the controller origin, jitters while held, or drops straight down with no throw momentum.

Prerequisites: session/renderer from xr-session-setup; this builds on controller access from xr-controller-input. Hand-tracking pinch-grab is a variant — same attach/release logic, different trigger event.

Get the controllers and their events

import * as THREE from 'three';

// Two controllers; each is an Object3D that tracks a hand. selectstart/end map to trigger/grip.
const controllers = [0, 1].map((i) => {
  const c = renderer.xr.getController(i);
  scene.add(c);
  c.addEventListener('selectstart', () => onGrabStart(c));
  c.addEventListener('selectend', () => onGrabEnd(c));
  return c;
});

Mark which objects can be picked up, and give each a velocity tracker:

interface Grabbable extends THREE.Object3D { userData: { grabbable: true } }
const grabbables: THREE.Object3D[] = []; // push your pickup meshes here

// Per-controller hold state
const held = new Map<THREE.Object3D, THREE.Object3D>(); // controller -> object

Grab: select nearest in reach, then attach

Use a small hand-sphere proximity test (see collision) to pick the closest grabbable within reach (~0.1 m). Object3D.attach reparents while preserving world transform, so the object doesn't snap.

const GRAB_RADIUS = 0.1;
const _handPos = new THREE.Vector3();
const _objPos = new THREE.Vector3();

function nearestGrabbable(controller: THREE.Object3D): THREE.Object3D | null {
  _handPos.setFromMatrixPosition(controller.matrixWorld);
  let best: THREE.Object3D | null = null;
  let bestDist = GRAB_RADIUS * GRAB_RADIUS;
  for (const obj of grabbables) {
    if (held.has(controller) || isHeld(obj)) continue;
    _objPos.setFromMatrixPosition(obj.matrixWorld);
    const d = _handPos.distanceToSquared(_objPos);
    if (d < bestDist) { bestDist = d; best = obj; }
  }
  return best;
}

function onGrabStart(controller: THREE.Object3D) {
  const obj = nearestGrabbable(controller);
  if (!obj) return;
  controller.attach(obj);        // reparent, keep world position/rotation — no snap
  held.set(controller, obj);
  pulse(controller, 0.6, 40);    // haptic confirm
}

const isHeld = (obj: THREE.Object3D) => [...held.values()].includes(obj);

Track hand velocity while held

To throw, you need the hand's velocity at the moment of release. Sample the controller's world position each frame and keep the last delta.

const prevPos = new Map<THREE.Object3D, THREE.Vector3>();
const velocity = new Map<THREE.Object3D, THREE.Vector3>();

// Call from your renderUpdate(dt) — see game-loop.
function trackControllerVelocity(controller: THREE.Object3D, dt: number) {
  const now = new THREE.Vector3().setFromMatrixPosition(controller.matrixWorld);
  const prev = prevPos.get(controller);
  if (prev && dt > 0) {
    velocity.set(controller, now.clone().sub(prev).divideScalar(dt)); // m/s
  }
  prevPos.set(controller, now);
}

Release: restore to scene + throw

const _m = new THREE.Matrix4();

function onGrabEnd(controller: THREE.Object3D) {
  const obj = held.get(controller);
  if (!obj) return;
  scene.attach(obj);             // reparent back to scene, keep world transform
  held.delete(controller);

  // Hand off the hand's velocity so the object flies (your physics/integrator uses this).
  const v = velocity.get(controller);
  if (v) (obj.userData as any).vel = v.clone(); // e.g. consumed by collision/physics integrator
  pulse(controller, 0.3, 25);
}

The thrown object's motion is then integrated by your loop / collision / physics — this skill just supplies the release velocity.

Haptics — confirm the grab in the hand

Pulse the controller's haptic actuator via its WebXR input source gamepad.

function pulse(controller: THREE.Object3D, intensity = 0.5, ms = 40) {
  const src = controller.userData.inputSource as XRInputSource | undefined;
  const actuator = (src?.gamepad as any)?.hapticActuators?.[0];
  actuator?.pulse?.(intensity, ms);
}

// three.js stores the input source on the controller when it connects:
controllers.forEach((c) =>
  c.addEventListener('connected', (e: any) => { c.userData.inputSource = e.data; }),
);

Domain principles honored here

• Input abstraction — uses the standard controller select events and the input-source gamepad, not a hardcoded device.
• 1:1 scale — grab radius and velocity are in meters / m·s⁻¹, matching real reach and throw.
• Comfort first — held objects track the hand exactly (no lag/jitter that fights the player); the world/head are never moved by a grab.

Common mistakes

Verify

• On-device: an object grabbed off-center stays where the hand holds it (no snap to origin).
• Held object tracks the hand 1:1 with no jitter.
• Releasing while swinging throws the object in the swing direction with believable speed.
• A haptic pulse fires on grab.
• Two-handed: each controller can hold a different object independently.

References

• xr-controller-input — controller access and button/trigger state.
• collision — hand-sphere proximity test and integrating the thrown object.
• vr-polish-checklist — haptics + spatial sound on grab/impact for game feel.