Welcome back! In this three-part blog series, I walk through how to make use of the Hand Input Module in your WebXR experiences and share my development experience in going from the API’s explainer to a functioning demo in A-Frame. The series will cover the following in 3 separate posts:
In the previous two blog post in this series, I walked through how to access the Hand Input module, draw hand skeletons and add a simple interaction by using the explainer as a guide. Following the gesture detection using this API section of the explainer, next on our list to wrap up this series is to add some simple gesture detection in our A-Frame scene.
For detecting gestures we are essentially describing a gesture by accessing the position and orientation of different joints, if certain joints on the hand match the gesture parameters we provide during the experience then something happens (e.g. move the object, change its shape, change its colour etc.). The example provided in the explainer is for a fist gesture which would require using the position and orientation values of four or five fingers. I thought I would simplify this further for this post by detecting a pinch gesture instead. A pinch gesture is a pretty common hand based interaction technique and we will minimise any room for error by dealing with fewer joints during development.
The simple idea I have is as follows: the box will change its shape to a torus knot shape if a pinch gesture is detected using the right hand, and it will change to a dodecahedron shape if a pinch is detected using the left hand.
Let’s start with adding the detectGesture() function in our tick() function:
detectGesture: function () {
const session = this.el.sceneEl.renderer.xr.getSession();
const inputSources = session.inputSources;let isRightPinching = false;
let isLeftPinching = false;
for (const inputSource of inputSources) {
if (inputSource.hand) {
const thumbTip = this.frame.getJointPose(inputSource.hand.get("thumb-tip"), this.referenceSpace);
const indexTip = this.frame.getJointPose(inputSource.hand.get("index-finger-tip"), this.referenceSpace);
if (thumbTip && indexTip) {
const distance = calculateDistance(thumbTip.transform.position, indexTip.transform.position);
if (distance < pinchDistance) {
if (inputSource.handedness === 'right') {
isRightPinching = true;
} else if (inputSource.handedness === 'left') {
isLeftPinching = true;
}
}
}
}
}
this.rightPinchDetected = isRightPinching;
this.leftPinchDetected = isLeftPinching;
// add helper function to change the shape of the box
},
In our function we are specifically getting the joint pose information for the thumb tip and index tip that are relevant for our pinching gesture. Then we simply use the calculateDistance() function from the previous post in this series to check if the distance between the two joints is within a specific threshold (we will specify this threshold shortly). I also check for the handedness to distinguish between a right and left pinches. Note that you can also check for the difference in joint orientation as done in the explainer, but I only worked on distance for pinching.
Helper functions can be placed outside of our custom hand-tracking component at the top-level of the JS file. The updateBoxAppearance() function will change the shape/geometry of the box based on the handedness of the pinch detected (i.e. right or left):
function updateBoxAppearance(box, isRightPinching, isLeftPinching) {
if (isRightPinching) {
box.setAttribute('geometry', {primitive: 'torusKnot', radius: 0.1, radiusTubular: 0.02});
} else if (isLeftPinching) {
box.setAttribute('geometry', {primitive: 'dodecahedron', radius: 0.15});
} else {
box.setAttribute('geometry', {primitive: 'box', width: 0.2, height: 0.2, depth: 0.2});
}
}
We also need to define the pinchDistance variable. I initially used the minimumDistance variable that was used for interaction in the previous post, but it turned out to be too big for detecting pinches, the detection was happening, way, too easily. Since this will not change during runtime, we can place it outside of our custom hand-tracking component at the top-level of the JS file:
const pinchDistance = 0.01;
The detectGesture() function should now have all the necessary information and look like this:
detectGesture: function () {
const session = this.el.sceneEl.renderer.xr.getSession();
const inputSources = session.inputSources;let isRightPinching = false;
let isLeftPinching = false;
for (const inputSource of inputSources) {
if (inputSource.hand) {
const thumbTip = this.frame.getJointPose(inputSource.hand.get("thumb-tip"), this.referenceSpace);
const indexTip = this.frame.getJointPose(inputSource.hand.get("index-finger-tip"), this.referenceSpace);
if (thumbTip && indexTip) {
const distance = calculateDistance(thumbTip.transform.position, indexTip.transform.position);
if (distance < pinchDistance) {
if (inputSource.handedness === 'right') {
isRightPinching = true;
} else if (inputSource.handedness === 'left') {
isLeftPinching = true;
}
}
}
}
}
this.rightPinchDetected = isRightPinching;
this.leftPinchDetected = isLeftPinching;
updateBoxAppearance(this.box, isRightPinching, isLeftPinching);
},
As mentioned in the first post in this series, I tried to follow the instructions of the explainer to a great degree in these posts, and upon reflection now that we have concluded the series, I had the following highlights from a self retrospective:
- Take advantage of the engine you’re using: tapping into the Hand Input Module like we did in this series provides a great level of control over what we can do with it (i.e. full control over joint information, how you draw skeletons, designing gestures etc.), however it may be worth checking what the engine you are using offers in terms of support for the Hand Input Module. That can save you much time if you are not interested in dealing with the intricacies of the spec. In A-Frame’s case, you can check out hand-tracking-controls and hand-tracking-grab-controls components that provide a quick start in using both hands as inputs for basic gesture and collision detection.
- Scalability: when thinking of scaling up this demo in terms of different objects, unique interactions and gestures etc., I found detecting interaction by calculating proximity may not be the best method, performance and development time wise, if you’re using A-Frame. In the spirit of taking advantage of the engine we are using, Colliders may instead be a more sensible choice.
- Gesture complications: gesture detection can get complicated to perform and test, this is mentioned in the explainer. Keep that in mind when developing custom gestures.
- Community components for the win: there are A-Frame components out there already that have made use of this Hand Input Module and can provide a great head start, namely: handy-work, webxr-handtracking and aframe-hand-tracking-controls-extras.
And that’s that. We are actually done! On the bright side, I am sure the Hand Input Module will pop up every now and then in future posts.
Muadh out — Until next time! 🤜🤛
PS.
The full script should now look like this (note this script includes code from all previous blog posts). You should be able to get it up and running in a platform like Glitch (or similar). You can also grab the full script from the hand-input-aframe GitHub repo:
Hand Input Module in A-Frame
const minimumDistance = 0.1;
const pinchDistance = 0.01;
function calculateDistance(point1, point2) {
return Math.sqrt((point1.x - point2.x) ** 2 + (point1.y - point2.y) ** 2 + (point1.z - point2.z) ** 2);
}
function updateBoxColour(box, rightHandClose, leftHandClose) {
if (rightHandClose) {
box.setAttribute('color', 'red');
} else if (leftHandClose) {
box.setAttribute('color', 'blue');
} else {
box.setAttribute('color', 'green');
}
}
function updateBoxAppearance(box, isRightPinching, isLeftPinching) {
if (isRightPinching) {
box.setAttribute('geometry', {primitive: 'torusKnot', radius: 0.1, radiusTubular: 0.02});
} else if (isLeftPinching) {
box.setAttribute('geometry', {primitive: 'dodecahedron', radius: 0.15}); // Changed from octahedron to tetrahedron
} else {
box.setAttribute('geometry', {primitive: 'box', width: 0.2, height: 0.2, depth: 0.2});
}
}
AFRAME.registerComponent('hand-skeleton', {
init: function () {
this.referenceSpace = null;
this.frame = null;
this.spheres = {};
this.box = document.getElementById('box');
this.rightHandClose = false;
this.leftHandClose = false;
},
tick: function () {
if (!this.frame) {
this.frame = this.el.sceneEl.frame;
this.referenceSpace = this.el.sceneEl.renderer.xr.getReferenceSpace();
} else {
this.renderHandSkeleton();
this.checkInteraction();
this.detectGesture();
}
},
renderHandSkeleton: function () {
const session = this.el.sceneEl.renderer.xr.getSession();
const inputSources = session.inputSources;
if (!this.frame || !this.referenceSpace) {
return;
}
for (const inputSource of inputSources) {
if (inputSource.hand) {
const hand = inputSource.hand;
const handedness = inputSource.handedness;
for (const finger of orderedJoints) {
for (const jointName of finger) {
const joint = hand.get(jointName);
if (joint) {
const jointPose = this.frame.getJointPose(joint, this.referenceSpace);
if (jointPose != null) {
const position = jointPose.transform.position;
if (!this.spheres[handedness + '_' + jointName]) {
this.spheres[handedness + '_' + jointName] = this.drawSphere(jointPose.radius, position);
} else {
this.spheres[handedness + '_' + jointName].object3D.position.set(position.x, position.y, position.z);
}
}
}
}
}
}
}
},
checkInteraction: function () {
let rightHandClose = false;
let leftHandClose = false;
for (const jointName in this.spheres) {
const jointSphere = this.spheres[jointName];
const jointPosition = jointSphere.object3D.position;
const distance = calculateDistance(jointPosition, this.box.object3D.position);
if (distance < minimumDistance) {
if (jointName.startsWith('right')) {
rightHandClose = true;
}
if (jointName.startsWith('left')) {
leftHandClose = true;
}
}
}
if (rightHandClose !== this.rightHandClose || leftHandClose !== this.leftHandClose) {
updateBoxColour(this.box, rightHandClose, leftHandClose);
}
this.rightHandClose = rightHandClose;
this.leftHandClose = leftHandClose;
},
detectGesture: function () {
const session = this.el.sceneEl.renderer.xr.getSession();
const inputSources = session.inputSources;
let isRightPinching = false;
let isLeftPinching = false;
for (const inputSource of inputSources) {
if (inputSource.hand) {
const thumbTip = this.frame.getJointPose(inputSource.hand.get("thumb-tip"), this.referenceSpace);
const indexTip = this.frame.getJointPose(inputSource.hand.get("index-finger-tip"), this.referenceSpace);
if (thumbTip && indexTip) {
const distance = calculateDistance(thumbTip.transform.position, indexTip.transform.position);
if (distance < pinchDistance) {
if (inputSource.handedness === 'right') {
isRightPinching = true;
} else if (inputSource.handedness === 'left') {
isLeftPinching = true;
}
}
}
}
}
this.rightPinchDetected = isRightPinching;
this.leftPinchDetected = isLeftPinching;
updateBoxAppearance(this.box, isRightPinching, isLeftPinching);
},
drawSphere: function (radius, position) {
const sphere = document.createElement('a-sphere');
sphere.setAttribute('radius', radius);
sphere.setAttribute('color', '#4d5cc1');
sphere.setAttribute('position', `${position.x} ${position.y} ${position.z}`);
this.el.appendChild(sphere);
return sphere;
},
remove: function () {
for (const jointName in this.spheres) {
this.spheres[jointName].parentNode.removeChild(this.spheres[jointName]);
}
},
});
