Welcome back! In this three-part blog series, I walk through how to make use of the Hand Input Module in your WebXR experiences and share my development experience in going from the API’s explainer to a functioning demo in A-Frame. The series will cover the following in 3 separate posts:
In the first blog post in this series, I walked through how to access the Hand Input module and draw hand skeletons by using the explainer as a guide. Following the hand interaction using this API section of the explainer, next on our list is to add a simple interaction with an object in our A-Frame scene.
First up, we will add an object in our scene to react to hand based interactions. This will be limited to one box as seen below for the sake of simplicity. Feel free to change the size and positioning of the box, these values worked well for my space.
The box will have an id of “box” (apologies for the lack of imagination here) so we can point to it later in our interaction functions, and will be defined in the init() function in the JS script.
init: function () {
this.referenceSpace = null;
this.frame = null;
this.box = document.getElementById('box');
},
Next we’ll add the checkInteraction() function in our tick() function. At the point of exploring this API I had a simple idea for interaction— the box would change colour to red if the right hand is within a minimum distance to the box, and to blue if the left hand was used. For this we first need to:
Define a couple of Booleans, rightHandClose and leftHandClose, in the init() function:
init: function () {
this.referenceSpace = null;
this.frame = null;
this.box = document.getElementById('box');
this.rightHandClose = false;
this.leftHandClose = false;
},
Define the minimumDistance variable. Since this will not change during runtime, we can place it outside of our custom hand-tracking component at the top-level of the JS file:
const minimumDistance = 0.02;
We now have (almost) all the needed information to complete the checkInteraction() function as below:
checkInteraction() {
let rightHandClose = false;
let leftHandClose = false;for (const jointName in this.spheres) {
const jointSphere = this.spheres[jointName];
const jointPosition = jointSphere.object3D.position;
// const distance = add helper function to measures the distance between two point
if (distance < minimumDistance) {
if (jointName.startsWith('right')) {
rightHandClose = true;
}
if (jointName.startsWith('left')) {
leftHandClose = true;
}
}
}
if (rightHandClose !== this.rightHandClose || leftHandClose !== this.leftHandClose) {
// add helper function to change the colour of the box
}
this.rightHandClose = rightHandClose;
this.leftHandClose = leftHandClose;
},
The interaction function is essentially a translation of the three.js script in the explainer, where we actively loop through the joints on each detected hand and measure the distance between each joint and the box in our scene. As you can probably tell by now, we still need to add a couple of helper functions for this to run as expected: one that calculates the distance between two points to measure the distance between hand joints and the box, and another that simply changes the colour of the box based on hand interactions.
Helper functions can be placed outside of our custom hand-tracking component at the top-level of the JS file. The calculateDistance() function will measure the Euclidian distance between two points in 3D space:
function calculateDistance(point1, point2) {
return Math.sqrt((point1.x - point2.x) ** 2 + (point1.y - point2.y) ** 2 + (point1.z - point2.z) ** 2);
}
The updateBoxColour() function will change the colour of the box according to hand interactions (i.e. whether the user is using their left hand, right one or neither):
function updateBoxColour(box, rightHandClose, leftHandClose) {
if (rightHandClose) {
box.setAttribute('color', 'red');
} else if (leftHandClose) {
box.setAttribute('color', 'blue');
} else {
box.setAttribute('color', 'green');
}
}
The checkInteraction() function should now have all the necessary information and look like this:
checkInteraction() {
let rightHandClose = false;
let leftHandClose = false;for (const jointName in this.spheres) {
const jointSphere = this.spheres[jointName];
const jointPosition = jointSphere.object3D.position;
const distance = calculateDistance(jointPosition, this.box.object3D.position);
if (distance < minimumDistance) {
if (jointName.startsWith('right')) {
rightHandClose = true;
}
if (jointName.startsWith('left')) {
leftHandClose = true;
}
}
}
if (rightHandClose !== this.rightHandClose || leftHandClose !== this.leftHandClose) {
updateBoxColour(this.box, rightHandClose, leftHandClose);
}
this.rightHandClose = rightHandClose;
this.leftHandClose = leftHandClose;
},
And there it is: hand based interaction in WebXR! I am a superfan of using the hand as an input method in XR for the reasons mentioned in the previous post in this series and many more. I couldn’t stop thinking of the type of applications one can start developing using this module: virtual keyboards, hand based shooter games, rehabilitation exercises and many many more..
Now that we have added hand interaction to our scene, in the next, and final, part of this blog series we will be performing some basic gesture detection. Stay tuned!
P.S.
The full script should now look like this (note this script includes code from the previous blog post). You can also grab the full script from the hand-input-aframe GitHub repo:
Hand Input Module in A-Frame
const minimumDistance = 0.1;
function calculateDistance(point1, point2) {
return Math.sqrt((point1.x - point2.x) ** 2 + (point1.y - point2.y) ** 2 + (point1.z - point2.z) ** 2);
}
function updateBoxColour(box, rightHandClose, leftHandClose) {
if (rightHandClose) {
box.setAttribute('color', 'red');
} else if (leftHandClose) {
box.setAttribute('color', 'blue');
} else {
box.setAttribute('color', 'green');
}
}
AFRAME.registerComponent('hand-skeleton', {
init: function () {
this.referenceSpace = null;
this.frame = null;
this.spheres = {};
this.box = document.getElementById('box');
this.rightHandClose = false;
this.leftHandClose = false;
},
tick: function () {
if (!this.frame) {
this.frame = this.el.sceneEl.frame;
this.referenceSpace = this.el.sceneEl.renderer.xr.getReferenceSpace();
} else {
this.renderHandSkeleton();
this.checkInteraction();
// perform gesture detection
}
},
renderHandSkeleton: function () {
const session = this.el.sceneEl.renderer.xr.getSession();
const inputSources = session.inputSources;
if (!this.frame || !this.referenceSpace) {
return;
}
for (const inputSource of inputSources) {
if (inputSource.hand) {
const hand = inputSource.hand;
const handedness = inputSource.handedness;
for (const finger of orderedJoints) {
for (const jointName of finger) {
const joint = hand.get(jointName);
if (joint) {
const jointPose = this.frame.getJointPose(joint, this.referenceSpace);
if (jointPose != null) {
const position = jointPose.transform.position;
if (!this.spheres[handedness + '_' + jointName]) {
this.spheres[handedness + '_' + jointName] = this.drawSphere(jointPose.radius, position);
} else {
this.spheres[handedness + '_' + jointName].object3D.position.set(position.x, position.y, position.z);
}
}
}
}
}
}
}
},
checkInteraction: function () {
let rightHandClose = false;
let leftHandClose = false;
for (const jointName in this.spheres) {
const jointSphere = this.spheres[jointName];
const jointPosition = jointSphere.object3D.position;
const distance = calculateDistance(jointPosition, this.box.object3D.position);
if (distance < minimumDistance) {
if (jointName.startsWith('right')) {
rightHandClose = true;
}
if (jointName.startsWith('left')) {
leftHandClose = true;
}
}
}
if (rightHandClose !== this.rightHandClose || leftHandClose !== this.leftHandClose) {
updateBoxColour(this.box, rightHandClose, leftHandClose);
}
this.rightHandClose = rightHandClose;
this.leftHandClose = leftHandClose;
},
drawSphere: function (radius, position) {
const sphere = document.createElement('a-sphere');
sphere.setAttribute('radius', radius);
sphere.setAttribute('color', '#4d5cc1');
sphere.setAttribute('position', `${position.x} ${position.y} ${position.z}`);
this.el.appendChild(sphere);
return sphere;
},
remove: function () {
// clean up rendered spheres
for (const jointName in this.spheres) {
this.spheres[jointName].parentNode.removeChild(this.spheres[jointName]);
}
},
});
