Human Computer Interaction (HCI) research has long been focused on developing interaction paradigms that describe how humans interact with technology — taking into account physical, virtual, perceptual and cognitive aspects of interaction. Over time, these paradigms have evolved in response to significant technological advances in computing, refining how users engage with digital systems.
In the realm of XR where user interaction is a core part of immersive experiences, these advances in interaction paradigms are particularly impactful. As XR technologies continue to push the boundaries of what’s possible, developers are increasingly focused on creating experiences that are not only immersive, but also feel natural and intuitive to the user. This is where Natural User Interfaces (NUIs) come into play. By enabling users to interact with virtual content in ways that mimic real-world actions, NUIs can enhance the user experience in XR by offering more seamless and instinctual ways to engage with immersive experiences.
Here is my inspiration for this post — I recently got a chance to observe how different users complete the simple task of picking up virtual cubes in a WebXR experience using their hand. The demo I developed only allowed users to perform a pinch gesture to pick and release virtual cubes. This observation lead me to the following question for this post: should we use predefined gestures for hand interactions in XR and WebXR experiences, or should we provide more natural methods of interaction for users to interact in a way that feels intuitive to them?
NUIs allow users to interact with virtual content in ways that mimic real world actions and behaviours. NUIs provide valuable alternative input methods in XR besides classic controllers by taking advantage of traditional human abilities and can leverage a variety, or a combination, of modalities to interact with virtual content such as touch, voice, gaze, body movements and hand gestures.
Here is a non-exhaustive list of reasons of why NUIs can be great in XR experiences:
- Higher sense of immersion: NUIs can deepen the sense of presence in immersive experiences by mirroring real life actions. This replication of reality when it comes to interaction with virtual content can be intriguing for users and provides some “wow” factor.
- Shallower learning curve: by relying on familiar, real world interactions, NUIs can reduce the time and training needed for users to get used to the XR experience. This allows users of your experience, especially first time users, to transition quickly from novice to expert.
- Improved information retention: when XR is used for training and skill development applications, replicating real-life movements and actions allows users to practice tasks they may encounter in the real world in a safe virtual setting. This “learning by doing” approach improves retention of skills and information and accordingly users are more likely to transfer those skills to their real everyday environment.
This is all great and cool, but here is another non-exhaustive list of reasons of why NUIs can be not so great in XR experiences:
- Escapism (escapes) reality: one of XR’s greatest strengths is escapism — allowing users to experience out of body experiences and do what they can’t in the real world. Flying, teleporting, or becoming someone else entirely isn’t just fun; it’s often essential for users who need an escape from their physical limitations. In these cases, NUIs and the general feeling of “realism” they can simulate becomes irrelevant — or even detrimental.
- Immersion is inherently subjective: what feels natural and realistic to one user in an XR environment may seem cumbersome or restrictive to another. For example — a user who is used to certain hand gestures in real life might find a predefined gesture (like a pinch) uncomfortable or unnatural, whereas a more seasoned XR user would prefer a pinch gesture as that feels like an “immersive and XR-y” technique for interaction in immersive environments. This variability in user expectations when it comes to interaction with virtual content highlights the need to prioritise flexible, intuitive interactions, rather than rigidly adhering to realistic or prescribed interactions.
- Physical limitations of users: NUIs in XR often assume a certain level of physical ability and while they aim to mimic real world actions, they can inadvertently exclude users with disabilities or those with limited physical abilities. This highlights the importance of designing for accessibility when adopting NUIs for your XR experience, as some users may find natural ways of interaction challenging or worse — impossible.
Interaction paradigms and NUIs are very big topics, and we can spend ages going through how best to design interactions using every modality that can be leveraged by a NUI. To simplify this, I will zoom into hand based interactions specifically. My main goal here is to illustrate how overthinking NUIs, or totally neglecting them, can be problematic.
I recently got a chance to observe how different users interact with the simple task of picking up virtual cubes (a.k.a some friendly testing for a demo I was working on). If you’ve used and developed XR experiences before, you’d be right think this is a straightforward task — pinch the cube, move it as you move your hand/arm and release the pinch to let go of the cube. But here is the thing — not all users would do this, here is how different users undertook the same task:
- User 1: grasp the cube realistically as they would for a real one (which wouldn’t work of course because only a pinch gesture is used).
- User 2: perform a closed fist action because they can’t really “feel” a virtual cube and expect a closed fist to capture the cube (might work at times but would generally be glitchy).
- User 3: pinch because they are aware that pinching is one of the main interaction techniques in XR (perfect — demo works!).
- User 4: perform a closed fist gesture and attempt throwing cubes away (because why wouldn’t an AR environment have physics? — again closed first may be glitchy and throwing wouldn’t work of course because physics were deemed by me as unimportant for this demo).
Because I only used a predefined pinch gesture as an interaction technique for my demo, three out of my four users couldn’t use the demo properly (or took longer to do so). This lack of responsiveness, due to my design choices, hindered their overall XR experience and led some testers to question whether they were doing something wrong or if the experience was functional at all. This scenario can even more damaging to the user experience if the context in which the user is in while trying out the XR experience does not allow time for learning or training on certain interaction techniques or gestures (e.g limited time, can only use one hand, unable to pinch).
Here is the interesting part in all of this —I actually played a video of the demo that showed my “correct” hand interaction using the pinch gesture with the cubes on a big screen before giving users the opportunity to try it hands on. Additionally all the users in my demo were aware of XR, even used it a few times before but they still behaved differently based on what they thought would be the right way to interact with the virtual content around them. The reason for this is pretty simple from a psychology point of view— human beings have different life experiences that yield different ways of interaction with objects around them. More importantly in XR, users have different ways of perceiving virtual environments and different ways of interpreting how they can interact with virtual content. This is key here, because we can’t really re-wire users with lifetimes’ worth of habits and experiences without some kind of training, especially those that haven’t used your XR experience before. They would simply expect the XR experience to interpret whatever they choose to do.
Rather than enforcing specific hand gestures, XR experiences should ideally strike a balance between learned predefined gestures and natural ones to respond to user intent and accommodate for more user choices when it comes to hand based interactions. Users shouldn’t need to “follow the rules” of predefined gestures (unless that’s super required for certain training or rehabilitation applications of course), and they also shouldn’t be in an immersive experience that stutters or becomes unresponsive if they choose a more natural gesture — they need the freedom to interact in ways that feel intuitive to them. This inclusive approach when designing interactions in XR provides the user with much more choice, and results in a better overall user experience by ensuring the XR experience is flexible enough to work for a wider user base in different contexts.
A great user experience in XR entails giving the user a feeling of control over their interactions in whatever way that makes sense to them, be that using natural or predefined gestures. Whether that’s pinching, grasping, or any other other gesture, the technology should adapt to the user, not the other way around.
Muadh out — Until next time! 🤜🤛
