Multi-Modal Interactions for 3D UX Design - Swevens - Medium

brickvisualvr

This article aims at developing the thinking approach towards any immersive experience (i.e Augmented Reality, Virtual Reality or Mixed Reality) that you are planning to develop. The design principles that I will be covering here should prove useful while developing any immersive experience.

P.S - I will be using the term XR mostly in the article. XR is the umbrella category that covers all the various forms of computer-altered reality, including Augmented Reality (AR), Mixed Reality (MR), and Virtual Reality (VR).

Multi-Modal Interactions

The moment one thinks of developing any experience in XR, one out of the five senses of the developer gets activated first i.e sight. No doubt it is the most important aspect, but overloading one sensory system is a bad idea. Users tend to perform better with multi-sensory UI’s namely speech, gaze and gestures could be a good blend. These give a pretty good connection with the kind of interaction that users want while interacting with objects in the virtual world. The idea is to make an intuitive experience. It’s a good thing that XR is multi-modal by nature which makes it so compelling. Let’s discuss some common principles that are being incorporated into developing immersive experiences.

The 3 input mechanisms that this article will cover are the following —

• Gaze
• Gestures
• Voice commands
• 6-dof controller(more on this later)
• Gloves and direct interaction eg. Leap Motion(more on this later)

The various feedback mechanisms are the following —

• Visual Feedback
• Audio Feedback
• Haptic Feedback (related to 6-dof controllers and phone vibration. Not covered in this article)

Gaze

Gaze is a very powerful UI. Just start looking at any random person for long. There are various outcomes that can happen —

• The person might just ignore and walk away
• The person might get curious and come towards you
• The person might start or stop doing an action

These outcomes if incorporated into your experience might prove beneficial in terms of making a convincing interface for your user. Thus bridging the gap between the real and the virtual world. However, gaze is well known for its cursor like capabilities in the world of immersive experiences. Below is one of the sample demos from the official Google VR repository

The white dot is the cursor

Gaze on an object and it gets highlighted. Also, the cursor expands.

This particular experience gives visual feedback to the user that there is an object to interact with and the element is active. A few Gaze guidelines —

• An object of interest in the environment that you are gazing at should be kept at a minimum distance of 0.5m as recommended by Oculus. 1 meter is a comfortable distance for menus and GUIs that users may focus on for extended periods of time.
• The optimal gaze angle is 10–20 degrees below the horizon because the user generally tends to tilt the head a bit downwards during the experiences.
• Keeping the subject in focus. If the user loses track of the subject it is the job of the developer to draw back his or her gaze back to the subject by providing audio cues or visual cues. Check out the Google ARCore Elements app that gives subtle cues and this section of the official Augmented Reality Design Guidelines by Google.

Want to dive deeper into the gaze guidelines? Check out Microsoft design guidelines

Gestures

Gestures serve as a simple technique for users to interact with virtual objects in mixed reality. The idea behind this input mechanism is that users tend to move their hands a lot while they are participating in an immersive experience. It just happens intuitively and thus there are several meaningful gestures that are being incorporated like the famous “air tap” gesture for the Hololens(image below)

The Air Tap gesture for selection/deselection of objects in Hololens

One thing to keep in mind while integrating a gesture is the arm fatigue of the user. Repeated gesture input is tiring for the user. Give the user a break! No seriously, incorporating short breaks or offering a blend of speech and gesture inputs could go a long way in increasing user engagement for your app.

Stop making me air tap all the time!

A few Gestures guidelines —

• Raising the arm so high that the elbow is not in contact with the user can be tiring.
• People lose track of their real-world physical surroundings and poor hand gestures can result in them hitting a wall and eventually getting involved in a bad accident.
• Hands that visually look realistic can be creepy because it gives the user a feeling of being inside someone else’s body. Therefore breaking the immersion

Voice

Voice(speech) interaction can add efficiency to the experience. Voice UI in XR is as simple as — See it, say it. This means that if you are gazing at an object and want to interact with that object, you give the voice commands instantly. For eg) Gaze at a dog, say “jump” and it jumps. You won’t say “selected dog jump”. Keeping speech UIs simple with short phrases is one key aspect of speech UI design principles.

Voice can be imperfect in many ways. For instance, if you want the speaker’s volume to increase you say “louder” but you can’t say “a little louder” because it is hard to quantify. Sometimes a voice system incorrectly hears a command or fails to hear a command. Recovering from such errors is a challenge in any interface. Besides voice may not be socially acceptable in public places.

Source: Microsoft docs

A few Voice guidelines —

• Give users a proper logging output of their voice while they are taking part in the experience. It is called user input logging. Check out this video
• Use simple vocabulary in your voice commands.
• Make sure there is a way to undo the voice command so as to prevent any unintended action due to some other person speaking.
• Test your app with different accents.

Conclusion

The guidelines covered in this article are just one topic of the various UX design principles that can be applied to this exciting new field. Will be coming up with more articles soon!

Additional Resources

“Coming from a traditional product design background — where most interactions have many established and proven patterns — the most effective way I’ve found so far to make progress and learn in VR is through trial and error.”

• Christophe Tauziet , Facebook social VR team