As for many UX professionals, my career so far has centered largely around performing UX research and design for Web and mobile applications. However, for the past year or two, I’ve been increasingly excited by virtual reality (VR) and augmented reality (AR) applications and their potential to positively impact our lives. My excitement stems from reimagining existing use cases in spaces such as education, workplace productivity, and entertainment, as well as from recognizing the potential for VR and AR to introduce entirely new digital experiences that go beyond what we’ve so far envisioned. The capabilities of the technology are quickly getting to where they need to be. The primary question people are asking now is: will the content be there?
Experience designers must rise to the challenge. Of course, transitioning from traditional digital platforms to the wild west of extended reality (XR)—a blanket term that encompasses VR, AR, and mixed reality (MR)—requires some prep work. While I’m by no means claiming to be an expert experience designer for XR quite yet, I want to share my journey as an XR fan-boy. I’ve been absorbing the relatively small amount of information that is currently available on designing VR and AR experiences—reading every article and watching every video—and tinkering first hand with my beloved head-mounted display (HMD).
In my next few columns, I’ll discuss some of my key takeaways from these explorations of a traditional Web and mobile UX designer who is preparing for a transition to designing for VR and AR. These columns will span everything from foundational concepts that lay the groundwork for XR design through some practical techniques, tips, and pitfalls to avoid. To begin, I want to consider imagination, immersion, and presence.
The Evolution of Imaginative Crutches
Technology seems to evolve in ways that perpetuate the need to accommodate our imaginations. For the most part, this means enabling users to perform tasks and engage in experiences in ways that are not possible without technology. Whether that’s flying from Chicago to Seattle in under four hours instead of traveling for weeks on horseback, calling and immediately speaking to your business partner on the other side of the globe instead of sending snail mail back and forth to finish a full conversation over the course of months, or searching the Internet day and night for any information your heart desires rather than sifting through stacks of thick books or asking people in your community whether they have the answers you need. As history demonstrates, reality eventually gives way to imagination through the means of technology.
The extent to which technology accommodates the imagination has continually increased over the years—and that’s not going to stop. VR and AR experience designers are reducing the gap between our imaginations and our mundane information and task pursuits by creating compelling, immersive, first-person visual experiences that edge closer and closer to a state in which possibility consistently keeps up with imagination.
Virtual reality and augmented reality are exciting because, in theory, the only limitation to our experience is our imagination—or, perhaps, the imagination of XR experience designers. In the physical world, reality has often fallen short of the demands of imagination. We can create and engage only in reality-based experiences that abide by the laws of physics and practical physical, financial, social, and geographical constraints. As we mature and endure regimented schooling, a 9-to-5 job, and constraining social norms, imagination eventually takes the backseat to more realistic endeavors. The practicalities of life make us forget how to dream. We lose that magic spark we once had as kids. But deep down, we really want to get it back.
“Reality has always been too small for the human imagination. We’re always trying to transcend.”—Brenda Laurel
Similarly, although to a lesser extent, imagination around traditional digital devices such as phones, computers, and tablets lacks a truly magical quality in comparison to the immersive essence of XR. While such digital experiences can certainly be more imaginative than physical reality, a dynamic still exists between the person and the user interface that implies separation. This sense of separation has caused mobile and desktop experience design to rely less and less on imagination because traditional digital mediums are nondiegetic in so many ways, and this has limited our ability to immerse ourselves in them.
In contrast, XR opens digital design up to a whole new realm of imagination. While its imaginative possibilities are endless, the value of an XR experience depends on how well it is designed. There are many components that separate good XR experiences from bad ones. In this multipart series, I’ll discuss quite a few of them. Let’s start with immersion and presence.
“There are no rules of architecture for a castle in the clouds.”—Gilbert K. Chesterton
Levels of Immersion
Typically, the realms of AR, MR, and VR are progressively more immersive. In AR, we are first and foremost experiencing physical reality, but with the added twist of imaginative reality being overlaid on top of actual reality. AR is the least immersive XR experience. In MR, we have some ability to augment our reality by adding imaginative functionality that lets us visually manipulate objects—for example, creating a focus effect on gaze and a blur effect for our peripheral vision. MR is more immersive than AR, but less immersive than VR. Finally, VR lets us enter whole new worlds. In the realm of VR, we do much more than simply add imagination to our base reality; we can create an entirely new base reality in any way we could possibly desire. The only limit is our imagination.
Presence and Immersion
Every XR experience should start by defining presence, which is achieved when the user truly feels that she is somewhere in particular. This is a desirable effect within VR. To create a proper VR experience, you must first establish the setting the user is inhabiting. If you do not firmly establish where the user is, she might not feel that she is anywhere in particular, causing an eerie sense of disorientation and presenting a huge barrier to the rest of the planned experience.
Presence demonstrates where the user is at any given moment. The more clearly you can establish presence, the more immersive an experience will be for the user. When the user confidently knows where she is, you allow her to feel that she is actually there. Self-doubt in the context of physical orientation breaks our sense of immersion. Immersion comprises many factors, but is highly dependent on presence.
Designing for Presence in Virtual Realities
Achieving presence within VR is one of the most fundamental challenges that VR designers encounter—albeit a worthy challenge. With presence, the user no longer has the perspective of the third eye, but that of the first-person actor, who is fully immersed in the virtual environment.
So how do we achieve presence? There are two primary factors to consider when designing for presence in VR: place illusion and plausibility illusion.
Place illusion convinces the user that she is actually in a particular environment, while plausibility illusion persuades the user that the experience she is having in that place is believable without a doubt. There are quite a few factors to keep in mind when designing for place and plausibility. Let’s go over a few.
The first factor that I’d like to talk about is sensory modality. Within the interactions you design, how many of the user’s sensory capabilities are you engaging? A good mix of visual, auditory, and haptic stimulation can make the user feel less removed from the real world, but not if the use of multiple senses feels forced. Designers must creatively plan multimodal interactions so they feel necessary and enjoyable.
A common mistake of early VR experiences has been to be overly concerned with 360-degree space and rely too heavily on perfecting just the visuals. Be sure to consider how auditory and haptic modalities can enhance the overall experience by complementing and expanding upon its visual component. The goal should be to design a fluent, cohesive, multimodal experience that appropriately engages several senses in giving and receiving information within an XR experience. This is particularly important in achieving place illusion. When users engage their senses in your designed experience just as they would in real life, they begin to reach out and touch things as if they were really there, without hesitation and, most importantly, without questioning the mechanics necessary to pursue their desired actions.
In addition to creating multimodal XR interactions, it is important to ensure that responses to the user’s actions are as expected. For example, if the user throws a rock at the bad guy in a VR action game, his reaction should be anger and perhaps to threaten the user. This would be the expected reaction. However, if the bad guy were to instead pick up the rock, as well as a few other rocks, and start juggling, the user would begin to doubt the veracity of the experience, breaking plausibility illusion. Carefully considering each and every possible user action and designing a believable system reaction is key in achieving plausibility illusion and strengthening presence within an immersive experience.
Designing for Presence in Augmented Realities
Designing for presence in AR is a bit different from designing for presence in VR. In AR, the real world is the canvas. So, rather than convincing the user that she is actually in a world that is entirely separate from the one she knows to be real, AR relies on maintaining a connection to the real world. Instead of escaping the real world, AR uses it as a canvas. So we must define presence in a way that maintains some connection to the real world by demonstrating an awareness of certain attributes that we know to be true.
This doesn’t mean that, as an AR designer, you should decrease the amount of magic you apply when augmenting, enhancing, or somehow modifying the user’s perception of the real world. However, it does mean accommodating the real-world objects, physical laws, and other constraints that surround the augmented experience you’re designing. The two should feel meshed. At every opportunity, AR designers should look for ways in which users can interact with augmented elements in ways that acknowledge the surrounding, real-world elements.
Acknowledgment of the real world requires thinking about the real-world space in which the user resides, the context of use it implies, the emotions it may evoke, and the actions or behavioral patterns that may be common to this space.
For example, if you are augmenting an office space, you might consider some of the behaviors and emotions that are typical within this context. In the office, users might commonly check their email and look at their calendar. Providing augmented calendar and email-notification user interfaces that the user could swipe into focus to quickly glance at updates might be perfect for this context of use. However, it might not be as useful or desirable within the context of a tennis match on the weekend—a very different context in which users are deliberately trying to escape the emotions that are characteristic of their behaviors within an office context.
Things That Break Presence
Whenever a user must stop to think about the mechanics of how to carry out a desired action, presence begins to deteriorate. These mechanics include sifting through a menu to find a particular item or action and wondering which gesture or control to use to initiate the desired action.
In addition to creating less than transparent interaction mechanics, XR designers also run the risk of reducing presence when the following discontinuities occur:
Virtual spaces fail to suggest where the user can physically go.
Nonplayable characters (NPCs) lack a conversational tone or cannot provide the user with adequate responses to inquiries that are likely predictable based on location and story context—assuming a story-type experience.
Real-life interactions—such as opening and closing doors or picking something up and throwing it—demand unusual user inputs or behave in a way that is vastly different from the real world.
There are quite a few more presence-breaking pitfalls that I could discuss, but the ones I’ve noted here are some key things to watch out for.
XR is an incredibly powerful new tool for bridging the gap between imagination and reality. It achieves this most effectively when full immersion occurs, a solid sense of presence exists, and a multimodal experience looks, feels, and sounds believable. These are foundational concepts to keep top of mind when you’re moving from designing experiences for traditional mediums to designing the magical world of XR.
In the next part of this series, I’ll cover some additional concepts to keep in mind, as well as some of the more practical considerations regarding the early stages of XR design.
Dash designs usable, enjoyable digital experiences that are driven by research and guided by the needs and desires of internal and external stakeholders. In his work, he draws upon his past experience in startups, UX consulting, internships, and freelancing, as well as the wealth of UX knowledge he gained through his journey to earn an MS in HCI. From concept to launch, Dash incorporates Lean and full-cycle UX tools and methods. He is always excited by future opportunities to play his part in delivering innovative digital solutions. Read More