If you have been to your local mall recently, you have probably noticed that virtual reality (VR) and augmented reality (AR) products and services are now hitting the market in much greater numbers than last year. These digital experiences mix with or even completely replace physical reality, letting users get out from behind their devices’ screens.
From sports to retail, entertainment, and medicine, there are clear signs that we are approaching a tipping point with immersive technology. These signs are similar to those we experienced before other major platforms—such as the Web and smartphones—exploded on the scene. Businesses are investing strategically in what will be the biggest platform introduction since mobile. For example, Mark Zuckerberg offered a strong business rationale for Facebook’s decision to pay $2 billion for Oculus Rift: “Strategically, we want to start building the next major computing platform that will come after mobile. … Immersive virtual and augmented reality will become a part of people’s everyday [lives].”
Such technological inflection points require UX professionals to make major career shifts to stay relevant in the marketplace. Designers will need to ramp up on new design skills to make a smooth career transition to the design of immersive experiences when the inevitable wave of new VR and AR design projects hits the pipeline. Although developers have taken an early lead in the design of immersive experiences, involving UX professionals on these projects is critical to making the new technology relevant, useful, and engaging.
Designing a seamless, believable VR or AR experience requires a deep understanding of how the technology works, its range of capabilities, and how humans perceive different aspects of immersive experiences. The design of immersive experiences is much more complicated than Web or mobile design, in ways I’ll discuss later in this article. The learning curve for UX professionals will be very steep. So, while you could wait until a VR or AR design assignment knocks at your door, I suggest you start ramping up on the design of immersive experiences now.
Why Dive in Now?
To explain why I’m choosing to dive into the design of immersive experiences now, I need to tell you a little about my personal UX-career journey.
In 1994, I was living in Amsterdam. A friend showed me something exciting that he had come across during a science meeting in Switzerland. On his computer screen was a photo of a researcher and some information about his work. It looked like a simple page layout for a magazine article, but viewable on a computer. I countered that I didn’t get what the big deal was. I was involved in multimedia design, creating CD-ROMs with 3D animations, video, and Dolby surround sound. My friend said something that stuck with me: “Yes, but everyone in the world can see this page with a simple Internet connection and a computer, whether they are in Tokyo, New Orleans, or Amsterdam.” While the concept was striking, it took a while to sink in just how this difference in distribution and access would change the world.
In 1995, I moved to Barcelona and started doing some self-study about the World Wide Web at the Pompeu Fabra University library. Based on what I read, I decided to shift my career away from information science and patent databases to the new Web. That year, Amazon appeared online. In 1996, Google launched—and so did one of the largest career Web sites, which a friend of mine developed in his basement.
We had reached the tipping point for the Web, but it took quite a few years before many designers, marketers, or business strategists understood that they needed to shift their careers to remain relevant after the massive transformation that Web products, services, and experiences would soon bring about.
Like Mark Zuckerberg, I think we are at the beginning of another major platform revolution. This time it’s not from paper to the Web, but from flat screens portraying pages and videos to either partially or fully immersive digital experiences. All of these immersive experiences will need to be designed. But there are very few UX designers who have the necessary skills to do this work today. This is a problem.
Having lived through a few of these technological Wild West scenarios in the past, I decided to do my best to help UX designers get quickly up to speed on what it will take for them to retool their careers for virtual and augmented reality. To this end, I’m organizing the XD Immersive conference, which will take place in San Francisco—the epicenter of the VR/AR revolution—on May 3-4, 2018, at the ultramodern Bespoke venue on Market Street. The first edition of this conference will focus on VR, AR, and voice. There is currently an open call for speakers and sponsors.
A Sliding Scale of Reality
The nomenclature of immersive technology is pretty straightforward—although there is some competition among the giants in this industry to have their nomenclature accepted as the standard. Let’s look at some common terms.
Physical reality is the world of mass and energy that Einstein described at the beginning of the 20th century. We can perceive physical reality with the five senses, without any digital enhancements.
Virtual reality (VR) refers to a completely immersive simulated digital reality. VR completely replaces the visual and auditory stimuli of physical reality with a designed environment. Smell, taste, and touch are not far behind. Some potential virtual-reality applications include the following:
viewing the Super Bowl in real time, from the sideline at the line of scrimmage or from the perspective of any camera on the field, with the ability to choose your view
taking a self-directed virtual tour of the Fiji Islands, exploring a wide range of hospitality and entertainment options first hand
learning a range of new interpersonal skills, interacting with virtual actors in realistic situations that adapt to your responses and actions
The term augmented reality (AR) typically describes digital data and imagery that are superimposed on physical reality. It’s like having a computer screen overlay what you normally perceive as your world. Some potential augmented-reality applications include the following:
touring Paris while seeing an overlay that presents interactive data and descriptions for any building, monument, or art work you select
approaching a highway exit and seeing immediately recognizable food, fuel, and hospitality options
Mixed reality (MR)—which Microsoft is pushing hard as a differentiated immersive experience—incorporates virtual or holographic objects that are anchored in place within physical reality. While AR data and imagery move with you when you walk around, MR virtual objects stay in the same place, so you can view them from different angles. They get larger as you move closer to them. Some potential mixed-reality applications include the following:
a group of engineers in different cities stand around a new engine, and each of them can pull out parts of the motor, discuss them, and see each other, as if they are in the same room
a medical-school professor and students stand around a virtual person and interact with the organs and systems, pulling them out or highlighting them as necessary for the lesson, as Figure 1 shows
To complete this continuum, I should probably include screen reality, or just screens, in the mix. UX designers have been designing screens for the past few decades—for desktop applications, Web pages, mobile apps, kiosks, and other display units. These are flat displays that have distinct boundaries, which users view at a comfortable distance.
To prepare for a career expansion or transition from screen reality to VR, AR, or MR, UX designers must learn new frameworks, patterns, and techniques. Plus, the experience design skills the various immersive technologies require are not necessarily the same. Creating a whole new reality in which people can fully immerse themselves is a very different challenge from designing digital overlays or holograms to fit specific needs and contexts. The most effective strategy for making this transition would probably be to target an area of special interest to you or a skill that you would like to develop first, then expand and adapt your skills as the technology and adoption patterns evolve. The approaching inflection point that immersive technology represents is a fantastic opportunity for you to become involved in work that you’ll love—work that employs your strongest skills. Throughout the rest of this article, I’ll discuss some career steps you can take to get started on this exciting journey.
Career Step 1: Dive in and Have Fun
The first step on this career journey is to explore the new immersive technologies for yourself. There are lots of opportunities to do this popping up all over the place. (This is what inspired me to write this article and organize the first XD Immersive conference.) Just have fun with it! Drive a virtual car through Provence. Use an AR headset that gives you ready access to detailed information about whatever scene you’re viewing. Try a mixed-reality headset that makes 3D holograms appear in the same room with you. In Figure 2, the UX STRAT team is trying out some VR applications.
As you become immersed in these VR and AR experiences, imagine what aspects of the experience design you could see yourself taking on: visual, interactive, narrative, strategic, or conceptual. Then, consider what steps you would need to take to expand or transition your professional activities from designing screen reality to designing virtual, augmented, or mixed reality.
Let’s look at some examples of immersive experiences that are already on the market, which you can try out for yourself.
The German company Realities.io has released a set of virtual-reality tours that work on a number of different VR headsets. As shown in Figure 3, one tour offers realistic immersive experiences of a number of famous cathedrals. The technique they use is photogrammetry, which involves compiling photographic images that have been taken from multiple angles, then stitching them together to create realistic depth, and optimizing them using 3D modeling. The result is a photorealistic 3D environment that you can explore in virtual reality. As you zoom in, the images become clearer, rooms keep their proper dimensions, and the light shifts realistically.
Lowe’s Holoroom virtual-reality experience combines an in-store iPad app for kitchen and bathroom design with an Oculus headset. Using this system, Lowe’s shoppers can design their ideal bathroom or kitchen, walk into it virtually, share it via YouTube 360, then purchase the products they’ve chosen.
Lowe’s has launched another virtual-reality system called Holoroom How To, which immerses customers in a DIY project—such as tiling a shower—giving them step-by-step instructions to complete the task. Haptic feedback—such as feeling the vibration of a drill through the controller—adds to the virtual experience.
Mercedes, Cadillac, and Audi have developed automotive VR experiences that allow customers to try out new models in three different ways: seated inside the car, walking around outside it, or driving down the road. Since these are car-buying VR applications, customers can usually change the interior and exterior colors, as well as choose from an extremely broad range of optional packages, creating the exact model they want to buy. In this way, VR lets car buyers see an infinitely larger array of car models and trim packages than a dealer could stock.
Marriott Virtual Reality Transporter
The Marriott Virtual Reality Transporter lets customers visit exotic Marriott destinations virtually—such as the Wai’anapanapa Black Sand Beach in Maui or the top of Tower 42 in London. The Marriott system uses Oculus Rift, 4D sensory elements, 360-degree live-action video, and film-quality computer graphics.
IKEA Place AR App
The IKEA Place AR app, shown in Figure 4, lets customers browse more than 2000 IKEA products and place them in their home, using the augmented-reality functionality built into iOS 11. Products include chairs, tables, sofas, and storage units. First, the customer scans a room and lets the app process the data. Next, the customer selects an item of furniture from the virtual catalog, then points the phone’s camera at the floor to position the item within the room.
BMW i Samsung Visualizer App
The BMW i Samsung Visualizer app lets customers experience the BMW i3 and i8 cars on a smartphone or tablet. They can select a range of options for the car—such as exterior colors and wheels—then test drive it. The app uses Google’s AR technology Tango, which brings spatial depth and a 360-degree view to devices by deploying computer vision, image processing, and special-purpose optical sensors. As the device moves, the environment changes in the same way it would in physical reality.
NextVR has struck a deal with the NBA (National Basketball Association) and NFL (National Football League) to provide a virtual-reality game experience. Because this is a paid service, it won’t be as straightforward for UXmatters readers to experience this example, but I’ve included it because this sort of sports application of VR could speed up the pace of the platform’s widespread commercial adoption. Figure 5 shows NextVR NBA, which puts viewers on the court.
Videos of VR and AR Applications
If you’d prefer simply to watch some videos that explore applications of VR and AR, follow these links to explore what immersive technology can do, without leaving your home or office:
Career Step 2: Learn the Concepts of Immersive Experience Design
The good news is that many UX principles and practices for designing screen reality carry over to immersive experience design: interactivity, pacing, narrative, affordances, structure, conversion, visual design, content, information architecture, mental models, analytics, accessibility, segmentation, front-end development, prototyping, and more.
At the same time, it’s clear that UX professionals who want to design immersive experiences will have to learn some new concepts, skills, and techniques. These vary depending on the specific type of immersive experience you’re designing—virtual, augmented, or mixed. Next, I’ll describe a few aspects of immersive experience design that are substantially different from screen-based experience design.
Reality as a Benchmark
One of the most challenging aspects of designing virtual-reality experiences is that the experience must seem real. When the human brain detects cues that what a person is seeing, hearing, or feeling does not match what that person would expect from any of the senses, this detracts from the experience. It’s not that you cannot ever bend or even break the rules of the physical world—otherwise dreams would not seem real. It’s just that a user’s degree of immersion in a virtual experience typically depends on how real it feels. As Agent Smith laments in the movie The Matrix: “The perfect world [we created] was a dream that your primitive cerebrum kept trying to wake up from.” However, because both augmented and mixed reality involve superimposing the digital onto the real world, achieving a high degree of reality is not as critical on these platforms as it is in virtual reality.
Designing in 3D
Whether you’re creating a VR, AR, or MR experience, the UX professionals on a team must design for three dimensions. Designing in 3D was a common task for visual designers back in the day of CD-ROM multimedia, but many of us left this skill behind when most design projects shifted to the flat Web. Designing in 3D is one of the first skills UX designers need to master in preparing for immersive experience design. As with many of these design concepts, people who are not UX specialists have most of the responsibility for creating the effects—in this case, 3D modelers. But UX professionals still need to understand the concepts to be able to support production.
For both Web and mobile design, a person using a device has only a single orientation to the screen—a perpendicular line of sight, between about 6 and 18 inches away from the screen. Immersive technology is very different because users can look in any direction—even in unexpected places such as under an object. The user’s view must look realistic from any angle and accurately incorporate peripheral vision. When the user’s brain perceives that his position is not what he expects from his previous real-world experience, this breaks the virtual experience. While position-tracking is the task of engineers, a UX professional who is designing an immersive experience must have a very clear understanding of the user’s ever-changing position within the experience.
The human brain has numerous capabilities that come into play in designing virtual and mixed reality, including developing a comprehensive sense of presence—that is, Where am I and what does this mean? Immersive experiences rely on simulating a sense of presence by creating believable virtual- and mixed-reality scenes. This is very difficult to do, and getting it right often requires numerous prototypes.
In Web and mobile design, the people using the technology remain themselves throughout the experience. However, in virtual reality, there must be a virtual representation of each user that is viewable by others as a virtual character. Designing the identity and attributes of this virtual character may not be a trivial exercise, depending on the specific application of virtual-reality technology. If the application is a flight or automotive simulator, the virtual character should represent the real physical characteristics of the user in terms of size, weight, and motor coordination. But, if an application is a psychiatric simulation, modifying brain patterns or building new memories or experiences, the virtual self might vary quite a bit from the real self. For example, on a team of engineers designing a new car, a team member I see working with me could simply be a video of a real person, or it could be an avatar representing only a subset of the characteristics of the real person.
Representing the Virtual World
In VR, a virtual world completely replaces the physical world. Creating new worlds is not something with which most UX professionals who design Web sites and mobile apps are typically familiar. For most business, non-game VR applications, the virtual world consists of a real-world environment that is accessible to people who are not physically present—for example, the summit of Kilimanjaro or the NBA finals. In both of these cases, the virtual world doesn’t need to be created so much as accurately, usefully, or interestingly represented. Nevertheless, this virtual world is something with which the UX designer must be intimately acquainted and that the designer must address in design documentation or sketches.
The structure of a virtual-reality world usually has a foreground, middle ground, and background. The relative size and placement of these three elements are aspects of UX design. Although you’ll share these decisions with a number of other members of your team. The foreground includes the user’s hands, controllers, menus, and objects with which users can interact directly. The middle ground contains objects or locations with which the user can interact and that the user can move toward. The background is a setting for the experience. The Leap Motion Design Guide teaches virtual-world design, as shown in Figure 6.
Head Tracking with Head-Mounted Display Units
When users are immersed in a VR, AR, or MR experience, they can look in any direction. The device they are using needs to track the position of their head and the direction of their eyes to present the user with a realistic view at every turn. The design of an immersive experience must take all possible head motions into account during the planning phase. Head tracking is accomplished using a head-mounted display unit.
The key new element of the immersive-technology platform is the head-mounted display (HMD) unit. The big battle today is over which HMD will become the standard and which will fall by the wayside. Should computing power be internal or external to the HMD? Should it have cables? What operating system and software architecture should it use? While competitive battle lines are drawn, we still need to prepare for the shootout.
The first thing users need when they enter a virtual world is training on how they can interact with it. VR orientation training focuses primarily on the haptics of the virtual world—that is, how to convey sensation and control. For example, users need to know how to grasp an item, pick it up, and move it from place to place.
In addition to moving objects, users need to learn how to move themselves from one place to another within the virtual experience. While motion within the context of Web and mobile interactions is typically either linear or branching, users must be able to move in any direction within immersive experiences. Although some game designers already know how to plan for omnidirectional motion, this will be new for many UX professionals.
Lighting and Parallax
Lighting and parallax are important aspects of human perception of physical reality. Our ability to detect different lighting patterns in the physical world is very highly developed. Recreating such lighting patterns in a virtual world requires great skill. In immersive experience design, this task will most likely fall to experts in 3D modeling, but it is an important aspect of UX design for VR.
Parallax refers to the way objects appear in relation to each other when the user moves—particularly with respect to their relative size, angles, and depth. If parallax is off, users may become nauseous.
New Types of Interactions
Early Web sites implemented very few interactions—for example, click, hover, scroll, and play. Smartphones introduced some new gestures such as pinch and flick. Immersive technologies will offer a host of new things that users can do—such as grasping items, picking them up, or tossing them. Some of these interactions are becoming standardized in early products, but the library of interaction possibilities will probably grow substantially as adoption increases and the technology evolves. In designing these interactions, UX professionals must rely on nonlinear approaches rather than traditional task flows. Exploration and experience play major roles in VR technology, as opposed to goal completion. While AR and MR interactions are more goal oriented, nonlinear exploration will still be much more common than on previous platforms.
Physics plays a much bigger role in VR design than in Web design. A major factor in making the user’s experience of a virtual world feel real has to do with force being proportional to mass and acceleration. While computer games such as Mario Cart take some aspects of physics into account, a fully immersive, designed world must feel internally consistent. Although an engineer or other specialist is responsible for the physics of an immersive experience, a UX professional designing the experience at a conceptual level must be acquainted with the issues, capabilities, and constraints that physics present.
In VR, sound design helps create the illusion of reality by mimicking a 3D environment. The sound of a car, moving from left to right in the background of the user’s perceived space, supports an image or a video of driving through a city. Sound specialists create the 3D sounds, but virtual experience designers must be able to specify and describe sound effects at a storyboarding or conceptual level.
User anatomy will play a much larger role in immersive experience design than it has in Web or mobile design. For example, pupillary distance is a key factor in the optimal perception of 3D imagery. Hand placement is important to successful interactions. When the user extends an arm, the same interaction must happen virtually as well. User representation—in terms of avatar shape and physical characteristics such as virtual height and line of sight—will be important design decisions early in the conceptual stage of experience design.
Protection from Shock
Immersive technology has the potential to shock users because the scenarios these applications portray look much more real than those in other media. Make sure users can opt out of content they may find repugnant, shocking, or jarring.
Engaging All Five Senses
As realism increases, so will the use of so-called 4D elements—that is, hearing, smell, touch, and taste. The human brain uses all of these senses to experience physical reality, so ultimately, creating a true virtual experience that supplants physical reality must involve all of the senses as well.
Ethics of Immersive Design
UX designers have already engaged in some ethical discussions—such as gender bias and dark patterns. But the availability of technology that can meet a whole range of human needs in a realistic way is bound to bring a host of ethical questions with it. Should things that are illegal in the physical world also be illegal in the virtual world? Are there societal boundaries in physical reality that we’ll ignore in virtual reality? There will inevitably some disagreements about the ethics of VR, so it’s important to start these conversations now.
New Approaches to Accessibility
Some accessibility experts are excited about creating virtual and augmented worlds that take advantage of the senses that are not impaired. At the same time, immersive experiences focus primarily on sight and sound, so accessibility must still play an important role in the design of immersive experiences.
Safety and Motion Sickness
It isn’t uncommon for people using VR headsets for the first time to report feeling nauseous from the experience. This is mainly because the brain expects one result when the user moves her head, but experiences something different—for example, uneven acceleration. Many VR design manuals provide instructions on how to give users a sense of position, presence, and motion within the virtual world, without making them feel ill. Safety is also a consideration for general usage guidelines—for example, preventing users from walking full force into walls or door jambs or tripping over items in their immediate physical environment.
Career Step 3: Find Opportunities to Ramp Up on Immersive Experience Design
Fortunately, there is an ever-increasing number of ways to expand your skills and capabilities in preparation for immersive experience design. I’ll describe a few resources from which you can learn now.
Attend Relevant Events to Grow Your Knowledge and Skills
XD Immersive will offer a great opportunity to hear about the latest immersive experience designs from the industry leaders who are creating them. You’ll also be able to try out a range of new immersive experience designs in the expo at the conference. There are many other conferences and workshops on this topic that serve a variety of objectives. The focus of XD Immersive is on experience design for viable immersive products rather than on the technology itself.
Read Articles and Books
Since VR and AR have been around for a while, awaiting greater adoption, many articles and videos have been published on this topic. Thus, it is relatively easy to find a broad range of learning materials.
Major players in the immersive experience space are creating their own standards documentation. However, the idea of standards this early in the game may be wishful thinking. Nevertheless, design best practices for the Oculus have been documented in detail—although these guidelines are for VR developers rather than UX designers. Leap Motion has also published a VR design manual for developers.
Seek Out Immersive Projects to Work On
Your company or agency may have some smaller, less risky projects involving immersive technology that would provide a gentler learning curve than being thrown into the deep end on a complex, risky project later on. Just showing an interest in the technology and ramping up on immersive experience design may be enough to get you an assignment that you can use as a stepping stone to prepare yourself for more involved projects.
Immersive technology is going to be the most significant platform introduction since mobile phones. Companies will need UX professionals to help design the vast array of VR, AR, and MR applications that they’ll release. However, there will be a substantial learning curve for UX professionals who want to design these new applications. Now is the perfect time to seek opportunities that will help you to prepare for UX design’s inevitable expansion to encompass immersive experience design. UX professionals must prepare for this transition.
Paul organizes the UX STRAT conferences and workshops to help experienced UX, CX, Product, and Service Design professionals continue to grow their skills, networks, and careers. A UX strategist and researcher, he also consults with companies to help them evaluate and grow their UX Strategy capabilities. He began designing ecommerce Web sites in 1995, in Barcelona, Spain; then founded Retail UX in 2002. Paul’s consulting clients have included some of the most successful corporations in the world—such as The Home Depot, Coca-Cola, SAP, Delta Air Lines, Philips, Macy’s, Bloomingdale’s, Cox, and GE. Paul manages the UX / CX / Product / Strategy Group on LinkedIn. Read More