Preparing Your UX Career for Virtual Reality

A Sliding Scale of Reality

The nomenclature of immersive technology is pretty straightforward—although there is some competition among the giants in this industry to have their nomenclature accepted as the standard. Let’s look at some common terms.

Physical reality is the world of mass and energy that Einstein described at the beginning of the 20th century. We can perceive physical reality with the five senses, without any digital enhancements.

Virtual reality (VR) refers to a completely immersive simulated digital reality. VR completely replaces the visual and auditory stimuli of physical reality with a designed environment. Smell, taste, and touch are not far behind. Some potential virtual-reality applications include the following:

• viewing the Super Bowl in real time, from the sideline at the line of scrimmage or from the perspective of any camera on the field, with the ability to choose your view
• taking a self-directed virtual tour of the Fiji Islands, exploring a wide range of hospitality and entertainment options first hand
• learning a range of new interpersonal skills, interacting with virtual actors in realistic situations that adapt to your responses and actions

The term augmented reality (AR) typically describes digital data and imagery that are superimposed on physical reality. It’s like having a computer screen overlay what you normally perceive as your world. Some potential augmented-reality applications include the following:

• touring Paris while seeing an overlay that presents interactive data and descriptions for any building, monument, or art work you select
• approaching a highway exit and seeing immediately recognizable food, fuel, and hospitality options

Mixed reality (MR)—which Microsoft is pushing hard as a differentiated immersive experience—incorporates virtual or holographic objects that are anchored in place within physical reality. While AR data and imagery move with you when you walk around, MR virtual objects stay in the same place, so you can view them from different angles. They get larger as you move closer to them. Some potential mixed-reality applications include the following:

• a group of engineers in different cities stand around a new engine, and each of them can pull out parts of the motor, discuss them, and see each other, as if they are in the same room
• a medical-school professor and students stand around a virtual person and interact with the organs and systems, pulling them out or highlighting them as necessary for the lesson, as Figure 1 shows

To complete this continuum, I should probably include screen reality, or just screens, in the mix. UX designers have been designing screens for the past few decades—for desktop applications, Web pages, mobile apps, kiosks, and other display units. These are flat displays that have distinct boundaries, which users view at a comfortable distance.

To prepare for a career expansion or transition from screen reality to VR, AR, or MR, UX designers must learn new frameworks, patterns, and techniques. Plus, the experience design skills the various immersive technologies require are not necessarily the same. Creating a whole new reality in which people can fully immerse themselves is a very different challenge from designing digital overlays or holograms to fit specific needs and contexts. The most effective strategy for making this transition would probably be to target an area of special interest to you or a skill that you would like to develop first, then expand and adapt your skills as the technology and adoption patterns evolve. The approaching inflection point that immersive technology represents is a fantastic opportunity for you to become involved in work that you’ll love—work that employs your strongest skills. Throughout the rest of this article, I’ll discuss some career steps you can take to get started on this exciting journey.

Career Step 1: Dive in and Have Fun

The first step on this career journey is to explore the new immersive technologies for yourself. There are lots of opportunities to do this popping up all over the place. (This is what inspired me to write this article and organize the first XD Immersive conference.) Just have fun with it! Drive a virtual car through Provence. Use an AR headset that gives you ready access to detailed information about whatever scene you’re viewing. Try a mixed-reality headset that makes 3D holograms appear in the same room with you. In Figure 2, the UX STRAT team is trying out some VR applications.

As you become immersed in these VR and AR experiences, imagine what aspects of the experience design you could see yourself taking on: visual, interactive, narrative, strategic, or conceptual. Then, consider what steps you would need to take to expand or transition your professional activities from designing screen reality to designing virtual, augmented, or mixed reality.

Let’s look at some examples of immersive experiences that are already on the market, which you can try out for yourself.

Realities.io

The German company Realities.io has released a set of virtual-reality tours that work on a number of different VR headsets. As shown in Figure 3, one tour offers realistic immersive experiences of a number of famous cathedrals. The technique they use is photogrammetry, which involves compiling photographic images that have been taken from multiple angles, then stitching them together to create realistic depth, and optimizing them using 3D modeling. The result is a photorealistic 3D environment that you can explore in virtual reality. As you zoom in, the images become clearer, rooms keep their proper dimensions, and the light shifts realistically.

Lowe’s Holoroom

Lowe’s Holoroom virtual-reality experience combines an in-store iPad app for kitchen and bathroom design with an Oculus headset. Using this system, Lowe’s shoppers can design their ideal bathroom or kitchen, walk into it virtually, share it via YouTube 360, then purchase the products they’ve chosen.

Lowe’s has launched another virtual-reality system called Holoroom How To, which immerses customers in a DIY project—such as tiling a shower—giving them step-by-step instructions to complete the task. Haptic feedback—such as feeling the vibration of a drill through the controller—adds to the virtual experience.

Automotive VR

Mercedes, Cadillac, and Audi have developed automotive VR experiences that allow customers to try out new models in three different ways: seated inside the car, walking around outside it, or driving down the road. Since these are car-buying VR applications, customers can usually change the interior and exterior colors, as well as choose from an extremely broad range of optional packages, creating the exact model they want to buy. In this way, VR lets car buyers see an infinitely larger array of car models and trim packages than a dealer could stock.

Marriott Virtual Reality Transporter

The Marriott Virtual Reality Transporter lets customers visit exotic Marriott destinations virtually—such as the Wai’anapanapa Black Sand Beach in Maui or the top of Tower 42 in London. The Marriott system uses Oculus Rift, 4D sensory elements, 360-degree live-action video, and film-quality computer graphics.

IKEA Place AR App

The IKEA Place AR app, shown in Figure 4, lets customers browse more than 2000 IKEA products and place them in their home, using the augmented-reality functionality built into iOS 11. Products include chairs, tables, sofas, and storage units. First, the customer scans a room and lets the app process the data. Next, the customer selects an item of furniture from the virtual catalog, then points the phone’s camera at the floor to position the item within the room.

BMW i Samsung Visualizer App

The BMW i Samsung Visualizer app lets customers experience the BMW i3 and i8 cars on a smartphone or tablet. They can select a range of options for the car—such as exterior colors and wheels—then test drive it. The app uses Google’s AR technology Tango, which brings spatial depth and a 360-degree view to devices by deploying computer vision, image processing, and special-purpose optical sensors. As the device moves, the environment changes in the same way it would in physical reality.

Sports Leagues

NextVR has struck a deal with the NBA (National Basketball Association) and NFL (National Football League) to provide a virtual-reality game experience. Because this is a paid service, it won’t be as straightforward for UXmatters readers to experience this example, but I’ve included it because this sort of sports application of VR could speed up the pace of the platform’s widespread commercial adoption. Figure 5 shows NextVR NBA, which puts viewers on the court.

Videos of VR and AR Applications

If you’d prefer simply to watch some videos that explore applications of VR and AR, follow these links to explore what immersive technology can do, without leaving your home or office:

• Shopping (VR)
• Psychiatry (VR)
• Science Education (VR)
• Factory Maintenance (AR)
• Flight (AR)
• Anatomy Education (MR)
• Space Station (MR)

Career Step 2: Learn the Concepts of Immersive Experience Design

The good news is that many UX principles and practices for designing screen reality carry over to immersive experience design: interactivity, pacing, narrative, affordances, structure, conversion, visual design, content, information architecture, mental models, analytics, accessibility, segmentation, front-end development, prototyping, and more.

At the same time, it’s clear that UX professionals who want to design immersive experiences will have to learn some new concepts, skills, and techniques. These vary depending on the specific type of immersive experience you’re designing—virtual, augmented, or mixed. Next, I’ll describe a few aspects of immersive experience design that are substantially different from screen-based experience design.

Reality as a Benchmark

One of the most challenging aspects of designing virtual-reality experiences is that the experience must seem real. When the human brain detects cues that what a person is seeing, hearing, or feeling does not match what that person would expect from any of the senses, this detracts from the experience. It’s not that you cannot ever bend or even break the rules of the physical world—otherwise dreams would not seem real. It’s just that a user’s degree of immersion in a virtual experience typically depends on how real it feels. As Agent Smith laments in the movie The Matrix: “The perfect world [we created] was a dream that your primitive cerebrum kept trying to wake up from.” However, because both augmented and mixed reality involve superimposing the digital onto the real world, achieving a high degree of reality is not as critical on these platforms as it is in virtual reality.

Designing in 3D

Whether you’re creating a VR, AR, or MR experience, the UX professionals on a team must design for three dimensions. Designing in 3D was a common task for visual designers back in the day of CD-ROM multimedia, but many of us left this skill behind when most design projects shifted to the flat Web. Designing in 3D is one of the first skills UX designers need to master in preparing for immersive experience design. As with many of these design concepts, people who are not UX specialists have most of the responsibility for creating the effects—in this case, 3D modelers. But UX professionals still need to understand the concepts to be able to support production.

Position Tracking

For both Web and mobile design, a person using a device has only a single orientation to the screen—a perpendicular line of sight, between about 6 and 18 inches away from the screen. Immersive technology is very different because users can look in any direction—even in unexpected places such as under an object. The user’s view must look realistic from any angle and accurately incorporate peripheral vision. When the user’s brain perceives that his position is not what he expects from his previous real-world experience, this breaks the virtual experience. While position-tracking is the task of engineers, a UX professional who is designing an immersive experience must have a very clear understanding of the user’s ever-changing position within the experience.

Presence

The human brain has numerous capabilities that come into play in designing virtual and mixed reality, including developing a comprehensive sense of presence—that is, Where am I and what does this mean? Immersive experiences rely on simulating a sense of presence by creating believable virtual- and mixed-reality scenes. This is very difficult to do, and getting it right often requires numerous prototypes.

Identity

In Web and mobile design, the people using the technology remain themselves throughout the experience. However, in virtual reality, there must be a virtual representation of each user that is viewable by others as a virtual character. Designing the identity and attributes of this virtual character may not be a trivial exercise, depending on the specific application of virtual-reality technology. If the application is a flight or automotive simulator, the virtual character should represent the real physical characteristics of the user in terms of size, weight, and motor coordination. But, if an application is a psychiatric simulation, modifying brain patterns or building new memories or experiences, the virtual self might vary quite a bit from the real self. For example, on a team of engineers designing a new car, a team member I see working with me could simply be a video of a real person, or it could be an avatar representing only a subset of the characteristics of the real person.

Representing the Virtual World

In VR, a virtual world completely replaces the physical world. Creating new worlds is not something with which most UX professionals who design Web sites and mobile apps are typically familiar. For most business, non-game VR applications, the virtual world consists of a real-world environment that is accessible to people who are not physically present—for example, the summit of Kilimanjaro or the NBA finals. In both of these cases, the virtual world doesn’t need to be created so much as accurately, usefully, or interestingly represented. Nevertheless, this virtual world is something with which the UX designer must be intimately acquainted and that the designer must address in design documentation or sketches.

The structure of a virtual-reality world usually has a foreground, middle ground, and background. The relative size and placement of these three elements are aspects of UX design. Although you’ll share these decisions with a number of other members of your team. The foreground includes the user’s hands, controllers, menus, and objects with which users can interact directly. The middle ground contains objects or locations with which the user can interact and that the user can move toward. The background is a setting for the experience. The Leap Motion Design Guide teaches virtual-world design, as shown in Figure 6.

Head Tracking with Head-Mounted Display Units

When users are immersed in a VR, AR, or MR experience, they can look in any direction. The device they are using needs to track the position of their head and the direction of their eyes to present the user with a realistic view at every turn. The design of an immersive experience must take all possible head motions into account during the planning phase. Head tracking is accomplished using a head-mounted display unit.

The key new element of the immersive-technology platform is the head-mounted display (HMD) unit. The big battle today is over which HMD will become the standard and which will fall by the wayside. Should computing power be internal or external to the HMD? Should it have cables? What operating system and software architecture should it use? While competitive battle lines are drawn, we still need to prepare for the shootout.

Haptics

The first thing users need when they enter a virtual world is training on how they can interact with it. VR orientation training focuses primarily on the haptics of the virtual world—that is, how to convey sensation and control. For example, users need to know how to grasp an item, pick it up, and move it from place to place.

Omnidirectional Motion

In addition to moving objects, users need to learn how to move themselves from one place to another within the virtual experience. While motion within the context of Web and mobile interactions is typically either linear or branching, users must be able to move in any direction within immersive experiences. Although some game designers already know how to plan for omnidirectional motion, this will be new for many UX professionals.

Lighting and Parallax

Lighting and parallax are important aspects of human perception of physical reality. Our ability to detect different lighting patterns in the physical world is very highly developed. Recreating such lighting patterns in a virtual world requires great skill. In immersive experience design, this task will most likely fall to experts in 3D modeling, but it is an important aspect of UX design for VR.

Parallax refers to the way objects appear in relation to each other when the user moves—particularly with respect to their relative size, angles, and depth. If parallax is off, users may become nauseous.

New Types of Interactions

Early Web sites implemented very few interactions—for example, click, hover, scroll, and play. Smartphones introduced some new gestures such as pinch and flick. Immersive technologies will offer a host of new things that users can do—such as grasping items, picking them up, or tossing them. Some of these interactions are becoming standardized in early products, but the library of interaction possibilities will probably grow substantially as adoption increases and the technology evolves. In designing these interactions, UX professionals must rely on nonlinear approaches rather than traditional task flows. Exploration and experience play major roles in VR technology, as opposed to goal completion. While AR and MR interactions are more goal oriented, nonlinear exploration will still be much more common than on previous platforms.

Physics

Physics plays a much bigger role in VR design than in Web design. A major factor in making the user’s experience of a virtual world feel real has to do with force being proportional to mass and acceleration. While computer games such as Mario Cart take some aspects of physics into account, a fully immersive, designed world must feel internally consistent. Although an engineer or other specialist is responsible for the physics of an immersive experience, a UX professional designing the experience at a conceptual level must be acquainted with the issues, capabilities, and constraints that physics present.

Sound Design

In VR, sound design helps create the illusion of reality by mimicking a 3D environment. The sound of a car, moving from left to right in the background of the user’s perceived space, supports an image or a video of driving through a city. Sound specialists create the 3D sounds, but virtual experience designers must be able to specify and describe sound effects at a storyboarding or conceptual level.

User Anatomy

User anatomy will play a much larger role in immersive experience design than it has in Web or mobile design. For example, pupillary distance is a key factor in the optimal perception of 3D imagery. Hand placement is important to successful interactions. When the user extends an arm, the same interaction must happen virtually as well. User representation—in terms of avatar shape and physical characteristics such as virtual height and line of sight—will be important design decisions early in the conceptual stage of experience design.

Protection from Shock

Immersive technology has the potential to shock users because the scenarios these applications portray look much more real than those in other media. Make sure users can opt out of content they may find repugnant, shocking, or jarring.

Engaging All Five Senses

As realism increases, so will the use of so-called 4D elements—that is, hearing, smell, touch, and taste. The human brain uses all of these senses to experience physical reality, so ultimately, creating a true virtual experience that supplants physical reality must involve all of the senses as well.

Ethics of Immersive Design

UX designers have already engaged in some ethical discussions—such as gender bias and dark patterns. But the availability of technology that can meet a whole range of human needs in a realistic way is bound to bring a host of ethical questions with it. Should things that are illegal in the physical world also be illegal in the virtual world? Are there societal boundaries in physical reality that we’ll ignore in virtual reality? There will inevitably some disagreements about the ethics of VR, so it’s important to start these conversations now.

New Approaches to Accessibility

Some accessibility experts are excited about creating virtual and augmented worlds that take advantage of the senses that are not impaired. At the same time, immersive experiences focus primarily on sight and sound, so accessibility must still play an important role in the design of immersive experiences.

Safety and Motion Sickness

It isn’t uncommon for people using VR headsets for the first time to report feeling nauseous from the experience. This is mainly because the brain expects one result when the user moves her head, but experiences something different—for example, uneven acceleration. Many VR design manuals provide instructions on how to give users a sense of position, presence, and motion within the virtual world, without making them feel ill. Safety is also a consideration for general usage guidelines—for example, preventing users from walking full force into walls or door jambs or tripping over items in their immediate physical environment.

Career Step 3: Find Opportunities to Ramp Up on Immersive Experience Design

Fortunately, there is an ever-increasing number of ways to expand your skills and capabilities in preparation for immersive experience design. I’ll describe a few resources from which you can learn now.

Attend Relevant Events to Grow Your Knowledge and Skills

XD Immersive will offer a great opportunity to hear about the latest immersive experience designs from the industry leaders who are creating them. You’ll also be able to try out a range of new immersive experience designs in the expo at the conference. There are many other conferences and workshops on this topic that serve a variety of objectives. The focus of XD Immersive is on experience design for viable immersive products rather than on the technology itself.

Read Articles and Books

Since VR and AR have been around for a while, awaiting greater adoption, many articles and videos have been published on this topic. Thus, it is relatively easy to find a broad range of learning materials.

UXmatters recently published a whole chapter of Jason Jerald’s book The VR Book: Human-Centered Design for Virtual Reality.

Major players in the immersive experience space are creating their own standards documentation. However, the idea of standards this early in the game may be wishful thinking. Nevertheless, design best practices for the Oculus have been documented in detail—although these guidelines are for VR developers rather than UX designers. Leap Motion has also published a VR design manual for developers.

Seek Out Immersive Projects to Work On

Your company or agency may have some smaller, less risky projects involving immersive technology that would provide a gentler learning curve than being thrown into the deep end on a complex, risky project later on. Just showing an interest in the technology and ramping up on immersive experience design may be enough to get you an assignment that you can use as a stepping stone to prepare yourself for more involved projects.

Conclusion

Immersive technology is going to be the most significant platform introduction since mobile phones. Companies will need UX professionals to help design the vast array of VR, AR, and MR applications that they’ll release. However, there will be a substantial learning curve for UX professionals who want to design these new applications. Now is the perfect time to seek opportunities that will help you to prepare for UX design’s inevitable expansion to encompass immersive experience design. UX professionals must prepare for this transition. 

Reference

Parkin, Simon. “What Zuckerberg Sees in Oculus Rift.” MIT Technology Review, March 26, 2014. Retrieved November 30, 2017.