Spatial Computing: A New Paradigm of Interaction

Conscious Experience Design

Designing for the evolving human+machine relationship

A column by Ken Olewiler
February 19, 2024

As UX designers, along with the rest of the world today, we’re hyperaware of the impact and momentum of generative artificial intelligence (AI)—so much so that we’re now wondering whether people might be focusing so much on a few trees that they’re forgetting to consider the forest. Allow me to explain. While AI is undeniably a sea change in computing, it ultimately represents a much broader revolution in which technology is becoming more human centric and human conscious. Essentially, technology is now learning to adapt to people, as opposed to people needing to learn and adapt to new technologies.

As part of this shift, technology is expanding not only its cognitive abilities but also its sensory, social, and ethical capabilities. Within the expansion of technology’s sensory abilities, we’re seeing advancements and growth in spatial computing. Spatial context and movement within three-dimensional spaces are core human-sensory abilities, and thus, likely new growth areas in humanizing machine interactions. Spatial computing has emerged as one of the most compelling paradigms that are melding with AI—so compelling that we can consider it the third wave of interactions in personal computing.

Champion Advertisement
Continue Reading…

Convergence of the Physical and the Digital

The first era of personal computing created two-dimensional (2D) digital experiences that exist within the constraints of a 2D screen. These 2D experiences evolved from desktop-computing applications to Web services. Next, the mobile-computing era brought our physical and digital experiences closer together with smartphones that combine the ability to access and share data from anywhere with user interfaces that combine services such as multimedia, location, personalization, and public and private data, all being delivered via hundreds of thousands of consumer and business apps.

Spatial computing blends our digital and physical experiences across nearly imperceptible boundaries that mesh the physical and digital worlds into a single digital canvas. Using AI and other technologies to support new interactions, spatial computing delivers spatial experiences that enable people to communicate and interact with each other, the physical world, and other computing machines. Unlike in previous eras, digital devices now have the ability to intuit and learn from both the physical world and human behaviors and have unbounded access to the world’s data and computing power. Spatial computing asks us to reimagine our interactions with others and our physical and digital worlds, rewriting the rules for living, working, playing, and communicating.

Platform Implications

Spatial computing seamlessly integrates mixed reality (MR), augmented reality (AR), and virtual reality (VR) into a cohesive new computing platform, laying the groundwork for a diverse spectrum of immersive experiences. As a platform, spatial computing lets us create new content, products, experiences, and services in both physical and virtual environments, and enables them to interact seamlessly. A key implication of spatial computing is that it extends computing into everything people can see, feel, touch, experience, or communicate with or through.

Although spatial computing is a relatively new term to the masses, since its introduction with the Apple Vision Pro, the shift toward more natural and spatial ways of interacting with machines has been underway for a while.

Formerly, we divided user experiences into physical environments, comprising people, devices, rooms, and furniture, and the digital render—that is, everything we see on a screen. Spatial computing removes that boundary. We’re integrating digital assets such as three-dimensional (3D) user interfaces, avatars, and audio into the physical world around the user, enabling people to interact with both digital information and the physical world in a blended and natural way.

Since spatial applications rely on AI, they can understand context. Implicit user cues inform sensors that track gestures, gaze, emotions, and other biometric information within the context of user objectives and activities. This means people can now interact with digital assets as naturally as they might with physical objects.

Four Design Principles for Spatial Computing

So how can UX designers get started with spatial computing? Punchcut’s Immersive Design team has been working on spatial-design solutions and have put together some starting principles relating to designing for spatial computing in its current state of evolution. In brief, these spatial-design principles include the following:

  1. Immerse intentionally.
  2. Strategize spatially.
  3. Naturalize interactions.
  4. Engage the senses.

These spatial-design principles provide a good place to start. Let’s explore them in greater depth.

1. Immerse Intentionally.

Ensure that the spatial-computing experience uses appropriate levels, or models, of immersion. Spatial computing can offer a full spectrum of levels of immersion, starting in physical space where the user currently exists; through mixed reality, which integrates digital information into the physical world; all the way to a fully virtual environment that exists only in the digital world. Although spatial experiences can be more fluid than traditional, screen-only experiences, there are optimal levels or degrees of immersion that work better for certain use cases.

UX designers can deliver immersive experiences with new forms of engagement and layers of functionality within a physical space that integrates into the user’s real-world surroundings. Or we can create spaces that transport users to an entirely new digital environment that aligns with their needs and objectives. For example, a virtual showroom could demonstrate products in ways that convincingly and naturally simulate physical-world experiences.

Because spatial computing offers different levels of immersion, we must choose the appropriate level of immersion to deliver a meaningful, engaging digital experience to users, as follows:

  1. Level 1—Augmenting the user’s actual physical space and real-life experience can result in the user’s having a richer experience of the world.
  2. Level 2Mixed-reality experiences bring digital information to the physical world, whether through 2D, flat-panel windows; three-dimensional (3D) volumes; or partially immersive experiences that transport users to new environments without losing their grounding in the real world.
  3. Level 3—Fully immersive, virtual-reality (VR) environments enable users to be fully present in a virtual environment, in which they can become deeply engaged.

2. Strategize Spatially.

Understand how to strategically position people and content optimally within a shared space. Therefore, you should design for scenes, not screens. Approach spatial computing as an infinite canvas or theater of space, where you’re free of the physical constraints that traditionally confine user experiences to the size of a device’s screen. Designing for immersive digital experiences mirrors the fundamentals of traditional stagecraft, including scenery, lighting, costumes, props, and sound engineering. Similar to physical stages, the spatial environment should pull the audience into the experience and help them forget the world outside, but with one distinct difference: the user is in the middle of the action. This shift requires new ways of researching, testing, and conceiving of spatial user experiences.

Proxemics, the study of the spatial relationships between people and objects within an environment, is a crucial element of design for spatial computing. UX designers must strategically position people and content in space. This extra depth requires designers to consider how to position elements in space in proximity-appropriate ways to ensure that the user feels comfortable. To ensure user comfort and efficiency, position elements within proxemic zones, as follows:

  • intimate space, or immediate space—This zone is closest to the user and, thus, is reserved for highly personal items and lean-in experiences. This ring occupies the space from zero up to 1.5 feet around the user.
  • personal space—The space in the next ring out, in which the user can position personal objects and interactions need to be within an arm’s reach. This ring is approximately 1.5 to 4 feet around the user.
  • social space—This space is perfect for lean-out experiences and collaborative environments. This ring is approximately 4 to 12 feet around the user.
  • public space—This is the largest area of space and houses large-scale objects, grand experiences, and interactions in which users are not engaging directly with the user interface. This ring of space generally comprises 12 to 25 feet around the user.

3. Naturalize Interactions.

Limit interactions with digital elements to those that are simple and easy to learn. Consider using both implicit and explicit interactions. To build natural, effortless user experiences and create maximal familiarity, start with natural human movements, gestures, and behaviors. Utilize ambient intelligence to intuit more about the user’s behavior when engaging in physical interactions or using physical controls. Avoid awkward new paradigms and metaphors that neither feel natural in the physical world nor are tailored to the context.

Spatial computing affords the ability to interact with digital elements in natural ways that enable explicit and implicit interactions without the use of a controller. Enable users to interact with the digital world naturally and effortlessly by doing the following:

  • Intentionally design elements within a scene, placing them relative to proxemic zones. Think about how the user interacts with these elements explicitly and implicitly.
  • The user consciously performs explicit interactions—for instance, pushing a button, grabbing a door handle, or turning a steering wheel—with an expected result in mind.
  • The user performs implicit interactions subconsciously, and the technology and others around the user infer the users intent. Implicit interactions include the user’s gaze, facial expressions, and body language. Spatial-computing platforms can understand the environment around the user by using contextual awareness to capture these signals.

4. Engage the Senses.

Improve the user’s core experience by considering all human perceptions. As you plan the user experience, account for the use of multisensory modalities. Spatial computing lets us holistically engage the user’s senses and facilitate sensory experiences that are not only natural but also engaging and memorable. For what was originally a single-modality experience, ensure that you provide multimodal channels to enhance, enrich, and provide more inclusive access to the user experience. Chart each sensory path within the sensory map to outline the user’s intended behavior and expression within each modality.

Spatial computing enables us to holistically engage the user’s senses. To achieve this level of interaction, UX designers must layer multiple senses to create more natural, memorable, and engaging interactions for users, as follows:

  1. Create richer, higher-value experiences by considering all human senses. Start with the basics: visual, auditory, and tactile. Then layer in more advanced senses to enrich the user’s environment and the user experience.
  2. Layer in advanced senses to enrich the user experience. Consider ways in which you can incorporate all the senses, including olfactory (smell), gustatory (taste), vestibular (balance and spatial orientation), proprioception (awareness of the body in space and the strength that is necessary to complete an action), and interoception (awareness of internal body states). 

Waking Up to New Realities

With so many potential advantages, what’s stopping companies from completely porting their existing applications and solutions straight into immersive worlds? At Punchcut, we believe that, just because you can, doesn’t mean you should. We’re in just the earliest phases of the spatial-computing era. Even with the remarkable advancements of Apple Vision Pro, some users find the experience isolating and awkward. However, as this technology advances and becomes less bulky, we believe that spatial computing will become universally applicable.

As businesses invest in spatial-computing experiences, it’s important to remember that beneath the novelty of any product demo, deep value must exist to achieve long-lasting user engagement and loyalty. As UX designers, it’s critical that we be intentional when designing spatial experiences, make sufficient space to conduct adequate research, and iterate on our design solutions to increase both our products’ lifespans and the enjoyment of users.

Adopting spatial computing requires careful planning. Industry leaders such as Meta, Apple, and Microsoft have laid solid foundations and established best practices that we can build upon. Starting with research and deliberate exploration of the applicability of spatial computing to specific products and objectives is critical. Spatial computing is still sufficiently novel that planning your product roadmap now can position your company and its products ahead of the rest when extended reality inevitably becomes mainstream.

Finally, I’d like to recognize the Punchcut Immersive Design and Labs team, who have led our exploration of spatial computing and come up with our approach to embracing it in real-world applications, which I’ve outlined in this column. 

Managing Partner at Punchcut

San Francisco, California, USA

Ken OlewilerKen was a co-founder of Punchcut and has driven the company’s vision, strategy, and creative direction for over 20 years—from the company’s inception as the first mobile-design consultancy to its position today as a design accelerator for business growth and transformation. Punchcut works with many of the world’s top companies—including Samsung, LG, Disney, Nissan, and Google—to envision and design transformative product experiences in wearables, smart home Internet of Things (IoT), autonomous vehicles, and extended reality (XR). As a UX leader and entrepreneur, Ken is a passionate advocate for a human-centered approach to design and business. He believes that design is all about shaping human’s relationships with products in ways that create sustainable value for people and businesses. He studied communication design at Kutztown University of Pennsylvania.  Read More

Other Columns by Ken Olewiler

Other Articles on Interaction Design

New on UXmatters