Audio and the User Experience

Beautiful Information

Discovering patterns in knowledge spaces

A column by Jonathan Follett
June 18, 2007

For most people, sound is an essential part of everyday living. Sound can deliver entertainment—like our favorite music or the play-by-play call of our hometown baseball—and vital information—like the traffic and news reports on the radio as we drive to work.

Audio signals also help us interact with our environment. Some of these signals are designed: We wake to the buzz of the alarm clock, answer the ringing telephone, and race to the kitchen when the shrill beep of the smoke alarm warns us that dinner is burning on the stove. Other audio signals are not deliberately designed, but help us nonetheless. For instance, we may know the proper sound of the central air conditioning starting, the gentle hum of the PC fan, or the noise of the refrigerator. So, when these systems go awry, we notice it immediately—something doesn’t sound right. Likewise, an excellent mechanic might be able to tell what is wrong with a car engine just by listening to it run.

Since people are accustomed to such a rich universe of offline sound, it’s notable that our digital user experiences—while far from silent—do not leverage audio information to the same extent that they do visual information. When designers and developers create user experiences—be they for Web applications, desktop applications, or digital devices—audio is often a missing ingredient.

Champion Advertisement
Continue Reading…

Digital audio is well established in many areas. It’s a common delivery medium for products and experiences like podcasts, music, and streaming Internet radio. Digital audio is becoming a common two-way communications medium through all the varied forms of voice over IP. And as a medium for interaction with computers of various kinds, voice interfaces are rapidly developing. In fact, advances in voice recognition and voice interfaces for phone support systems are making it increasingly likely that you’ll talk to a computer when you dial a customer service line. But despite these robust applications, the integration of audio into graphic user interfaces is still a rarity.

Seen But Not Heard: Audio’s Limited Role in UX

There are good reasons for audio’s limited role in UX: First and foremost, unwanted sound can be intrusive and annoying. Rather than enhancing a user experience, audio can be a distraction and reduce our effectiveness.

In the early days of Flash, circa 1999, poor use of audio aggravated users. I was as much to blame as any other designer when I discovered the rich tools Flash provided and proceeded to use them in a way that was detrimental to the user experience—from invasive background music loops, to clicks and bleeps giving user feedback for no good reason, to zooming and swooshing and other sounds accompanying animations. Spam and advertising audio on Web sites—perhaps announcing “You’ve won a free trip” or “Take our survey and receive a gift certificate”—made users want to turn off their computer speakers altogether.

Secondly, for many UX practitioners like myself, our backgrounds, schooling, and professional training were in the areas of visual design—often, in the silent medium of print—information architecture, and interaction design, not in audio or sound design. So, while the integration of text and graphics is a familiar and common occurrence, the use of audio is unfamiliar territory and rarely considered. Most of us have not learned and honed techniques for developing effective audio for user interfaces. And sound design is not an area of expertise that we can quickly understand: It’s a field with great depth and history, coinciding with that of modern music, theater, and film. For a sound design primer for UX professionals, read “Why Is That Thing Beeping? A Sound Design Primer on Boxes and Arrows.

Add to these factors the limited toolset that most designers have for incorporating sound into design projects, constrained bandwidth, lack of access to high-fidelity audio source material, and perhaps most importantly, no budget for a sound designer—unlike in film or TV—and it’s easy to see how audio gets relegated to second-tier status when it comes to delivering information through user interfaces.

Sounds Like Success

Despite all of these obstacles to our incorporating audio into our design solutions, there are some high-profile examples of successful sound design in UX. The most famous is AOL: An enthusiastic voice declared, “You’ve got mail!” and welcomed a generation of Internet newcomers—even spawning a movie with that title in 1998, starring Tom Hanks and Meg Ryan.

Or, more recently, Southwest Airlines has built an entire ad campaign around the importance users place on hearing the “Ding” audio cue through which the company’s discount airfare notification application lets people know a cheap flight is available. In one ad, a man lets his date—whose eyes are closed in anticipation of a kiss—fall to the floor when he hears that sound and rushes to check the message on his computer.

And there’s no end to the examples of good audio branding. Have someone start up a PC or Mac, and you can tell which computer is which—without looking—just by the distinctive sounds of their operating system jingles.

Expanding the Role of Audio

As applications move online, digital devices proliferate, and user interfaces spring up like dandelions, it’s worth asking whether user interfaces must remain mute or there are useful ways in which we can expand the role of audio in UX design. Specifically, are there ways to provide effective audio cues to better help direct or assist users?

We can examine other areas of design—industrial design and game design, in particular—for inspiration.

The video game industry has a long history of successfully using audio to enhance the user experience. Musical themes help establish key characters. Music and ambient sound build moods—whether cheery or spooky—based on game play. And games use sounds to steer gamers toward certain goals or notify them of the rewards they’ve earned.

The Philips HeartStart Home Defibrillator provides a compelling example of audio in industrial design. The device uses audio instructions along with information graphics to guide a lay person through the process of administering a shock to a victim in sudden cardiac arrest. Audio also saves lives in the complex user environment of the commercial airline cockpit, where proximity alarms that warn pilots of impending mid-air collisions give explicit spoken commands to ascend or descend in order to avoid disaster.

In emergency situations, where there is an urgent need for decision making, audio can be a compelling way of ensuring users take a certain action. We’re hardwired to obey audio commands, which is why these systems are effective. No clutter. No ambiguity. It’s loud and clear.

Audio Usage in UX Design

There are three areas where we can incorporate audio into our user experiences.

Ambient Background Sound

Our day-to-day lives are filled with background sound, some of which we enjoy—say, ocean waves at the beach—and other sounds that we strongly dislike—like construction noise.

While ambient sound works well in video game design to establish mood and location, it doesn’t have a clear role in other user experiences like software applications. However, as computers become quieter, will we miss the sound of the fan that indicated our device was running smoothly? Will people—especially in cubicle-filled offices—need the equivalent of a white-noise generator in their computers to raise the background sound level high enough to drown out distractions? (Anyone who has worked in an office in which the HVAC system has suddenly shut down knows just how noticeable their neighbors’ paper shuffling, coughs, and phone conversations can become.)

Task- and Event-Based Signals

When it comes to sound design for digital user experiences, this is the use of sound that is the most developed: Blips and bleeps indicate you’ve sent an item to the trash or an email message has just arrived.

But we can expand task-based audio cues to include actions that are specific to expert users. For example, in Photoshop, visually aligning an object to a pixel-specific guideline can be a challenge, especially when viewing an image at less than 100%. An audio signal—say, a simple click—to let you know the object is aligned to the guide—could increase the productivity of digital artists and visual designers.

Additionally, we often take for granted the offline audio cues that assist input to digital devices. The click of the mouse and the tap-tap of the keyboard reinforce the fact that we’ve completed a particular action—for instance, entering a line of text or selecting an object. This is fine when a user has a physical keyboard, but in cases where the user inputs data via a touch screen displaying a virtual keyboard, these sounds are noticeably absent.

Warnings and Notifications

In a 2003 column examining voice recognition and audio interfaces, usability guru Jakob Neilsen—after pointing out the numerous detrimental aspects of such systems—makes this semi-positive comment:

“…voice could be used to direct the user’s attention to important events or elements on the screen in a richer way than the obnoxious beep that currently constitutes most computers’ audio vocabulary.”

We could use voice audio to help walk users through critical situations or even to avoid such situations altogether. Laptop users might benefit from a voice reminder that their battery needs recharging or to back up their files before shutting down or the holy grail—a warning about an imminent hard drive failure.

Similarly, in less critical situations, we could use a voice reminder to indicate that a user has missed a field when filling out a form. Instead of forcing a user to scan an entire page to find the red text indicating which of a dozen fields he left blank, a voice could simply say, “Please type your name.”

The disadvantage to such notifications is that people quickly tune out false alarms. So, extensive testing would be necessary to keep inaccuracies to a minimum. And because audio is intrusive, it’s best to ask for a user’s permission before using audio notifications—and always provide a clear way of disabling the audio.

Audio, when used judiciously, enhances the user experience through its engagement of another human sense and by providing a richer atmosphere for interaction. We shouldn’t allow the abuses of audio in the past—which have resulted in the banishment of most audio from our daily online interactions—to prevent us from trying again to use audio more effectively. While incorporating audio cues and other sounds in UX projects may be foreign territory for most visually oriented UX professionals, it is territory well worth exploring. 

Principal at GoInvo

Boston, Massachusetts, USA

Jonathan FollettAt GoInvo, a healthcare design and innovation firm, Jon leads the company’s emerging technologies practice, working with clients such as Partners HealthCare, the Personal Genome Project, and Walgreens. Articles in The Atlantic, Forbes, The Huffington Post, and WIRED have featured his work. Jon has written or contributed to half a dozen non-fiction books on design, technology, and popular culture. He was the editor for O’Reilly Media’s Designing for Emerging Technologies, which came out in 2014. One of the first UX books of its kind, the work offers a glimpse into what future interactions and user experiences may be for rapidly developing technologies such as genomics, nano printers, or workforce robotics. Jon’s articles on UX and information design have been translated into Russian, Chinese, Spanish, Polish, and Portuguese. Jon has also coauthored a series of alt-culture books on UFOs and millennial madness and coauthored a science-fiction novel for young readers with New York Times bestselling author Matthew Holm, Marvin and the Moths, which Scholastic published in 2016. Jon holds a Bachelor’s degree in Advertising, with an English Minor, from Boston University.  Read More

Other Columns by Jonathan Follett

Other Articles on UX Design

New on UXmatters