Audio and the User Experience

By Jonathan Follett

Published: June 18, 2007

“Our digital user experiences—while far from silent—do not leverage audio information to the same extent that they do visual information.”

For most people, sound is an essential part of everyday living. Sound can deliver entertainment—like our favorite music or the play-by-play call of our hometown baseball—and vital information—like the traffic and news reports on the radio as we drive to work.

Audio signals also help us interact with our environment. Some of these signals are designed: We wake to the buzz of the alarm clock, answer the ringing telephone, and race to the kitchen when the shrill beep of the smoke alarm warns us that dinner is burning on the stove. Other audio signals are not deliberately designed, but help us nonetheless. For instance, we may know the proper sound of the central air conditioning starting, the gentle hum of the PC fan, or the noise of the refrigerator. So, when these systems go awry, we notice it immediately—something doesn’t sound right. Likewise, an excellent mechanic might be able to tell what is wrong with a car engine just by listening to it run.

Since people are accustomed to such a rich universe of offline sound, it’s notable that our digital user experiences—while far from silent—do not leverage audio information to the same extent that they do visual information. When designers and developers create user experiences—be they for Web applications, desktop applications, or digital devices—audio is often a missing ingredient.

Digital audio is well established in many areas. It’s a common delivery medium for products and experiences like podcasts, music, and streaming Internet radio. Digital audio is becoming a common two-way communications medium through all the varied forms of voice over IP. And as a medium for interaction with computers of various kinds, voice interfaces are rapidly developing. In fact, advances in voice recognition and voice interfaces for phone support systems are making it increasingly likely that you’ll talk to a computer when you dial a customer service line. But despite these robust applications, the integration of audio into graphic user interfaces is still a rarity.

Seen But Not Heard: Audio’s Limited Role in UX

“Rather than enhancing a user experience, audio can be a distraction and reduce our effectiveness.”

There are good reasons for audio’s limited role in UX: First and foremost, unwanted sound can be intrusive and annoying. Rather than enhancing a user experience, audio can be a distraction and reduce our effectiveness.

In the early days of Flash, circa 1999, poor use of audio aggravated users. I was as much to blame as any other designer when I discovered the rich tools Flash provided and proceeded to use them in a way that was detrimental to the user experience—from invasive background music loops, to clicks and bleeps giving user feedback for no good reason, to zooming and swooshing and other sounds accompanying animations. Spam and advertising audio on Web sites—perhaps announcing “You’ve won a free trip” or “Take our survey and receive a gift certificate”—made users want to turn off their computer speakers altogether.

Secondly, for many UX practitioners like myself, our backgrounds, schooling, and professional training were in the areas of visual design—often, in the silent medium of print—information architecture, and interaction design, not in audio or sound design. So, while the integration of text and graphics is a familiar and common occurrence, the use of audio is unfamiliar territory and rarely considered. Most of us have not learned and honed techniques for developing effective audio for user interfaces. And sound design is not an area of expertise that we can quickly understand: It’s a field with great depth and history, coinciding with that of modern music, theater, and film. For a sound design primer for UX professionals, read “Why Is That Thing Beeping? A Sound Design Primer on Boxes and Arrows.

Add to these factors the limited toolset that most designers have for incorporating sound into design projects, constrained bandwidth, lack of access to high-fidelity audio source material, and perhaps most importantly, no budget for a sound designer—unlike in film or TV—and it’s easy to see how audio gets relegated to second-tier status when it comes to delivering information through user interfaces.

Sounds Like Success

“Have someone start up a PC or Mac, and you can tell which computer is which—without looking—just by the distinctive sounds of their operating system jingles.”

Despite all of these obstacles to our incorporating audio into our design solutions, there are some high-profile examples of successful sound design in UX. The most famous is AOL: An enthusiastic voice declared, “You’ve got mail!” and welcomed a generation of Internet newcomers—even spawning a movie with that title in 1998, starring Tom Hanks and Meg Ryan.

Or, more recently, Southwest Airlines has built an entire ad campaign around the importance users place on hearing the “Ding” audio cue through which the company’s discount airfare notification application lets people know a cheap flight is available. In one ad, a man lets his date—whose eyes are closed in anticipation of a kiss—fall to the floor when he hears that sound and rushes to check the message on his computer.

And there’s no end to the examples of good audio branding. Have someone start up a PC or Mac, and you can tell which computer is which—without looking—just by the distinctive sounds of their operating system jingles.

Expanding the Role of Audio

“We can examine other areas of design—industrial design and game design, in particular—for inspiration.”

As applications move online, digital devices proliferate, and user interfaces spring up like dandelions, it’s worth asking whether user interfaces must remain mute or there are useful ways in which we can expand the role of audio in UX design. Specifically, are there ways to provide effective audio cues to better help direct or assist users?

We can examine other areas of design—industrial design and game design, in particular—for inspiration.

The video game industry has a long history of successfully using audio to enhance the user experience. Musical themes help establish key characters. Music and ambient sound build moods—whether cheery or spooky—based on game play. And games use sounds to steer gamers toward certain goals or notify them of the rewards they’ve earned.

The Philips HeartStart Home Defibrillator provides a compelling example of audio in industrial design. The device uses audio instructions along with information graphics to guide a lay person through the process of administering a shock to a victim in sudden cardiac arrest. Audio also saves lives in the complex user environment of the commercial airline cockpit, where proximity alarms that warn pilots of impending mid-air collisions give explicit spoken commands to ascend or descend in order to avoid disaster.

In emergency situations, where there is an urgent need for decision making, audio can be a compelling way of ensuring users take a certain action. We’re hardwired to obey audio commands, which is why these systems are effective. No clutter. No ambiguity. It’s loud and clear.

Audio Usage in UX Design

There are three areas where we can incorporate audio into our user experiences.

Ambient Background Sound

“Will people—especially in cubicle-filled offices—need the equivalent of a white-noise generator in their computers to raise the background sound level high enough to drown out distractions?”

Our day-to-day lives are filled with background sound, some of which we enjoy—say, ocean waves at the beach—and other sounds that we strongly dislike—like construction noise.

While ambient sound works well in video game design to establish mood and location, it doesn’t have a clear role in other user experiences like software applications. However, as computers become quieter, will we miss the sound of the fan that indicated our device was running smoothly? Will people—especially in cubicle-filled offices—need the equivalent of a white-noise generator in their computers to raise the background sound level high enough to drown out distractions? (Anyone who has worked in an office in which the HVAC system has suddenly shut down knows just how noticeable their neighbors’ paper shuffling, coughs, and phone conversations can become.)

Task- and Event-Based Signals

When it comes to sound design for digital user experiences, this is the use of sound that is the most developed: Blips and bleeps indicate you’ve sent an item to the trash or an email message has just arrived.

But we can expand task-based audio cues to include actions that are specific to expert users. For example, in Photoshop, visually aligning an object to a pixel-specific guideline can be a challenge, especially when viewing an image at less than 100%. An audio signal—say, a simple click—to let you know the object is aligned to the guide—could increase the productivity of digital artists and visual designers.

Additionally, we often take for granted the offline audio cues that assist input to digital devices. The click of the mouse and the tap-tap of the keyboard reinforce the fact that we’ve completed a particular action—for instance, entering a line of text or selecting an object. This is fine when a user has a physical keyboard, but in cases where the user inputs data via a touch screen displaying a virtual keyboard, these sounds are noticeably absent.

Warnings and Notifications

In a 2003 column examining voice recognition and audio interfaces, usability guru Jakob Neilsen—after pointing out the numerous detrimental aspects of such systems—makes this semi-positive comment:

“…voice could be used to direct the user’s attention to important events or elements on the screen in a richer way than the obnoxious beep that currently constitutes most computers’ audio vocabulary.”

We could use voice audio to help walk users through critical situations or even to avoid such situations altogether. Laptop users might benefit from a voice reminder that their battery needs recharging or to back up their files before shutting down or the holy grail—a warning about an imminent hard drive failure.

“Because audio is intrusive, it’s best to ask for a user’s permission before using audio notifications—and always provide a clear way of disabling the audio.”

Similarly, in less critical situations, we could use a voice reminder to indicate that a user has missed a field when filling out a form. Instead of forcing a user to scan an entire page to find the red text indicating which of a dozen fields he left blank, a voice could simply say, “Please type your name.”

The disadvantage to such notifications is that people quickly tune out false alarms. So, extensive testing would be necessary to keep inaccuracies to a minimum. And because audio is intrusive, it’s best to ask for a user’s permission before using audio notifications—and always provide a clear way of disabling the audio.

Audio, when used judiciously, enhances the user experience through its engagement of another human sense and by providing a richer atmosphere for interaction. We shouldn’t allow the abuses of audio in the past—which have resulted in the banishment of most audio from our daily online interactions—to prevent us from trying again to use audio more effectively. While incorporating audio cues and other sounds in UX projects may be foreign territory for most visually oriented UX professionals, it is territory well worth exploring.

6 Comments

Nice summary on using audio for computer interfaces. I have a couple of comments:

1. Many users, especially in a business setting, disable computer audio or don’t have speakers or headphones. Or, they are talking on the phone as they work—sales, customer service.

2. People who do have speakers or headphones often use them to listen to music as they compute, and they might find audio features annoying.

So it’s important to carefully consider your target audience, as well as making audio features optional. Very good article, however.

Interesting article. Usually noises in interfaces or applications tend to annoy or distract me, but I agree that there are many valid reasons and uses for sound in the UI.

Another area that uses voice is GPS units in cars, giving instructions on which way to turn or how far a destination is. This area will undoubtedly grow as voice activated and interactive systems become more prevalent.

Great overview. Agree completely with you in respect to the opportunities for effective sound design in product usage; sound is generally plopped into a product because the designers can, not necessarily because they should. The general principles of user-centered design apply to sound, too.

I’ve been developing a list of heuristics for the effective use of sound in products, and I’d like to offer two of them up here:

  1. Great sound design cannot compensate for an otherwise poor user experience. (You can’t put lipstick on that pig and hope we won’t notice that it’s still a pig.) If you’re looking to improve a product’s sonic interface, do it in conjunction with the rest of the project’s design and development, not after.
  2. Effective sounds, as useful as they may be, should not live on an island of their own. Just as brand-conscious companies understand the value of a standardized approach to product design or visual design, so they should with sound as well. Utilize sounds and audio palettes across multiple products and experiences to ensure clarity for customers wherever they experience your brand.

Thanks again for the article…

Here is a link to an interesting retirement calculator that uses an audio message with humor. For me, it’s a much more interesting way to perform a retirement calculation. I can’t help but wonder what kind of results they have experienced with this type of interaction.

Go to Fidelity.com, then click MyPlan—the box with the blue background.

Hi Everyone,

Thanks for the terrific comments.

Noel, the first two audio design qualities in your list of heuristics are great. Care to share any others? I think audio design as a part of UI development is still in its infancy. I’d be interested in learning how other designers approach its challenges.

Miguel and Joshua, you’re right that sound has great potential to annoy and the needs of the audience must come first. At the same time, there are many possibilities for audio interaction in application design that may not be immediately apparent.

I experienced a great example of what I’d consider well-designed audio the other day at Stop and Shop—a grocery store chain with many locations in the Eastern United States. The self-checkout aisle at my Stop and Shop features a minimalist touch-screen user interface, coupled with a scanning bed and voice instructions. I found the instructions the computerized voice interface gave me to be helpful—not because I couldn’t figure out how to use the device myself, but because it made the experience easier. In a self-serve environment, the audio reminder to “Please sign for your credit card below the numbered keypad.” was just as useful to me as a clerk pointing to the signature box on the screen.

I’m excited about the possibilities for audio in the user interface.

Thanks, Jon

Nielson / NetRatings has issued a study showing that the top 10 social networking sites saw traffic grow 47% over the last year, with MySpace seeing the biggest growth (367% increase) and MSN Spaces (286%) seeing the nextbiggest growth. Hosted blogging systems were included in the study.

One thing to note about those numbers is that while Classmates had one of the lowest positive growth rates at 10%, they spend loads on advertising while MySpace, Youtube, and Facebook haven’t spent a penny.

If I recall correctly, a couple of years ago, Classmates.com was one of the 10 largest spenders on online advertising.

There are plenty of new social networking sites popping up, but what gets me is why can’t MySpace get their instant messenger working. $580 million and they can’t afford to fix instant messenger. Bad MySpace. There are so many better ones now. How about cubiclelife.net for example. It has all the features of MySpace plus quizzes, polls, Webchat with audio and video, and oh, hey, they have instant messenger. You have a long way to go MySpace.

Join the Discussion

Asterisks (*) indicate required information.