Testing Your Own Designs

Practical Usability

Moving toward a more usable world

A column by Jim Ross
November 7, 2016

There’s an old saying in the field of UX design that you shouldn’t conduct usability testing on your own designs. While this sounds like a good ideal to strive for, is it really practical? In reality, UX designers test their own designs all the time. Often, it’s because they’re the only UX professional available to test the designs. If they don’t do it, it won’t get tested at all. Is it really a big mistake to test your own designs, or is some testing better than no testing at all?

Many have written articles about this topic—including a great column by Paul Sherman on UXmatters from 2009. But, as someone who has experienced all sides of this issue, I think I have a unique perspective to add to this discussion. Through testing my own designs, having other people test my designs, and testing other people’s designs, I have experienced the advantages and disadvantages of each of these situations. In this column, I’ll discuss whether it’s possible to test your own designs effectively and provide tips for UX designers who are either testing their own designs or having other UX professionals test their designs.

Champion Advertisement
Continue Reading…

Why You Shouldn’t Test Your Own Designs

First, let’s look at the main reasons people argue that it’s best not to test your own designs.

You Can’t Be Objective About Your Own Designs

The main argument against testing your own designs is that you’re too invested in your own design solutions to be objective. Even if you try to be impartial and are aware of potential bias, it’s very difficult to avoid having biases, which can affect your body language, the questions you ask, and the questions you don’t ask participants. Bias can creep into your analysis and interpretation of the findings. Especially when the findings don’t point to obvious problems and solutions, you may consciously or unconsciously interpret the findings to confirm your beliefs and dismiss findings that don’t support your design decisions.

You Are Too Close to the Design

As the UX designer, you know the design better than anyone else. You are aware of all the requirements, decisions, technical limitations, and trade-offs that went into the design. As a result, you see the design very differently from the average participant encountering it for the first time. Your insider knowledge can make it more difficult for you to see the user’s perspective.

You May Be Under Pressure Not to Find Too Many Problems

The healthiest attitude about usability testing is that it’s an iterative, learning activity that helps inform design. It’s natural and expected that you’ll find usability problems—and those problems should not reflect poorly on the designer.

Unfortunately, not every organization shares this healthy attitude about testing. When an organization sees usability testing as an evaluation of a design and its designer, anything more than minor problems may reflect poorly on the designer. In such an unhealthy atmosphere, designers who are testing their own designs have a strong incentive not to find many problems. This is especially true when the first usability study occurs later in the design process, because finding major problems at a late stage can result in major project delays.

Other People May Perceive a Conflict of Interest

Even if an organization has a healthy attitude about usability testing and the designer makes a great effort to be impartial, other people may still perceive a conflict of interest. They can use this concern as a reason to dispute any of your findings and recommendations with which they don’t agree.

You May Be Too Busy to Do Both Design and Testing

While it may seem that hiring generalists who can both design and test their own designs will save the organization money, that’s a lot of work for one person. Often, a designer is too busy designing and creating prototypes to take the time to plan the testing, recruit participants, run the test sessions, and analyze the data. Dividing the design and testing work between two employees enables them to perform these activities simultaneously. This is more efficient, lets you complete usability studies more quickly, and shortens the overall length of the development cycle.

Participants May Feel Uncomfortable Providing Honest Feedback

Participants are often uncomfortable providing critical or negative feedback directly to the person who created a design. Instead, they may try to be polite and soft-pedal their criticism. Whenever I test something I didn’t design, I make a big point of saying, “I didn’t design this myself. I’ve been asked to get people’s feedback. So you won’t offend me or hurt my feelings if you criticize it.” Participants usually laugh at that point and feel freer to provide their honest opinions. But, when I am the designer, testing my own design, I can’t honestly say that.

Why You Should Test Your Own Designs

Despite these powerful reasons for not testing your own designs, there are also some advantages to testing your own designs. Now, let’s consider those advantages.

You Know the Design Better Than Anyone Else

As the designer, no one knows your design better than you. If you’ve done stakeholder and user research, you’re the person who is most intimately familiar with the business and user requirements. You know all about the design decisions, questions, objections, and trade-offs that got you to this point in the design process. So you’re the most logical person to plan the usability testing so you can ensure that it focuses on answering the most important questions.

You Know the Prototype Better Than Anyone Else

As the person who created the prototype, you know which links, buttons, and other interactive elements are active and which are not. You know where the dead ends are, and if something unusual were to happen during a test session, you’d know best how to get the participant back on track. If a prototype element is inactive, you can describe to the participant how it would normally work. If you need to fix a problem with the prototype, you may be able to do that quickly between sessions.

You Can Learn Firsthand from the Testing

The UX designer is the person for whom it’s most important to understand the test findings. So what better way to ensure you get that understanding than to facilitate and observe the testing firsthand? Yes, the designer could simply observe the testing. However, facilitating the testing requires more active attention to and interaction with participants than just passively observing. Plus, it lets you ask pertinent questions in the moment. With a deeper understanding of the findings, you’ll be able to make better decisions about which elements of the design to change.

You May Be the Most Qualified Person to Do the Testing

Some people are equally skilled in research and design and enjoy doing both. When you’ve already conducted user research and created the design, it can be difficult to give over the usability testing to someone else. You care the most about the design, so you may be the best informed and most motivated person to test it.

Testing Your Own Design Is Better Than Not Doing Testing at All

Even if you think it’s ideal to have a different person test your design, that’s not always feasible. When a company doesn’t have usability-testing specialists, a common alternative is to have another designer conduct the testing—a designer who is not working on the project. But for busy design teams, even that is not always possible. So, when there’s no one else available to do the testing, it’s certainly preferable for UX designers to test their own designs than it would be to skip usability testing altogether.

Usability Testing Tips

No matter whether you think it’s better for UX designers not to test their own designs or you think it’s okay for designers to test their own designs, here are some ways to make the best of each of these situations.

When You’re Testing Your Own Designs

When you have to test your own designs, here are some tips to avoid the problems I described earlier:

  • Remind your client and the project team that usability testing is a learning process. The goal is to find problems and answer questions about the design. Encourage this healthy attitude about testing to avoid the misconception that testing is an evaluation of the designer’s skills.
  • Conduct at least two rounds of usability testing, starting early in the design phase. This will enable you to catch and fix problems early in the development cycle, when design changes are the least time consuming and expensive to make. The problems you discover won’t seem so catastrophic when you have time to fix them, then test again.
  • Consider conducting comparative testing of multiple versions of a design solution. This can refocus testing from the evaluation of a single design—and your design skills—to a comparison of which aspects of each version work and which don’t. Comparative testing changes the tone of testing to more of a learning process.
  • When you begin the usability-testing phase of the project, try to compartmentalize by putting yourself in a usability-testing mindset. You’re no longer playing the role of the designer. Instead, think of yourself as an impartial, usability professional and try to get some distance from the design.
  • Try to put yourself into a neutral mode by simply asking questions and noting the information participants provide. Avoid being defensive or explaining why you made certain decisions. Think of yourself as an actor taking on a new role as a usability tester, instead of your usual designer role.
  • Try to create a discussion guide with non-biased tasks and questions. Since it’s easy to include biased questions inadvertently, have someone else with usability-testing experience review your discussion guide, so they can spot any problems of potential bias or any leading questions.
  • Tell the participants that you’re testing an early design with the goal of finding and fixing problems. Reassure them that you want their honest feedback—both positive and negative—and they won’t hurt your feelings. Of course, you should never be dishonest and state or imply that a solution you’ve designed isn’t your design, but you do not need to volunteer the information that you’re the designer.
  • If you’re unsure whether you can be unbiased, stick closely to the discussion guide, consistently asking only the questions that are in the guide.
  • Focus on taking objective notes on what you observe and hear. Trying to draw conclusions or insights during the test sessions is where your bias as the designer can lead you astray.
  • When you report your findings, freely admit the problems and failures of the design. The ability to discuss the problems and successes of your own design objectively enhances your credibility with others. View the discovery of usability problems as a valuable learning experience that can only improve the design.

When You’re Not Testing Your Design

If someone else is going to be testing your design, the following are some tips on making the most out of the situation:

  • Involve the person who will conduct the usability testing from the beginning of the project, to ensure he or she fully understands the business requirements, the users, and how the design has evolved. Try to avoid bringing in someone to do usability testing who knows nothing about the project.
  • Work closely with the user researcher during any research activities—throughout the design process and during usability testing. Even though the researcher is primarily responsible for user research and usability testing and you’re primarily responsible for design, make sure that these aren’t siloed activities.
  • When the usability specialist is planning the testing, provide a list of important tasks to test, questions that you have about the design, and questions or concerns clients or other project team members have raised about the design.
  • Observe every test session and take notes to help you stay focused.
  • Collaborate with the usability specialist in discussing the findings and coming up with potential solutions to the problems the testing has identified.
  • When your team has no dedicated user-research specialists and, instead, consists only of UX generalists, designers can test each other’s designs. However, you’ll still need to fill in those other designers on the project details, observe the testing, and work closely with them in interpreting the testing results.

The Ideal Situation

In my opinion, the ideal situation is to have both a user researcher and a designer working closely together on a project, collaborating on the user research, the design, and the usability testing. Of course, that’s not always possible. So, if you must test your own designs, you can minimize any potential problems by following the advice I’ve detailed in this column. 

Principal UX Researcher at AnswerLab

Philadelphia, Pennsylvania, USA

Jim RossJim has spent most of the 21st Century researching and designing intuitive and satisfying user experiences. As a UX consultant, he has worked on Web sites, mobile apps, intranets, Web applications, software, and business applications for financial, pharmaceutical, medical, entertainment, retail, technology, and government clients. He has a Masters of Science degree in Human-Computer Interaction from DePaul University.  Read More

Other Columns by Jim Ross

Other Articles on Usability Testing

New on UXmatters