Doing User Research at an eCommerce Startup

By Janet M. Six

Published: November 25, 2013

Send your questions to Ask UXmatters and get answers from some of the top professionals in UX.

In this edition of Ask UXmatters, our experts discuss how best to conduct user research at an ecommerce startup.

Each month in Ask UXmatters, our panel of UX experts answers our readers’ questions, which may be about a broad range of user experience matters. To get answers to your own questions about UX strategy, design, user research, or any other topic of interest to UX professionals in an upcoming edition of Ask UXmatters, please send your questions to: ask.uxmatters@uxmatters.com.

The following experts have contributed answers to this edition of Ask UXmatters:

  • Drew Davidson—Senior Experience Director at ÄKTA
  • Steven Hoober—Mobile Interaction Designer and Owner at 4ourth Mobile; author of Designing Mobile Interfaces; UXmatters columnist
  • Caroline Jarrett—Owner and Director at Effortmark Limited; UXmatters columnist
  • David Kozatch—Principal at DIG
  • Whitney Quesenbery—Principal Consultant at Whitney Interactive Design; Past-President, Usability Professionals’ Association (UPA); Fellow, Society for Technical Communications (STC); UXmatters columnist
  • Jim Ross—Senior UX Architect at Infragistics; UXmatters columnist
  • Simon White—Responsable Expérience Utilisateur at Voyages-sncf.com

Q: My team and I work for an ecommerce startup and want to conduct some user research. We don’t actually know our users and are planning to conduct user research as input to developing personas. Is this the right thing to do? What should we do first? What technique should we use: questionnaires, direct interviews, or something else?—Tri Nugraha

“Do as much research as possible that lets you interact directly with some of your potential users. Interviews are good. Visiting users in their own context is even better.”
—Whitney Quesenbery

“First, kudos for realizing that you need to find ways to get to know your users,” responds Whitney. “That’s your critical goal. Everything else is just a way to get there. I urge you to do as much research as possible that lets you interact directly with some of your potential users. Interviews are good. Visiting users in their own context is even better. Tagging along with users while they show you how they perform their activities, then talking to them about the way they do them is even better. Remember not to define users’ tasks too narrowly! You want to observe the whole activity, even if it happens across different times, places, or devices or with different people.

“Personas are a great way to pull together everything that you learn into a form that not only has a human face, but that you can easily share with others at your company. You can then use your personas during your design work to make sure that you are staying true to what you’ve learned about your users.”

Getting Off to a Good Start

“You shouldn’t begin your research until you’ve asked yourselves the right questions: What are you trying to determine from the research? What exactly are you looking to find out about your users? Who is the audience…?”—Drew Davidson

“You shouldn’t begin your research until you’ve asked yourselves the right questions,” answers Drew. “What are you trying to determine from the research? What exactly are you looking to find out about your users? Who is the audience for this startup, and what are you trying to find out about them? Answering these questions should help you to reframe what kind of questions to ask users and what kind of information you need to get from them to make the research worthwhile.

“Truthfully, creating personas would probably not be useful to you at your current stage. Before focusing on detailed personas, you first need to understand the more basic needs and expectations of your users in general. That information will provide the foundation for everything that comes after, including more detailed personas that you can create to help you convey the needs of more specific types of users.

“Regarding techniques, 1-on-1 interviews are probably your best bet. But you should also look at known consumer behaviors within your targeted market segment. While interviews are great, oftentimes there is a wealth of existing information on user behaviors within large market segments like ecommerce.”

Doing User Research on a Shoestring Budget

“There is no wrong way of doing [user research] except not doing it all.”—David Kozatch

“Ah, user research,” muses David. “There is no wrong way of doing it except not doing it all. If you already have users and at least basic profiles of your users, you might explore some secondary sources of information—that is, anyplace online where users comment on your product or similar products.

Many researchers would choose to go to a survey site like SurveyMonkey next, but here’s a low cost way to connect directly with your users: Post a message on a social-media site, inviting people in your target audience to take part in a survey and offering to enter them in a raffle for some reward—like an Amazon gift card—if they provide their email address and complete your survey. Create a brief questionnaire using Google Form, then pretest it with employees to find any bugs. Send an email message to your list of users, including a link to your survey and inviting them to fill it out. Google will put all of their responses into a spreadsheet for you. Analyze the results, then create a report of your findings. Rinse and repeat.”

Asking Users Questions

“Questionnaires and direct interviews aren’t different methods; they are aspects of the same method: asking users questions.”—Caroline Jarrett

“Learning about your users is definitely an excellent idea, and asking them some questions about themselves is one of the techniques that can help you to do that,” recommends Caroline. “But questionnaires and direct interviews aren’t different methods; they are aspects of the same method: asking users questions.

“If you’re doing a direct interview, you need to have

  • a sense of the topics that you want to explore with your users
  • some openness to the topics that they might want to explore with you
  • sensitivity about whether they want to talk to you at all

“It could be that, if people don’t even want to talk to you, maybe they don’t want to engage with your organization in any way—and certainly not to spend money on your products or services.

“One of the benefits of using a human interviewer is that a person can be much more flexible when it comes to choosing topics that users want to explore with you.”—Caroline Jarrett

“But let’s go back to the topics that you want to explore. One good way of getting a handle on those topics is to write some questions to use as prompts during your interviews. You now have a questionnaire. Maybe it’s just one that you’re going to use loosely, as a rough interview guide—initially, with just a few users—but it’s still a questionnaire. If those first-draft questions hang together well enough for you to have a conversation with your users and start to elicit answers that can guide your decision making, you might want to refine your draft questionnaire further, creating something that you can use as a scripted interview with a bigger sample of users.

“Then, if that’s working well, you might start to think about making it into a questionnaire that you can send to even more users—possibly by email or as a Web-based survey—so they can answer your questionnaire at their convenience rather than as part of an interviewer-led conversation,” continues Caroline. “Survey methodologists call this a self-administered interview.

“On the other hand, one of the benefits of using a human interviewer is that a person can be much more flexible when it comes to choosing topics that users want to explore with you. Without that human intervention, you’ve got to drastically reduce the range of the questions that you can ask and have much less flexibility in following users’ train of thought.

“Finally: no survey methodologist would ever create a new survey without doing lots of interviews first. Do interviews with respondents—the users you want to learn about—but also with the people who will use the answers that the survey gathers. So get interviewing and enjoy the learning experience.”

Contextual Inquiries: Combining Observation and Interviewing

“The best user research involves observing people performing their actual tasks in their natural environment. … Conduct contextual inquiries—which combine observation and interviewing.”—Jim Ross

“Of course it’s the right thing to do—especially since you don’t know your users,” advises Jim. “Conducting user research and developing personas are definitely the right steps to take in understanding your users and improving your site. However, I don’t recommend questionnaires or direct interviews. Both of these methods involve asking people questions and listening to what they say without actually observing their behavior or context. The best user research involves observing people performing their actual tasks in their natural environment. So go to participants’ homes or workplaces and conduct contextual inquiries—which combine observation and interviewing. You can start with traditional interview questions about their lifestyle, interests, shopping habits, and similar ecommerce sites on which they shop. Then you could ask them to show you how they would shop on your site—if it’s already live—or on other similar sites. To make the experience more realistic, you could give them money to spend or reimburse them for their purchases.

“If your site is already live, you should do research with your existing customers—and people who might be likely to shop at your site in the future. You can easily reach your existing customers through sales data, and you could use a recruiting company to help you reach your likely customers.”

Prototyping and Usability Testing

“Test either a prototype or your actual product with potential users and get their feedback.”—Simon White

“My first reaction to your question about doing user research in a startup environment would be to test either a prototype or your actual product with potential users and get their feedback,” suggests Simon. “Give them a scenario with a task to complete and watch them as they work through it—getting them to comment on what they are thinking as they do it. In this way, you’ll get valuable feedback on obstacles and misunderstandings that they encounter. A laboratory setting helps—whether permanent or faked—with a camera filming the user and using software such as Moray. If necessary, you can rent cheap, temporary office space by the day to do your research anonymously.

“Personas are great, but when you’re in startup mode, they’re unlikely to be as valuable as just doing direct testing with potential users and getting qualitative feedback. Beware of drilling too much into details or asking leading questions. Just do task-completion exercises. As a bonus, at each step, ask what level of confidence a participant has—before clicking or tapping—that he or she can predict what the next screen or element that appears will be.”

“If you have more specific questions about the user experience of your site and its features, you should conduct usability testing.”—Jim Ross

“If you have more specific questions about the user experience of your site and its features, you should conduct usability testing,” advises Jim. “You could either do that in a usability lab or go to your customers’ location—possibly their home or workplace. Give participants specific tasks to perform on your site to see how well they can do them and what their experience is like.

“The choice between doing contextual inquiries or usability testing is not an either/or proposition. You could combine both methods in one visit—both observing and asking questions. You might conduct part of a session as a contextual inquiry, then also have a usability-testing component when you ask participants to try to perform some specific tasks on the site. The benefit is that you can gather some information about your users, their characteristics, interests, behavior, and environment; and you can also use a few structured tasks to answer specific user-interface questions that you have.”

Steven has offered suggestions for doing quick-and-dirty usability testing in his blog post “A Common-Sense Crash Course for Improving Your User Experience and his DevLearn|13 presentation “Mobile Design: Adding Mobile to Your Learning Ecosystem.” (Start at page 62.)

Using Analytics

“Every method of doing user research has it pros and cons. … What participants say is often not what they will actually do.”—Steven Hoober

“Every method of doing user research has it pros and cons,” replies Steven. “For your team, one key thing to remember is that many research techniques require some experience or training. UX professionals, in general, say that focus groups are a bad form of research, because it’s so very hard to run a good one. Likewise, it’s hard to use the data from many forms of research properly. What participants say is often not what they will actually do.

“When you get the money to do some research, aside from hiring some good contract researchers, my suggestion for startups’ doing usability research is two-fold: First, analytics are very, very useful. You can get unequivocal answers to how people click through your product—whether a Web site or application, for mobile or desktop. Spend the time and money to install a robust analytics tool, so you can get full tracking and have the right tools to visualize what’s happening.

“Then, to find out why your users behave as they do, watch them use the product. Always observe real users, not just developers’ friends—preferably in their natural environment. Visit them in their home or office, have them use your product, and ask them to talk aloud about what they expect or why they clicked an item.

“And tie analytics and user research together. To make sure that you get all the information that you need, use the analytics to find areas that are worse than you expected, then do ethnographic research and usability testing to find out why and how you can improve your design.”

Join the Discussion

Asterisks (*) indicate required information.