Top

Visual Data Collection: Making Sense of Tally Sheets

Discovery

Insights from UX research

A column by Michael Morgan
October 21, 2019

For those of you who are wondering exactly what Visual Data Collection (VDC) is, you can read my article “Increasing Your Research Velocity with Visual Data Collection,” which describes the method in full detail. You can also read my subsequent Discovery column, “Sensemaking with Annotations,” which covers the various annotations I use for VDC notetaking. That column also provides insights on the use of annotations to inspire those using the VDC method to extend their set of annotations and adapt the method to their UX research workflow.

In this column, I’ll focus on how to go about collecting useful annotations from the discussion guides from various UX research sessions into one place, using handwritten tally sheets. Once you’ve completed a study, you’ll analyze these tally sheets and report your findings—perhaps by adding your insights to a storage repository for future curation. I’ll provide some examples of tally sheets on which I compiled data I had collected during UX research sessions. (You could alternatively use a spreadsheet to accomplish the same task.)

For the examples in this column, I’ll use a fictional Web site for runners, Middlepacker.com, to make the concepts easier to understand.

The Tally Sheet

Once you’ve completed a multitude of research sessions for a study and your annotations are full of juicy findings, pertinent quotations, and indications of what participants do and don’t understand, how can you efficiently analyze the data you’ve captured during all of the sessions? (You can read more about making sense of what participants do and don’t understand in “Sensemaking with Annotations.”)

Paging through multiple stacks of participant notes—going back and forth, over and over again—to discover themes and patterns is extremely confusing and time consuming. If each finding were a needle, it would immediately get lost in an entire wall of haystacks. It would be impossible to find just what you were looking for!

So before you should even attempt to analyze the data from your research, you must first organize that data. You can use a tally sheet to organize this data. A tally sheet is a copy of your discussion guide that you’ve marked up with the data from all of the research sessions. It is the sum of all the notes from the discussion guides for individual participants.

Before analyzing UX research data, you need to get all the data in one place so you can analyze the data more easily. The tally sheet serves this purpose. Once you’ve gathered all the data in one location, analysis becomes significantly more efficient and much faster.

Figure 1 shows an example of a marked-up section of a tally sheet. (Don’t worry about what the annotations on this document mean. I’ll cover that in the next section.)

Figure 1—Example of a tally sheet
Example of a tally sheet

Adding Participant Annotations and Connectors to a Tally Sheet

In addition to the annotations you’ve used to capture data on the discussion guides for individual participants, there are some annotations that are specifically for tally sheets. Because the purpose of tally sheets is to assist the UX researcher in analyzing data across all participants, there is an annotation for each specific participants, as shown in Figure 2.

Figure 2—Key to participant annotations for a tally sheet
Key to participant annotations for a tally sheet

When referring to a particular participant on a tally sheet, draw a line that connects the annotation for that participant to his or her response or comment in the discussion guide, as shown in Figure 3. For example, a participant annotation might be a response to a closed-ended question for which a participant chooses a response from a finite set of options or a comment about an element in the design.

Figure 3—Participant annotations for various types of responses
Participant annotations for various types of responses

Participant Annotations for Discussion-Guide Questions

Gathering participant annotations onto a tally sheet is most useful for capturing responses to the key design questions a study needs to answer. These might be open-ended questions or closed-ended questions. Figure 3 demonstrates their use in gathering responses to closed-ended questions, with lines connecting participant annotations to each of the responses. As you collect more data in your tally sheets, you’ll continue adding more participants to the responses. As Figure 3 shows, Participants #1 and #2 did not understand the purpose of the site they were viewing, while Participant #3 appears to have understood the site’s purpose.

Participant Annotations on Design Elements

As Figure 3 shows, you can link participant annotations to specific elements within a design. Rather than writing down what a participant is describing, use the visuals within the discussion guide to replace those words. For example, instead of jotting down “Participant #3 is referring to the Find a race button when she says, ‘It looks like a running site because I can find races,’” the researcher simply draws a connector line to the Find a race button on the screenshot in the discussion guide, adding the participant’s quotation next to it. This approach makes notetaking substantially easier and interpreting your annotations becomes very obvious later on. The researcher won’t have to wonder, “What are they referring to in the design?!”

Participant Annotations to Capture Emerging Trends

The surprising insights that come from participant or customer feedback are what make UX research so interesting and valuable to product teams. Perhaps, during the data-collection process, you might discover that you had overlooked asking a very important question in the early sessions that then arose in a later research session.

The tally sheet is an ideal place to gather such new, after-the-fact questions, ensuring that you’ll capture everything you need to support your research goals. Even though you were unable to gather specific data from the earlier participants on such emerging trends, you can go back to your transcripts or recordings later on and revisit evidence that can help you to answer these new questions. At a minimum, you’ll have captured the emerging trend somewhere rather than just keeping it your head, risking that you might forget it later on.

For example, perhaps a number of participants—presumably people who are interested in running—commented on how important receiving advice from experts has been in their lives as runners, but your discussion guide didn’t facilitate your capturing such insights. You can still capture this feedback in a structured, albeit ad-hoc fashion by noting the emerging trend you’ve identified and using participant annotations with it, as shown in Figure 4.

Figure 4—Participant annotation for an ad-hoc, closed-ended question
Participant annotation for an ad-hoc, closed-ended question

In the lower-left corner of Figure 4, the researcher captured the finding that participants desired expert advice from experienced runners in the form of a question, with two possible responses—yes and no. Lines connect specific participant annotations to each of these responses, depending on whether participants expressed interest in reading expert advice on Middlepacker.com during a research session.

Because there was no plan in place for the researcher to ask participants whether they were interested in expert advice from other runners, the researcher would likely have to go back and mine the data from the earlier sessions to check for the presence of this finding.

Managing a Cluttered Tally Sheet

As you might have noticed in Figure 1, tally sheets tend to become quite cluttered. Of course, if you’re running a study with only a handful of participants, you’re less likely to run into space problems when adding more data. However, as your sample size increases to perhaps to ten or more participants, the tally sheet can become overcrowded as you add more data, making the job of analyzing the data later on much more difficult. Therefore, working with the tally sheet becomes unwieldy.

Using Connectors More Efficiently

One way to manage the clutter is to use connecting lines more efficiently. For example, when logging the same finding for multiple participants, rather than repeating that finding multiple times, which would take up a lot of space, connect multiple participant annotations to just one instance of the finding. If you need to elaborate further, just add a branch to a related finding, as shown in Figure 5.

Figure 5—Connect multiple participant annotations to a finding
Connect multiple participant annotations to a finding

As Figure 5 shows, participants found the foot ad on the site unappealing. Connecting lines demonstrate specific participants’ relationship to this finding. Plus, the researcher added quotations from a couple of participants that add dimensions to the finding—beyond the foot ad’s just being tacky.

Switching to Landscape Orientation

Another way to create more space for annotations is to lay out your discussion guide in a landscape orientation. Also, if you end up with a larger sample size, consider adding data to the back of a tally sheet, where you should have plenty of space. If you decide to reserve the back of each tally sheet for additional data, make sure your printing preferences are set to print on one side rather than two sides of each sheet. If all else fails, you can put your data into a spreadsheet.

Analyzing Tally Sheets

Once you’ve collected all of the data from your annotated discussion guides onto tally sheets, you can start analyzing the data for useful findings and insights.

All of your up-front work on completing the tally sheets minimizes the effort it takes to analyze the data. There is usually little reason to go back and revisit an individual participant’s discussion guide to locate a finding once you’ve consolidated everything onto the tally sheets—unless you’ve inadvertently omitted logging some data.

In a future edition of Discovery, I’ll describe how to analyze the data you’ve collected onto your tally sheets, so stay tuned! 

Senior UX Researcher at Bloomberg L.P.

New York, New York, USA

Michael MorganMichael has worked in the field of IT (Information Technology) for more than 20 years—as an engineer, business analyst, and, for the last ten years, as a UX researcher. He has written on UX topics such as research methodology, UX strategy, and innovation for industry publications that include UXmatters, UX Mastery, Boxes and Arrows, UX Planet, and UX Collective. In Discovery, his quarterly column on UXmatters, Michael writes about the insights that derive from formative UX-research studies. He has a B.A. in Creative Writing from Binghamton University, an M.B.A. in Finance and Strategy from NYU Stern, and an M.S. in Human-Computer Interaction from Iowa State University.  Read More

Other Columns by Michael Morgan

Other Articles on User Research

New on UXmatters