Top

Learning from Closed Card Sorts with Different Inputs

January 5, 2015

The card sort is one of the most common research techniques for developing information architectures. The concept is pretty straightforward: you ask participants to place various items into different categories and, by doing this, get a sense of what pieces of content should go where. As with many research techniques, however, the reality is more complicated. In fact, entire books have been written about the subject.

At PledgeMusic, we have an expansive, form-based system that musicians and our staff use when creating and managing content, merchandise, and communications with fans. One of our upcoming projects is to redesign this system to make it easier and faster to perform all of the tasks that it supports. As part of our discovery process, a colleague and I recently had the opportunity to run a series of workshops at several company offices in different countries. Our task was to gather information about this system from our coworkers who use it daily in their work to get a sense of what parts do and do not work for them.

Supporter Advertisement
Continue Reading…

Because our workshop participants belonged to the teams in our company that use the system on a daily basis, they were both our subject-matter experts and our target audience. I ran a card sort with each group as part of the workshop. In between sessions, I used what we had learned so far to change and refocus the exercise for the next group. What I found was that every card sort was useful, but in different ways.

This article isn’t about whether there are right or wrong ways to do a card sort. It’s about how different ways of conducting card sorts inevitably lead to different results. Our workshops illustrated many ways in which different approaches can yield very informative results. In this article, I want to share our findings to give the UX community a real-world example of the thought processes behind setting up similar card-sort exercises and what to expect from the results. I’ll discuss the pros and cons of each approach. Hopefully, this article will help other UX researchers to choose the right approach at the beginning of a project, depending on the kind of information they want to gain from a card sort.

Three Different Inputs

We conducted three card-sort exercises, which differed mainly in the information that I provided to the team at the start. (Going forward, I’ll use the term items to refer to the pieces of content that participants sorted and the term categories to refer to the groups into which they sorted the items.) The inputs to these exercises were as follows:

  • categories, but no items—This exercise had the most open input. I provided large sheets of paper that I’d labeled with predefined categories, but just a blank set of cards on which participants could write individual items of content, then categorize them. Participants were free to add or remove categories if they wished.
  • categories and items—I provided both the predefined categories and a set of cards, each containing an item of content from either the current system or my previous research. We asked participants to categorize the items, but blank cards were available if they wanted to add new content as well.
  • categories and presorted items—We prepopulated the sheets of paper for predefined categories with cards representing content items, according to where the content resides in the current system. We asked participants to think about what does or does not work in the current system, then rearrange the cards accordingly. As for the other sessions, we also encouraged participants to add or remove items or categories as necessary.

One thing that we did not do was ask participants to create new categories themselves. There were various reasons that I didn’t feel this was appropriate for the project, because of our goals for the sessions. There are obviously cases when an open sort is the best tool for the job, and I encourage researchers to read some of the existing writings on that subject before starting their research. Next, I’ll provide some examples of the useful information that you can obtain through these various approaches to closed card sorts.

Card Sort 1: Categories, but Not Items

For this workshop, we provided categories, but no items to place in them. I’d based the categories primarily on the existing system, but with a few modifications resulting from some other research that my colleague and I had already done. We left it to participants to come up with items that they would expect to find in each category, as shown in Figure 1.

Figure 1—Categories, but not items
Categories, but not items

This exercise took the longest. You generally need to allow more time for participants to come up with ideas from scratch than you do for them to comment on or work with something that you give to them. Plus, it’s important to be able to give a few examples from your own experience, in case you need something to start the ideas flowing. Once the brainstorming opens up and participants hear what other people are saying, they’ll usually understand what you’re asking them to do, and this exercise will go faster.

As we expected, although we had told the participants that they could modify the categories that we had given them, they didn’t make many changes. They created only two new categories—basically, because there was some debate about where several particular items should go. Still, this provided useful information because it not only gave us another potential section for the site architecture, but also revealed a discrepancy in our thinking about how people viewed certain items.

Benefits

The most important result of doing this type of card sort is learning what items participants come up with. Without our feeding participants pieces of content or functionality, the items they think of and write down end up being those that are most important to them. In this way, a researcher can get a sense of priorities that is absent from the results of a card sort in which all items are present with equal weight from the start. Plus, participants inevitably come up with items that you would never have thought of.

During this session, several different participants added pieces of functionality that do not currently exist in the system. Since we have not used the system extensively ourselves, we never would have known how important this missing functionality is. Thus, this session provided insights into key content and the prioritization of functionality, as well as making us aware of the need for some new items.

Drawbacks

The drawbacks of this approach—beyond those resulting from doing a closed instead of an open card sort—are twofold. First, while the aforementioned time requirement may not be an issue, depending on your situation and testing setup, if your time is limited or you have a large group of participants, the open-ended nature of this exercise may prove unwieldy.

The other drawback is that, lacking significant preparation, participants are likely to miss a lot. If you’re not dealing with an existing system, as we were, or one that your participants are very familiar with, they’ll almost certainly forget about many items that are nevertheless important. There’s usually just too much for anyone to remember everything. So you’ll have to expect that there will be items that participants don’t categorize at all and figure out how to balance the relative importance of those items with those that participants mentioned during the session.

Card Sort 2:  Categories and Items

This is probably the type of card sort that most people who haven’t run a card sort before would naturally think of. For this card sort, we provided a set of categories that were consistent with the existing system, as well as a large pool of existing items of content and functionality, as shown in Figure 2. We asked participants to sort the items and encouraged them to add or remove either items or categories as necessary.

Figure 2—Categories and items
Categories and items

When running a card sort in this way, the general concern is that participants who are familiar with an existing system will tend to put items into the buckets they’re already used to. Since we expected that this might happen, we intentionally added some items that didn’t already exist, and we were also sometimes slightly vague in describing existing items on the cards.

We were hoping that, at the very least, participants would sort the new elements that we had added—and perhaps reclassify a few of the existing items that they thought were awkwardly placed. What we actually found was that the card sort produced a very different information architecture from that of the current system.

In part, this occurred because—as we already knew—some things in the existing system were confusing and participants were able to identify them. But more importantly, this happened because of the way my colleague and I worded the items that we provided. For instance, an item that we had labeled Payments, which we intended to indicate the place where a user enters how they would like to get paid, participants alternatively thought meant one of the following:

  • the place where they see when they will get paid
  • a list of payments they have already received
  • a list of other people who have purchased things on their site

Clearly, our use of the term payments—which is prevalent throughout the existing system—was an inherent source of confusion. We might not have learned this had we not asked participants to define the term for us.

Benefits

As the Payments example illustrates, the chief benefit of this kind of closed card sort was the clarification of our existing content and terminology. This type of learning is most important when you are redesigning an existing system—as we were. Even when you’re creating a site or application from scratch, you may be able to discover some of the organizational predispositions that your business owners have.

The varied interpretations of Payments also told us something about prioritization We had assumed that, to some degree, people would naturally base their classifications of items on what they thought was most important—or at least, what came first to mind. This, in turn, taught us a little about how different people in the company saw the business requirements for the system, because there were correlations between how participants defined items and the roles and positions of the people who were defining them.

I was actually pleasantly surprised by the variety in the participants’ categorization of the items. It was further from the current system than we had expected, plus participants actually added several categories that hadn’t previously existed to enable them to adequately bucket their items.

Drawbacks

The major drawback of providing this set of inputs was, as you would expect, the inherent limitation in the potential knowledge that we might gain because we gave participants a set of items to categorize at the outset, and they tended not to add many more items of their own creation. So this is not the best method to use if you’re trying to find out what might be missing from an existing information architecture.

Despite participants’ creating some new categories, they split the only really new items off from existing items. In the case of Payments, for example, we realized that we would need several different items to cover all of the things that term had previously comprehended.

You should expect to do a lot of explaining during a card sort like this, because participants are likely to ask what you mean by certain terms. Whether you should answer them depends, as always, on what you’re looking for. If you’re interested strictly in categorization, you might want to answer them and describe any confusing items. However, I have found that I often gain more interesting knowledge by not providing answers or by directing the questions to other participants, asking them what they think, to see how they interpret items.

Card Sort 3: Categories and Presorted Items

Our strictest set of starting inputs once again included both predefined categories and items, but also grouped the items into categories using the current system’s information architecture. We asked participants to think about their experience with the current system and identify painpoints, then to rearrange the classification of items in a way that they thought would reduce its complexity and remedy known problems. For instance, if an existing workflow was inconvenient because two items of information that a user needed to complete a task were on separate pages, maybe those two things should instead be grouped together.

Figure 3—Categories and presorted items
Categories and presorted items

As we expected, this workshop took a little bit of time to get going. Regardless of the clarity or confusion of an existing system, once an audience becomes familiar with it, it’s difficult for them to imagine anything else. As with the other methods of card sorting that we’ve described, as a moderator, you must sometimes give an example or two from your own experience to start the ball rolling. Once participants realize that this is their chance to remedy some known painpoints and complexities, they’ll start thinking about their day-to-day routine and will be better able to propose ideas.

Nevertheless, we found that this setup was a wonderful conversation starter. In fact, this approach led to the most in-depth discussions and analyses of the existing system. When we presented participants with what were essentially a set of visual aids that mapped out the system that they used every day, they thought about its familiar patterns more than is typical for more open card sorts. This seemed to make talking about those patterns and recalling the issues that they have with them easier.

We came out of this session with a host of insights that we had gained largely from very detailed conversations that got sparked by participants’ commenting on a single item’s placement. For example, a participant’s moving just one card from one category to another started an informative, ten-minute long conversation about why he had moved it. Overall, participants may have done less sorting of the cards than with open card sorts, but we learned something important just the same.

Benefits

The most prominent benefit of going into an exercise having already provided this many inputs to participants is familiarity. We found that, although it took a while for them to start making changes, it was very easy to get them to think about the current system and start discussing it.

There is also the added benefit of specificity. Card sort 1, as I described earlier, had the downside of missing a lot of the smaller items that participants wouldn’t necessarily think of. For card sort 2, that was less of a drawback, but participants still tended to focus mostly on more major elements. Since card sort 3 kind of represented the warts-and-all current system, it led to the most granular and specific discussions and feedback. If one of your business requirements is the preservation of many pieces of functionality in a complicated system, exercises like this one can give you insights into some things that are less obvious, but still important, which may cause problems if you do not address them in your redesign.

Drawbacks

As expected, the primary drawback with such a closed card sort was an exaggeration of the same problem that we saw with predefined categories and items in cart sort 2. When you proscribe the inputs by presenting participants with cards that conform to the existing system, they tend to make few changes. So your new information architecture may end up looking very similar to the existing one—even when there are actually many changes that you should make.

At the end of our session, the buckets of items were very similar to those that we had started with, because we had not forced participants to think up a new categorization scheme from scratch. Furthermore, our placement of items within existing categories eliminated most of the terminology confusion that had appeared during card sort 2, when we had only predefined categories and items. So participants did not create new categories as a result of any reinterpretation of their meanings. As a consequence, I do not recommend this method of card sorting when your goal is to restructure an information architecture—unless you use it in combination with other more open card sorts.

The other major drawback of this approach is the time that the discussions take. This was one of the longest workshop that we ran. To gain significant knowledge from this method, you need to let the sorting process jump start discussions among participants, and those discussions can take a long time. Of course, those discussions can be well worth the time, and you can gain valuable information from them, but you need to give them time to breathe. Of all the methods that we used, this is the one that might benefit most from doing multiple sessions. It might even be better not to approach this discussion through a card sort at all and, instead, get participants to talk about just a few sections or categories at a time. Of course, you should allow participants to point out items that seem to be misplaced.

Guidelines for Selecting a Closed Card-Sort Method

Ultimately, each workshop that we ran provided useful information. The question, then, is how should a prospective workshop facilitator decide which inputs or combinations of inputs to use? The answer depends on what you want to get out of a session.

Based on the outcomes of our workshops, here are some guidelines:

  • Provide categories, but not items. Use this method if your primary goal is to find out what is missing from an existing system or you need to understand participants’ priorities.
  • Provide both categories and items. Use this method if you primarily need to learn about the clarity of an existing system’s terminology and areas of confusion—particularly as different groups of stakeholders perceive them.
  • Provide items that are already categorized. Use this method if your goal is not to completely change an information architecture, but to get detailed feedback about specific categories, items, or parts of the system.

Since I’ve based these suggestions on the findings from this particular workshop series, they are primarily applicable to redesigns of existing systems or projects within established organizations. Open card sorts are more useful when you’re designing a system from scratch or doing an overall redesign. My purpose in writing this article was simply to describe a few ways in which a more closed card sort can be very helpful, depending on where you are in a project and what you’re trying to learn. 

Director of User Experience at PledgeMusic

New York, New York, USA

Kevin SheldonKevin started working in the digital world of writing and content strategy as an Editor and Developer for the City of New York, then transitioned to user experience and interaction design. He spent years as a UX Lead at Funny Garbage, Havas Worldwide, and Iris Worldwide, before leaving the agency world to join PledgeMusic’s product team. He has degrees from Cornell and New York University. His goal is to find projects that embrace evolving technologies and steer them toward making people’s lives better.  Read More

Other Articles on Card Sorts

New on UXmatters