Extending Card-Sorting Techniques to Inform the Design of Web Site Hierarchies

By Michael Hawley

Published: October 6, 2008

“When developing hierarchies for information-rich Web sites, designers and usability researchers often turn to card sorting for help in making design decisions.”

When developing hierarchies for information-rich Web sites, designers and usability researchers often turn to card sorting for help in making design decisions. Card sorting offers a systematic and statistically significant process for answering questions about hierarchy design. However, those of us who have run card sorts know there is an art to conducting successful card sort studies, and there are many variables that can affect the usefulness of results. In this column, I’ll discuss the challenges and limitations of card sorting and review alternative and complementary techniques that designers can leverage when developing an information hierarchy for a large-scale Web site.

Challenges of Card Sorting

“In-person card sorts let researchers interact with participants and ask probing questions to determine their organizational strategies, as well as other follow-up questions.”

Part of the appeal of card sorting today is that researchers have the option of conducting studies either online or in person. When using an online tool, large numbers of participants can complete the exercise, lending additional statistical weight to the findings. In some organizations, the large sample size and statistical basis of online card sorting is helpful in dealing with decision makers. In-person card sorts let researchers interact with participants and ask probing questions to determine their organizational strategies, as well as other follow-up questions. A number of resources are available online that provide detailed steps on running and analyzing studies in both contexts.

Sounds easy, right? Just load your content into a card sort application or create actual cards, have participants complete the exercise, and the right organization for your site will be revealed! Unfortunately, it is not usually quite that easy. As Sam Ng wrote in an article on UXmatters, there are a number of issues to consider when conducting a study, from the timing of a card sort to setting proper expectations with stakeholders. Two of the points Sam mentioned in his article ring true and concur with my experience:

  • Run multiple studies. Designers can expect participants to sort a maximum of 80–100 cards during any one study. For sites with a large volume of content—such as intranets, retail stores, or research portals—such a limited number of cards may not accurately reflect the full breadth of content on the site. For such a site, you’ll need multiple studies to understand each individual level or section of the site. Running multiple studies adds significantly to the time and effort completing the process requires, and you should account for this when planning.
  • Count on doing some subjective analysis. Although you can apply statistical methods to results, you’ll still need a certain level of subjective analysis to interpret the intent of card sort participants, as well as the consequences of content relationships. As a result of this necessary subjectivity, designers may have different interpretations of the same results, depending on their depth of domain knowledge and previous experience with the content. User researchers have proposed several analysis templates, involving spreadsheets and visualizations in place of dendrograms, but their analysis still involves a level of subjectivity.

In addition to the points Sam Ng raised, I’ve found some other challenges to conducting successful card sort studies.

  • selecting and naming cards—Creating the cards for an online sort is very difficult in some domains. For physical objects, the process is easier. Choosing objects that have universally recognized names—and, potentially, using images—helps ensure respondents will understand the cards. But on an informational Web site, complex pieces of content are more difficult to describe, so you must make significant effort to ensure respondents will interpret the names of cards consistently, without introducing bias to the groupings participants will create.
  • labeling groups—One of the most important elements of a site hierarchy is the labeling for each category or menu item. While open card sorting can provide insights to users’ view of content relationships, designers should not necessarily expect open card sorting to provide useful solutions for the names or labels of the resulting content groups. During a card-sorting exercise, participants give names to the groups they create, but each participant may create groups of varying sizes, with different intents. As a result, looking for trends among group names from different participants is difficult and may not yield helpful insights. While closed card sorting lets you test group labels, you can test only one set of labels at a time, and it does not allow any interplay between the labels for multiple levels of the hierarchy.

Further Considerations for Card Sorting

“My experience with card sorting has led me to question a few primary assumptions about the method.”

While there are certainly challenges to conducting effective card sort studies, we can overcome those challenges. For example, user researchers can run multiple studies, pilot test their studies to uncover biases in card names, and collaborate with others to gain different perspectives on the subjective analysis of results. However, my experience with card sorting has led me to question a few primary assumptions about the method. To my knowledge, there isn’t any specific research that validates these assumptions, but it may be important to be aware of them when conducting a study.

Sorting Versus Finding

When participants complete an open card-sorting exercise, they sort groups of related items. It is not clear whether the results would be inherently different if the research method were based on a finding task.

Designing Outside the Design

“Studying content relationships outside the context of the site design may not accurately reflect users’ perception of the site hierarchy as they use it.”

Researchers intentionally conduct card sorts outside the context of a particular site design. The idea is to get a pure view into participants’ understanding of the relationships between content items. However, for a Web site hierarchy, a number of factors impact users’ ability to find content, including the paths they followed to get to the site and contextual design clues that orient them to the site and related content. Studying content relationships outside the context of the site design may not accurately reflect users’ perception of the site hierarchy as they use it.

Pointing to a Single Answer

At the outset of a site reorganization process, project teams are generally excited about card sorting’s potential for settling questions about the labeling of menu items or the placement of particular content on the site. They hope that card sorting can provide the answers they need. Sam Ng mentioned this phenomenon in his article on UXmatters. The problem is that there is not always one answer. Certain users might look for a piece of content in one category, while others would look for it in a different category. Or perhaps users might look for content differently, depending on the context of their visit to the site. By focusing on the aggregate results that show the average content relationships from a card sort study, teams may miss the variety of perspectives the site should accommodate.

Focusing Only on a Categorical Hierarchy Solution

Additionally, card sorting may not give insights into alternative navigational paradigms. For example, perhaps the best navigation system for the site is listing content alphabetically, by task, or by audience type, or is metaphor driven. Focusing on the results of a categorical card sort can blind a design team to these other possibilities.

Extending and Complementing Traditional Card Sorting

“Traditional open and closed card sorts work best for categorical organizations when both the context and the users are consistent.”

In my experience, traditional open and closed card sorts work best for categorical organizations when both the context and the users are consistent. If there is potential for an organizational scheme that does not rely on categories or user types and the contexts of visits may differ among users, there are other research activities and variations on traditional card sorts that designers and user researchers can employ to help them better organize their content. Many of these methods are more qualitative than quantitative, but they can provide unique insights that lead to creative solutions for the problems we encounter in designing hierarchies.

Free Listing

One challenge of card sorting is determining what cards to include in the sort. The best content to evaluate is the content that would be most valuable to users. Free listing is a research method you can use to identify high-value content or elements that are representative of a particular category. To conduct a free listing exercise:

  • Ask a number of participants to: List all of the [content] you can think of.
  • Optionally, have participants build on each other’s responses.
  • Evaluate the frequency of use for each content element.

You can use free listing as a precursor to an open card sort, to inform the selection of high-value content that you should include in the study. Alternatively, you can use free listing on its own to reveal clues to the primary names and content for categories.

Findability Testing (a.k.a. Tree Testing)

“Findability testing helps focus your team’s efforts on the weaknesses of an initial information hierarchy.”

Instead of doing user research to inform the original creation of an information hierarchy, designers can use their own expertise, domain knowledge, and ideas to create an initial draft of a site hierarchy, then apply testing methods to refine and iterate initial ideas. To conduct a findability test:

  • Develop an initial information hierarchy, or use an existing site.
  • Document the hierarchy in a paper site map.
  • In a research setting, ask participants to find information in the paper site map.
  • Finish up each test session with a one-on-one discussion, regarding organizational approaches.
  • Analyze the number of times participants looked for particular pieces of information by selecting different menu items.

Findability testing helps focus your team’s efforts on the weaknesses of an initial information hierarchy—areas that need improvement. Most importantly, in findability testing, users attempt to find information using your proposed labels, following a process of elimination of options that closely resembles an actual finding task on a site. In addition, representing an information hierarchy using a number of levels simulates the environment users would find on a site.

Focus Group Card Sorting

“A focus group card sort is a method that leverages the best of both online and in-person techniques.”

As I mentioned earlier, the appeal of online card sorting is that large numbers of users can participate in a study. However, with an online study, designers may miss out on the insights and comments users can provide in person. Interpreting users’ intentions from the data they submit online can be challenging. A focus group card sort is a method that leverages the best of both online and in-person techniques. To conduct a focus group card sort:

Invite 8–15 participants who are representative of the target audience to a conference room, with a computer for each person—either a computer you’ve provided or a participant’s notebook computer.

  • First, ask participants to complete the card sort individually.
  • Then, lead a discussion with participants, regarding the organizational strategies they used during the card sort.
  • Optionally, project the results of the study for review and discussion.

A focus group card sort gives you both quantitative data from the card sort, plus insights into the participants’ rationales for different organizational strategies from the discussion. Additionally, hearing the participants discuss the pros and cons of different approaches can uncover perspectives the designers have not considered.

Naming Exercises

While an open card sort can reveal users’ perspectives on content relationships, it is not the ideal method for helping you make decisions on category names. Once you’ve created some initial content groups, based on an open card sort, conduct a naming exercise, as follows:

  • Develop an online survey with the same question for each proposed menu group: What would you name a group containing the following items…?
  • Ask participants to complete the survey online.
  • Perform a qualitative analysis of the names—either manually or using text analysis software (SPS).

Conducting a naming exercise between an open card sort and hierarchy validation testing lets designers solicit feedback on the labels for menu items they’re considering for actual groups of items on the site.

Delphi Card Sorting

“Delphi card sorting is similar to open card sorting, except you allow participants to build on the results from other test sessions.”

Delphi card sorting is similar to open card sorting, except you allow participants to build on the results from other test sessions. Instead of asking each participant to start from scratch, participants can iteratively improve a proposed hierarchy. To conduct a Delphi card sort:

  • Have a subject-matter expert (SME) create an initial information hierarchy.
  • Present the initial information hierarchy to the first participant, and ask the participant to make changes to it.
  • Follow this process iteratively with other participants, each of whom contributes to the evolving hierarchy, until participants are no longer making major changes to the organizational hierarchy.

The idea of Delphi card sorting is to gain insights into designing a hierarchy with fewer participants and potentially less effort. Because participants do not start from scratch, there is a lower level of effort per participant and less chance that fatigue will contribute to poor data.

Conclusion

“Alternative methods…can extend and complement card sorting and thus provide the most comprehensive insights for designing an effective information hierarchy.”

Card sorting is a useful technique for discovering user perspectives on site navigation. However, designers or user researchers who conduct card-sorting exercises should be aware of the method’s challenges and assumptions. In this column, I have presented a number of alternative methods that can extend and complement card sorting and thus provide the most comprehensive insights for designing an effective information hierarchy.

Resources

Maurer, Donna. “Card-Based Classification Evaluation.” Boxes and Arrows, April 7, 2003. Retrieved September 22, 2008.

Ng, Sam. “Card Sorting: Mistakes Made and Lessons Learned.” UXmatters, September 10, 2007. Retrieved September 22, 2008.

Sinha, Rashmi. “Beyond Cardsorting: Free-Listing Methods to Explore User Categorizations.” Boxes and Arrows, February 24, 2003. Retrieved September 22, 2008.

4 Comments

Thanks Michael. I love the idea of the free listing exercise before an open sort, and the naming exercise after. It’s not too much extra work and would give you more confidence in the results.

Thank you for this succinct and useful article!!!

We are preparing for a closed online card sort, using Optimal Sort as our tool. What are some opinions of using a category heading labeled Stand-Alone Items, which would be for topics that don’t fit into any particular category?

Hi Vicky,

With regard to a “stand-alone items” category for a closed sort, I am not aware of any specific research on the advantages or disadvantages of including such a category. I’ve been of the opinion that closed card sorting should be used more for validation than for understanding participants’ mental models of content organization. As such, I don’t recommend using an unknown category in a card sort. You wouldn’t have such a category on your site or in an application, so why test it that way. An unknown category may give you insight into which items are hard for participants to classify, but it doesn’t give you any insight into where those hard to group items could fit. Better to define categories and see where the hard-to-classify items gravitate. In addition, if you did some sort of tree testing or findability testing, you might get a better sense of the overall, high-level category those items would fit into. Thanks for your comment!

Join the Discussion

Asterisks (*) indicate required information.