Generative IA Research Methods
Information architecture user research seeks to understand how people think about information to determine the best ways of organizing and labeling content. This research can be either
- generative—gathering user input on the organization and labeling of content—or
- evaluative—determining whether people can correctly find things in an organizational structure we’ve created
Using both methods in an iterative approach to user research is often the best way to ensure an intuitive information architecture.
Initial User Research
User research is an important first step in any design project. For an information architecture project, it is important to first understand a site’s content and the users’ vocabulary before conducting generative research such as card sorting. This understanding helps you to choose what items to include and the terminology to use on the cards.
Open Card Sorting
The original method of getting user input on information architecture was open card sorting. In this method, researchers give participants a set of index cards, each containing a piece of content and, optionally, a description. Participants sort these pieces of content into categories that make sense to them, then name those categories. It’s called an open card sort because participants create their own categories, as shown in Figure 1—unlike a closed card sort, in which participants sort cards into predefined categories. There are various ways of analyzing the data from an open card sort—from informally eyeballing the data to putting it into software that produces a cluster analysis diagram showing the common groupings, similar to that shown in Figure 2.
Individual Versus Group Card Sorting
Traditionally, we have done card sorting with individual participants to learn how they think about a site’s content, but you can also do card sorting as a group exercise. Having several people sort the cards together allows you to hear their discussions and the reasoning around how they group the cards. The disadvantage is that the results represent a consensus rather than reflecting each person’s thoughts about the content. Having each person do a card sort individually, then getting all participants together as a group to discuss everyone’s thought process in grouping and labeling the cards might result in a better group activity.
Benefits of Open Card Sorting
Open card sorting provides insight into how people think about content, including the following:
- their mental models for the content
- intuitive content groupings
- alternative methods of grouping or accessing content
- category and subcategory labels
- relationships between items, which could be useful in determining related links and recommendations
Use card-sorting results only as a guide to organization and labeling. You should never take the results and use them as your actual site structure. It’s important to make sure others on a project team don’t take the results too literally.
Limitations of Open Card Sorting
Because card sorting involves in-person sessions, with physical cards, there are a few limitations:
- It’s time consuming to run the sessions, record the groupings, and combine the data across participants.
- Researchers and participants need to be together for the sessions, which either limits participants’ geographical representativeness or requires a lot of travel.
- Open card sorting results in a lot of data with no easy tools or methods for analyzing the data.
- Sometimes participants sort items based on superficial aspects rather than thinking about the meaning of the items.?
Online Open Card Sorting
The limitations of in-person card sorting led to the creation of card-sorting software such as IBM’s Usort and EZcalc, then to the creation of online card-sorting tools such as WebSort, shown in Figure 3, and OptimalSort.
Online card-sorting tools provide the following advantages:
- These online tools remove location limitations. Participants from anywhere in the world can participate without incurring the costs or taking the time that travel involves.
- An almost unlimited number of people can participate. This is important because studies have shown that it is necessary to include many participants—at least 30 to 50—to get results that you can generalize to an entire user population. 
- Unmoderated sessions let participants complete a study in their own time, free up researchers’ time, and let researchers complete a study in much less time.
- These tools automatically capture, analyze, and present data through various useful visualizations. This eliminates a lot of the manual data recording and number crunching traditional card sorting requires.
However, online card sorting has some disadvantages:
- Because the physical space on a computer screen is much smaller than a large table, participants can see fewer cards at a time, which makes it more difficult for them to sort and keep track of the categories.
- Since no researcher is present, participants have no one to answer their questions or ensure they are doing the card sort correctly.
- Researchers can’t observe and discuss grouping and labeling decisions with participants.
- Some online card-sorting tools do not allow participants the flexibility to rename cards, leave cards unsorted, add new cards, put a card in more than one location, or create subgroups.
Combining Traditional and Online Card Sorting
It’s possible to get the best of both worlds by combining the advantages of online and in-person card sorting. For example, you can use online card sorting to get data from a large number of participants and hold a smaller number of in-person sessions to get qualitative feedback about participants’ grouping decisions.
Modified-Delphi Card Sorting
As an alternative to open card sorting, information architect Celeste Lynn Paul created an alternative method called Modified-Delphi card sorting.  Instead of having participants do their own card sorts in turn, then analyzing the combined data, when following the Modified-Delphi method, participants work one after another, refining a single model.
How the Modified-Delphi Method Works
When using the Modified-Delphi method of card sorting, the first participant does a traditional open card sort. Then, throughout the remaining sessions, each participant starts with the organization the previous participant created. A participant can choose either to modify that organization, as shown in Figure 4, or to scrap it and start over, creating a completely different organization. A study continues until the organizational structure stabilizes and participants are no longer making any significant changes. The result is an organization participants have reached by consensus.
Benefits of the Modified-Delphi Method
In creating the Modified-Delphi method of card sorting, Paul’s intent was to create a time-saving and more accurate method to replace open card sorting. It offers the following benefits:
- It requires fewer participants.
- Participants find it easier to think aloud because of the reduced cognitive load of refining an existing organization rather than creating an organization from scratch.
- There is less data to analyze because participants work on a single organizational model. The final organizational structure is ready for review immediately after the final participant finishes sorting the cards.
- According to Paul, a Modified-Delphi card sort results in a better organizational structure than open card sorting does. Having participants refine a single model provides less random results with fewer outliers.
Limitations of the Modified-Delphi Method
The Modified-Delphi method seems to have the following risks:
- Since each participant is free to start the organization over from scratch, and there are a small number of participants, an outlier participant could compromise the study.
- Each participant is influenced by the structure that previous participants have created. This may cause them to overlook other ways in which they might normally have thought about organizing the content and can lead to a groupthink effect.
- Although it’s possible to modify some online card-sorting tools to do a Modified-Delphi card sort, there are no online tools specifically for this purpose.
Evaluative IA Research Methods
Once you’ve created a taxonomy, you can evaluate it with users to determine whether they can find things where you’ve put them. Three methods of IA evaluation are closed card sorting, tree testing, and usability testing with low-fidelity prototypes.
Closed Card Sorting
Closed card sorting is similar to open card sorting except, instead of asking participants to create their own categories, you ask them to organize the cards into the categories you’ve created. This shows you whether participants group the items in the same way you’ve grouped them. It’s an easy way to see where your structure doesn’t match user expectations.
Traditionally, participants do closed card sorts using physical index cards, but you can also conduct closed card sorts with online card-sorting tools like WebSort, shown in Figure 5. Since this is an evaluative activity that is primarily concerned with whether people choose a particular category for a card, it’s well suited to online research with many participants.
Benefits of Closed Card Sorting
Closed card sorts provide the following benefits:
- They provide an easy and quick way of evaluating whether your organization and labeling make sense to users.
- The results clearly show which items do not make sense in your organization and where they should go instead.
- The results are easier to interpret than the results of an open card sort.
Limitations of Closed Card Sorting
Closed card sorts have the following limitations:
- Online tools allow participants to sort items only into top-level categories. You can’t see in what subcategories they would place items.
- Closed card sorting is based on the premise that people would look for items on a Web site in the categories where they placed cards when organizing them during a card sort, but some have questioned whether a categorizing activity like card sorting equates to the finding activity that is characteristic of people’s use of Web sites.