Review of Information Architecture Evaluation Tools: Chalkmark and Treejack
Published: February 22, 2010
Recently, Optimal Workshop, the creators of the online, card-sorting tool Optimal Sort, released two new information architecture evaluation tools: Chalkmark and Treejack. Though these tools are not perfect, they do provide the ability to quickly and easily test early designs and information hierarchies with large numbers of participants.
As a UX consultancy, Optimal Workshop has been very open to receiving user feedback and has incorporated that feedback in updates to their software. I offer this review in the spirit of providing constructive feedback that I hope will benefit the products. I’m sure these tools will continue to improve over time.
Testing Findability with Chalkmark
An online, unmoderated testing tool, Chalkmark lets you test findability in a Web application design. Chalkmark gives participants a task such as Find special offers on cruises and presents a screenshot, as shown in Figure 1. Participants click links in the screenshot where they think they would find the information they need. Then, Chalkmark presents the next task and the next screen. The test results are heatmaps, showing where participants clicked during each task. The heatmaps show concentrations of clicks and how many participants clicked each area of a screen.
Figure 1—A task in a Chalkmark study
Chalkmark is ideal for testing wireframes and early design concepts. Because each test task involves clicking a static screenshot, the only information Chalkmark captures is the participants’ clicks on a single screen. While you can use closed card-sorting tools and tree-testing tools like Treejack to test an information hierarchy in isolation, Chalkmark lets you evaluate an information architecture within the context of a page design.
Setting Up a Chalkmark Study
It is fairly easy to set up a study with Chalkmark’s simple user interface. Once you’ve named the study and set up a welcome message, instructions, and a concluding message, you can add tasks and select images to present with the tasks, as shown in Figure 2. A minor weakness of the setup user interface is that it shows only the file names for the images accompanying the tasks. Thumbnails would be helpful in identifying the images you’ve chosen for each task.
Figure 2—The Chalkmark study setup screen
A problem throughout both Chalkmark and Treejack is the lack of online Help. While that is less of a problem with the relatively simple and intuitive setup user interface, because this is a new tool, information about how to create a study and get the most out of this technique and other tips would be helpful.
Testing with the Participant User Interface
Once you’ve created a study, Chalkmark provides a link for you to send to participants. When participants click the link, the welcome screen shown in Figure 3 appears, displaying a brief explanation of the study’s purpose. You can either ask participants to provide an email address or another type of identification or make the study anonymous. You can optionally require a password on this screen, too.
Figure 3—The Chalkmark welcome screen
Since there is no moderator, clear instructions are extremely important to ensure participants know what to do. Unfortunately, the instructions in both Chalkmark and Treejack are text only, and they offer little capability for formatting the text, as you can see in Figure 4. Participants may skim over this dull-looking text, assuming they can just figure it out as they go along. Example images or illustrations would be more helpful in explaining a test concept and give participants a better idea of what to expect next. Unfortunately, once participants leave the instructions page, there is no way to view the instructions again or any other Help.
Figure 4—Chalkmark instructions
Fortunately, the task screens in Chalkmark are extremely simple, with a task description appearing above a screenshot and a link to skip the task. The header clearly shows a participant’s current location in a study, with the current task number and total number of tasks.
After reading the instructions for a task, participants click links in a screenshot they think would let them accomplish that task. Each click displays a message page, thanking the participant and indicating the next task is loading. I have found that some participants are surprised when their first click immediately ends the task and starts another one. They expect their clicks to take them to another page where they can continue the task. Some have said this fails to give them any sense of whether they’ve gotten the task right or wrong.
Unfortunately, unlike most online card-sorting and usability-testing tools, Chalkmark provides no method of getting additional feedback from participants, either by letting them make comments or through survey questions. The first click on each static screenshot is all that Chalkmark collects, which makes it a very simple, focused tool, but also a very limited tool.
Viewing the Test Results for a Chalkmark Study
The test results for each Chalkmark task appear on click heatmaps like those shown in Figures 5 and 6. Hotspots show where participants clicked. Statistics boxes for areas that received multiple clicks show details about the number and the percentage of clicks. These visualizations clearly show whether participants clicked the correct areas or, if not, where else they expected to find things.
Figure 5—A Chalkmark heatmap, showing clicks as hotspots
Figure 6—A closer view of the hotspots and statistics boxes
The heatmaps suffer from a few technical problems. For example, sometimes the statistics boxes appear directly over the hotspots or on top of one another, making it difficult to see the hotspots, what links participants clicked, and any statistic boxes they overlap, as you can see in Figure 7. It would be helpful if there were a way to drag these boxes out of the way.
Figure 7—Overlapping statistics boxes
There are controls that let you change how Chalkmark clusters clicks in heatmaps, as well as the size of clusters, but the meanings of those settings and how Chalkmark clusters clicks are not clear. Some explanation would be helpful, but again, Chalkmark provides no instructions or Help explaining how to interpret the results.
As shown in Figure 8, the header for the heatmap shows the total number of clicks, the number of participants who skipped the task, and the average time participants took to complete the task. As with any unmoderated test, a single participant’s being interrupted during a task could easily throw off the average task time. Therefore, to provide more accurate task times, it would be helpful to see a list of all participants and their individual task times and be able to eliminate outliers.
Figure 8—Average time to complete a task
The heatmaps are obviously something you would want to use in a report or presentation, yet Chalkmark provides no specific function for downloading the test results or saving the heatmaps. You have to take your own screenshots to save them.
My Overall Assessment of Chalkmark
Because of Chalkmark’s simplicity, it is ideally suited for straightforward findability tests. Within the context of a page design, you can determine whether a page’s information hierarchy and labeling let participants find things.
Compared to other online, unmoderated testing tools, Chalkmark is much easier to use—both in terms of setting up a study and analyzing the results. It is far cheaper and more flexible in its pricing plans, too.
However, the downside of this simplicity is that Chalkmark is very limited in what it provides. Seeing only participants’ first clicks in a heatmap fails to reveal the other places in a screenshot participants considered clicking. Plus, Chalkmark does not show whether participants would find what they were looking for once they’ve clicked a link on the first page. You cannot use Chalkmark for test tasks requiring participants to click several things on a page—for example, clicking or hovering to open a navigation menu before clicking a link. The fact that Chalkmark cannot gather any other type of user feedback such as comments or survey responses is very limiting.
Despite these limitations, Chalkmark has its place as a simple, quick, and inexpensive evaluation tool.
Testing Information Architectures with Treejack
Treejack is a tool for testing information architectures. Participants’ tasks involve finding something in a clickable tree structure, as shown in Figure 9.
Figure 9—A task in a Treejack study
Each branch of a tree structure expands with a click, showing the subcategories within that branch. To choose the final category in which participants would expect to find an item, they click an I’d find it here button, shown in Figure 10. The test results show which categories participants chose during each task, the percentage of correct choices, the paths taken, and other related data.
Figure 10—Completing a Treejack task
Treejack is ideal for testing a proposed information hierarchy. In many ways, it is superior to other information architecture evaluation techniques such as closed card sorting and reverse card sorting. Like reverse card sorting, Treejack asks participants to complete finding tasks. Thus, Treejack is not like closed card sorting, which asks participants to organize content into categories. But unlike either closed or reverse card sorting, which limit evaluation to the first level of categories, Treejack evaluates the entire depth of an information hierarchy.
Setting Up a Treejack Study
Similar to Chalkmark, the first steps in creating a new study in Treejack are specifying the participant login information, as shown in Figure 11, and creating the welcome message, instructions, and concluding message, which appears once participants complete a study.
Figure 11—Initial setup of a Treejack study
Setting up your tree structure in Treejack can be somewhat complicated. To facilitate the process, create a spreadsheet containing the data first, then copy and paste it into Treejack, as shown in Figure 12. It is worth noting that indentation must be correct for Treejack to render a tree correctly. Helpfully, a Preview tree button lets you make sure Treejack created it correctly.
Figure 12—Setting up a tree hierarchy
The next step is creating the tasks—for example, Find acne treatment cream. For each task, you select the correct answer—that is, the subcategory or subcategories in which an item resides, as shown in Figure 13. However, when setting up tasks, the need to select the correct answers and the way to do this are not obvious at first, so it is easy to overlook this until you unsuccessfully try to launch a study.
Figure 13—Setting up tasks in Treejack
There are several helpful settings you can select, such as randomizing tasks, revealing only a certain number of tasks to each participant—so you can have a large list of tasks, but each participant does only a reasonable subset—and ignoring results from participants who skip more than a certain number of tasks.
Testing with the Participant User Interface
Treejack suffers from some of the same problems as Chalkmark—a limited instructions page that contains only text, with no images or visual examples, and no way to view the instructions once the tasks begin. Fortunately, the task pages are fairly intuitive.
Unlike Chalkmark, which displays a task and a screenshot at the same time, Treejack shows a task first, then waits for a participant to click a Start this task button, shown in Figure 14, before showing a tree structure like that shown in Figure 9. This helps ensure participants are ready, lets participants take a break between tasks, and probably results in more accurate task times.
Figure 14—Starting a Treejack task
Clicking through a tree structure is fairly intuitive, because it looks like a navigation menu and expands to reveal subcategories. Once a participant clicks a subcategory, a Top link appears at the top of the tree, as shown in Figure 15, allowing participants to go back to the top of the hierarchy. I have found that some participants do not notice this link or do not understand its purpose. As a result, they did not realize they could go back up in the hierarchy. Both the label Top and the concept of the top of a hierarchy may be unfamiliar to most people. While a Back link might better match the familiar concept of Web navigation, it would go back up a hierarchy only one level at a time.
Figure 15—The Top link lets participants go back to the top of a hierarchy
Participants complete each task by clicking an item in the tree, then clicking an I’d find it here button, as shown in Figure 16. Participants have said they prefer this interaction over Chalkmark’s, because it lets them choose their final answer in two steps, giving them a greater sense of control. The I’d find it here button gives them a chance to go back and choose something else, while in Chalkmark, their first click unexpectedly takes them directly to the next task. Thus, Treejack provided a more satisfying experience.
Figure 16—Selecting a subcategory during a Treejack task
Working with the Test Results for a Treejack Study
Treejack provides test results both in an online summary, shown in Figure 17, and as a downloadable Microsoft Excel file. The online results page provides very little information describing what it is showing and, as throughout the rest of the application, there is no link to online Help that would explain how to analyze the results.
Figure 17—Online summary of Treejack results
From the online summary page alone, one might conclude that Treejack provides little value. Each task and the study overall receives a score. Bar charts show percentages for success, speed, and directness. While the meaning of success is obvious, it is unclear how Treejack can represent speed as a percentage, and the meaning of directness is unclear. Plus, there are pie charts with no labels at all. Again, some additional information about how to interpret the results would be helpful, but Treejack does not provide it.
Fortunately, there is also a link that lets you download an Excel file, which provides far more comprehensive and useful information. In the Excel file, the first tab, shown in Figure 18, shows the overall success rate and participants’ choices for each task. It also shows the average success rate across all tasks, the direct success average—showing the frequency with which participants chose the correct category right away—the indirect success average—showing the frequency with which participants eventually made the correct choice, after backtracking through other categories—the failure average, and the average percentage of those who skipped tasks.
Figure 18—First tab in the downloaded results for a Treejack study
The second tab in the Excel file, shown in Figure 19, shows participants’ first clicks for each task, including the percentage of first clicks for each top-level category and the percentage of participants who clicked each top-level category during the tasks.
Figure 19—Second tab in the downloaded results for a Treejack study
The third tab, shown in Figure 20, shows the path each participant took through the hierarchy during each task. Paths are color coded, with green for direct success, yellow for indirect success, orange for failure, and gray for skipped tasks, making it easy to find and focus on specific results such as failures.
Figure 20—Treejack study results, showing paths participants took during a task
My Overall Assessment of Treejack
Treejack is a great tool for testing a proposed information hierarchy before incorporating it into a page design. The ability for participants to drill down through multiple levels in a hierarchy makes Treejack far more useful than either closed card sorting or reverse card sorting. Using Treejack more closely matches people’s actual behavior on Web sites, because it asks participants to find things within an information hierarchy rather than asking them to organize content into categories, as with closed card sorting.
Trying and Buying Chalkmark and Treejack
Both Chalkmark and Treejack let you create a free account with a limit of three test tasks, allowing you to run test studies with friends and family and evaluate these tools for yourself. It is especially important to view the results and determine whether they will give you the type of information you need to make design decisions.
The pricing of these tools is quite reasonable when compared to other online research and evaluation tools. Both Chalkmark and Treejack are available individually, at the following prices:
|30-day plan, with unlimited tasks||$109|
|365-day plan, with unlimited tasks||$559|
An annual plan that includes Treejack, Chalkmark, and Optimal Sort—an online, card-sorting tool—is available at a reduced price of $1,350.
Like most usability professionals, those of us at my company, Electronic Ink, are usually on the lookout for new ways of evaluating information architectures and other user experiences quickly and inexpensively. Chalkmark and Treejack fit the bill, with easy-to-use and inexpensive tools that let you gather extra user feedback and evaluate information architectures during your design process.