Testing Content Concepts

What Is a Content Concept?

A content concept is a mockup or draft of your content. I find testing at three different levels of fidelity useful, depending on what qualities of the content I want to test. The higher the fidelity, the more variables you can test. However, it can be harder to identify which variable is causing your results. (See “How Do I Test a Content Concept” later in this column.)

Table 1—Content concept fidelities

Fidelity of Content Concept	Most Helpful for Testing These Qualities…
Content only Content independent of the design and, possibly, formatting	tone and style comprehension usefulness some readability some organization some engagement and influence
Content in a low-fidelity design Content in rough wireframes or prototypes, including basic formatting	tone and style comprehension usefulness some readability organization navigation some interaction some engagement and influence
Content in a high-fidelity design Content in polished wireframes or prototypes, including polished page formatting	tone and style—integrated with the overall page design comprehension usefulness readability organization navigation interaction engagement and influence

What Content Should I Test?

You probably can’t test every single piece of content that might be a part of your project, so you’ll need to test a sample. I find the best approaches to sampling content are critical case sampling and common case sampling.

Critical Case Sampling

Select the instances of content that are most important to your project goals. For instance, if your project involves convincing people to sign up for something, you should test the content for the sign-up module or landing page.

Common Case Sampling

Choose the types of content that comprise the majority of the content for your Web site or interactive user experience. For example, if the goal of your Web site is to provide credible healthcare information to improve people’s healthcare decisions, test the content for a typical healthcare topic.

When Should I Test Content Concepts?

Test content concepts at the beginning of your project. Rachel Lovinger, Content Strategy Lead at Razorfish, offers a helpful visual showing the evolution of content strategy throughout the project process in Figure 2. [2] Concept is part of the early discovery phase.

I find testing a concept works best once you have a clear understanding of your project goals and have gained enough insights from other data sources—such as analytics, call metrics, best practices, content analysis, market research, or competitive analysis—to form a reasonable concept.

How Do I Test Content Concepts?

The specific approach you should take to testing concepts might vary, depending on your project, but I find these basic steps always apply. If you have solid experience in testing designs, you simply need to make a few adjustments to focus your testing on content. However, if you have not conducted such testing before, I recommend that you read a book, attend a conference, or take a class about usability or similar types of testing.

1. Identify your testing goals.

The overall purpose of testing a content concept is to figure out what content strategy you should follow throughout your project. Consequently, make sure your specific testing goals align with the project goals. For instance, if the goal of your project is to help people make better health decisions as they travel, make sure your test covers related content and explores whether it influences people’s decisions.

2. Choose critical or common content cases to test.

When choosing the samples of content you’ll test, use the sampling method that makes the most sense for your project and testing goals. (See “What Content Should I Test?”)

3. Create concepts that explore the right qualities.

Whether your team is crafting the content from scratch or adapting it from existing source content, you need to ensure the concepts of the sample content reflect the qualities you want to test. For example, if you want to test the tone and style of the content, ensure the content concept actually has a distinct tone and style.

For comparison, you could test more than one concept, with different tones and styles. I like using comparison to test less tangible qualities, because it elicits more and better-quality feedback from people. [3] Usually, however, testing two or more concepts does not result in a winner, so do not position a comparison test as a contest when communicating with your stakeholders, client, or boss. Rather, testing multiple concepts reveals the best and worst of each concept. Most likely, you’ll try to combine the best—while avoiding the worst—into your final content approach.

4. Plan the right method to test the content qualities.

Usually, you’ll ask people to complete some tasks and participate in an interview. Table 2 offers a breakdown of what methods are best for testing what quality.

Table 2—Methods for testing content qualities

Method	Content Quality to Test	Tips
Task completion Observe and, optionally, ask participants to think aloud	comprehension usefulness readability organization navigation interaction some tone and style some engagement and influence	Ensure task scenarios do not lead participants. For example, do not use exactly the same wording as for the content concept. Focus observations of tasks on how people work with the content. Consider the full content experience—for example, moving from an email or social media message to a Web site.
Interview	tone and style comprehension usefulness engagement and influence more reasons why for other qualities you’ve discovered through tasks	Ensure interview questions do not lead participants. Focus discussion on the content. For engagement and influence, don’t press too hard on why.

Regarding other logistics, you can conduct test sessions in person or remotely. I prefer testing with eight or nine people to guarantee that some patterns emerge. For a good, detailed resource about testing logistics, see Handbook of Usability Testing, by Jeffrey Rubin and Dana Chisnell.

5. Measure the appropriate results.

Evaluating the test results in the right way is important. Table 3 offers some basics. Handbook of Usability Testing offers helpful insights, as well.

Table 3—Basic measurements for testing content

Method	Basic Measurement
Task completion	success finding and interacting with content success reading content success understanding content success remembering key messages or facts
Interview	preference for and opinion about content tone and style opinion about usefulness of content opinion about effectiveness of other content qualities opinion about and rating of the brand, topic, product, or decision—both before and after seeing the content

A Special Challenge: Measuring Influence and Engagement

We need to figure out the right way to assess intangible qualities such as engagement and influence. The difficulty is that, when trying to describe such qualities, people cannot necessarily articulate why. What’s more, if we press people to articulate why something does or does not engage or influence them, they are likely to rationalize—and perhaps even talk themselves into a completely different response from their real reaction. [3]

Some have used measurements from the gaming industry to help assess engagement. My concern about using such measurements is that games are designed to be uniquely immersive, making someone feel lost or completely involved in an experience. Most content-focused experiences strive for engagement, not complete immersion.

Demetrius Madrigal and Bryan McClain’s UXmatters article, “Testing the User Experience: Consumer Emotions and Brand Success,” explores subjective and objective measures for emotion—a close cousin of influence and engagement. The objective measures involve highly specialized techniques such as face reading. In a similar vein, some marketers actually scan people’s brains to assess their emotional reactions and preferences for products or marketing materials. [4] While I find these approaches fascinating, they are currently impractical for most user experience projects.

So, what approach to assessing content’s influence and engagement is practical now? We should capture people’s first reactions to content. Then, through some expertly moderated discussions with participants, we can try to understand their reactions, ensuring we don’t press too hard and, thus, end up with misleading rationalizations. One way of avoiding such rationalizations is to discuss two different concepts in comparison. Finally, we need to understand whether content affects people’s decisions or perspectives. For example, an opening questionnaire could ask participants about what their decision might typically be in a particular case and why. A closing questionnaire could ask them whether they would now make a decision that differs from their typical decision and to rate how well the content informed their decision.

Conclusion

The UX community has amassed extensive knowledge about testing interactive designs. Applying that knowledge to content has exciting potential to help companies and organizations answer the question Does our content work for people? One valuable approach is to test content concepts early in a project lifecycle, enabling you to form a clear content direction.

Both a tough challenge and a big opportunity is testing how content influences or engages people. We need practical, yet reliable approaches to help us understand whether content has impact on people’s perceptions, decisions, and more. I am personally exploring ways of addressing this need, and I look forward to the rest of the UX community doing so, too. When I think about our accomplishments in the realm of testing, I’m confident our collective brainpower can meet the challenge. 

References

[1] Halvorson, Kristina. Content Strategy for the Web. Berkeley, CA: New Riders, 2009.

[2] Lovinger, Rachel. “Content Gone Wild.” MIMA Summit, October 5, 2009. Retrieved from SlideShare on December 7, 2009.

[3] Lehrer, Jonah. How We Decide. Boston: Houghton Mifflin, 2009.

[4] Park, Alice. “The Brain: Marketing to Your Mind.” Time Magazine, January 19, 2007. Retrieved December 10, 2009.