Putting the “Long” into Longitudinal: UX Lessons from Survey Research

Good Questions

Asking and answering users' questions

A column by Caroline Jarrett
August 8, 2011

How long is your typical project? Are you working in 6-week agile sprints? Running monthly usability tests? Trying an A/B test for a week? Updating a Twitter stream hourly? The demands of Internet time keep us focused on shorter and shorter time intervals, with experiences measured in days, minutes, or even the first 50 milliseconds of exposure to a Web page, according to a team of researchers at Carleton University in Toronto led by Gitte Lindgaard. [1]

What happens if you turn that around and think in terms of months, years, or lifetimes? Longitudinal studies look at long-term user experience. Usually, that means over a few months or possibly a few years. But recently, at the European Survey Research Association Conference, I learned about some much longer-term studies that offer some lessons about how to conduct our rather shorter investigations.

Sponsor Advertisement
Continue Reading…

The European Survey Research Association Conference: When Long-Term Means Years

The European Survey Research Association had their conference in Lausanne, Switzerland, from July 18–22, 2011. I wanted to attend the conference as part of my research for my forthcoming book on surveys, because it presented an opportunity to learn about the current concerns of the survey methodologists.

Figure 1—European Survey Research Association Conference 2011
European Survey Research Association Conference 2011

I was also delighted that a current client joined me in a presentation of one of our projects. You can take a look at our presentation, “Usability Testing of Market Research Surveys,” on SlideShare.

The conference included more than 150 sessions, each with three or four papers, in up to 12 parallel tracks, so it was impossible to attend more than a small selection of them. So what to focus on? As I looked back over my notes, I spotted a theme: many of these presentations were about research that happens over years, decades, or lifetimes. To give just two examples from this conference:

  • Todd Hughes from the US Census Bureau talked about the usability tests they conducted in 2010—not to inform the 2010 Census, but looking ahead to 2020. These things take a long time to design.
  • Stephan Huber, a professor at the Institut für Bildungsmanagement und Bildungsökonomie in Switzerland, talked about the latest developments in a study of Swiss teenagers that started in 1854.

In this review of the conference, I’m going to discuss what I learned from three other presentations that gave me ideas for my own longer-term studies.

UX Tip #1: Try Asking Less, But More Often and Earlier

Jennifer S. Barber of the Population Studies Center at the University of Michigan talked about a study of 18- and 19-year-old women and what experiences might lead to pregnancy—especially unintended pregnancy—that is, unwanted or unexpectedly earlier than planned. Unintended pregnancy can be a health risk for mother and baby. The challenge is that, once a woman is pregnant, it’s too late to ask her about the finer details of her life in the months before the pregnancy.

So they set up a panel to track around 1000 women over five years. But what factors to look at? Ideally, the researchers wanted to know pretty much everything about the women’s lives, and one solution might have been to do periodic interviews that covered, well, just about everything.

They started with an hour-long interview with each woman at the start of the project. (I just realized that’s at least 25 person-weeks of interviewing.) The temptation might have been to continue with similar hour-long interviews at rather long intervals throughout the study. Instead, they opted for weekly 5-minute interviews—with a mixture of standard and variable questions each week—and gave participants the option of answering questions on the Web or by phone. This has worked very well. Five minutes isn’t too onerous for anyone; the variety of questions helps to keep the interview interesting; panelists like the regular incentive, even though it’s small—just $1 per interview, plus a small bonus every 5 weeks.

Tip for UX studies—If you want to learn about users’ engagement with your product over a longer-term experience, might you try asking your participants less, but more often?

UX Tip #2: Sometimes It’s Better Not To Ask

The Centre for Longitudinal Studies at the Institute of Education, University of London, carries out multiple surveys of large numbers of individuals from birth and throughout their lives, tracking three cohorts:

  • 17,634 people born in 1958
  • a similar sample of people born in 1970
  • a similar sample of people born in 2000, from the Millennium Cohort Study (MCS)

The MCS study asked participants’ parents about them when they were nine months old, then again at three and five years. When they were seven, their parents did the main questionnaire, and the children got a little questionnaire of their own, asking about their hobbies, their friends, their feelings, and their attitudes toward school.

If you’re designing for children anywhere, but especially in the UK, the overview of the findings from the study at seven years [2] gives robust insights into the lives of children at that age. Should you be willing to tangle with complex sample analysis in SPSS (originally, Statistical Package for the Social Sciences), the actual data is also available. It’s accompanied by a fearsome—to me, anyway—user’s guide about how to manipulate the data to ensure appropriately statistically reliable results.

The researchers are preparing for the survey next year for age 11—rising 12—and reported on a variety of different topics. The paper I caught looked at the best way to ask about alcohol consumption—yes, 11-year-olds are experimenting with alcohol in the UK.

One point that really struck me: deciding not to ask. The researchers have to balance the desire to ask about everything in the children’s lives with the desire to keep them active in the panel. One topic that got excluded: puberty. This is a subject that was too horrible for 11-year-olds to contemplate; even the word puberty was embarrassing to them.

Tip for UX studies—Do you need to ask about everything, right now? Could you, maybe, postpone a difficult topic for another time, or even exclude it altogether?

UX Tip #3: Think About Sharing Your Results

Most of the research groups presenting at the conference publish extensively to academic and government audiences. Many, like the Centre for Longitudinal Studies, make their actual data sets available for further analysis.

The final survey I want to tell you about took this a step further: they published their results about a survey of children in a child-friendly format. Kids’ Life and Times (KLT) [3] is an annual survey of 10- and 11-year-old children in Northern Ireland that began in 2008. The children answer the questions themselves during school time. The researchers have obtained permission to use cartoon characters from Bang on the Door to help with kid appeal.

The research team is very committed to making sure that children’s voices are heard—and they extended that to the children themselves. So, each year, in addition to their more formal publications, they publish a comic-book version of the results, shown in Figure 2. [4]

Figure 2—The children’s version of the results in a comic-book format
The children’s version of the results in a comic-book format

Tip for UX studies—How about publishing more UX survey results? Perhaps not the whole data set, perhaps not even the report. But what about giving participants a headline summary of the key findings from the research, if they want it? This could be the way to help keep our users interested and involved in our ongoing projects.

A Final Reflection

Most of us in user experience are used to thinking short-term—in weeks or months. But, increasingly, organizations are planning to build long-term relationships with their customers. Few of us will need to think in terms of decades or how to keep users engaged over their entire lifetimes, but I believe that we can learn a lot from longitudinal research projects like those I’ve described. 


[1] Lindgaard, Gitte, G.J. Fernandes, C. Dudek, and J. Brown. “Attention Web Designers: You Have 50 Milliseconds to Make a Good First Impression!” Behaviour and Information Technology, Number 25, 2006.

[2] Hansen, Kirstine, Elizabeth Jones, Heather Joshi, and David Budge. “Millennium Cohort Study: Fourth Survey: A User’s Guide to Initial Findings,”PDF 2nd edition. Centre for Longitudinal Studies, December 2010. Retrieved August 7, 2011.

[3] ARK (Access Research Knowledge) Northern Ireland. “Kids’ Life and Times (KLT) Survey,” June 6, 2011. Retrieved August 7, 2011.

[4] ARK (Access Research Knowledge) Northern Ireland. “Kids’ Life and Times Comics,” July 29, 2011. Retrieved August 7, 2011.

Usability Consultant at Effortmark Ltd

Leighton Buzzard, UK

Caroline JarrettCaroline became interested in forms when delivering OCR (Optical Character Recognition) systems to the UK Inland Revenue. The systems didn’t work very well, and it turned out that problems arose because people made mistakes when filling in forms. Since then, she’s developed a fascination with the challenge of making forms easy to fill in—a fascination that shows no signs of wearing off over 15 years later. These days, forms are usually part of information-rich Web sites, so Caroline now spends much of her time helping clients with content strategy on huge Web sites. Caroline is coauthor, with Gerry Gaffney, of Forms that Work: Designing Web Forms for Usability, the companion volume to Ginny Redish’s hugely popular book Letting Go of the Words: Writing Web Content That Works.  Read More

Other Columns by Caroline Jarrett

Other Articles on User Research

New on UXmatters