After heading a small business that created and sold educational products for ten years and seeing technology evolve and customer expectations shift, I realized that we needed more user research and product iterations to meet the scale of the challenge. Through my having conversations with researchers working across the domains of software as a service (SaaS), education, and medicine, as well as private-sector companies, plus countless hours of research, I have gained valuable knowledge that I’d like to pass on to you and other researchers who care about User Experience.
UX researchers influence how teams create and iterate on products. We rely on user research to identify and understand user problems and assess the usability and effectiveness of the solutions we create. But the user-research methods that we select in a given scenario depend on the phase of the product-development lifecycle and the questions we need to answer.
In this article, I’ll discuss diary studies so you can understand what they are, what questions they aim to answer, how to conduct them, what their limitations and benefits are, and how to earn buy-in for a diary study from key stakeholders. You can do diary studies for new products or existing ones. Once you analyze the resulting data, your findings enable teams to add context to customer-journey maps, which are critical tools for understanding and improving the end-to-end user experience.
With new and consistent evidence that user-centered design is a critical component of the successful development of software products, it is now easier for UX researchers to defend the costs of such user-research efforts. By sharing what can go right and wrong when conducting diary studies, I hope to help steer you away from some common pitfalls and toward getting the funding you need to achieve successful research outcomes. I’ll also shine a light on the importance of the research–participant recruitment process to ensure the validity of the outcomes of your research studies.
What Are Diary Studies?
A diary study is a longitudinal user-research method that collects self-reported data from study participants. Diary studies convey what users do, think, and feel when they actually interact with your product. Their data can be both qualitative and quantitative, depending on the specific questions for which you’re seeking answers. However, because researchers usually choose this research method to improve their understanding of the evolution of participants’ sentiments and motivations, most researchers generally view a diary study as a qualitative research method.
Diary-study participants can be either the current users of a product or research participants who match the target demographic and professional traits that you’ve determined from your earlier research. Study participants complete a predetermined set of key activities over an extended period of time, which can last for days, weeks, or longer. Diary entries comprise specific activities and answer specific questions.
A Helpful Data-Collection Tip
One best practice is to collect the diary data electronically, so you can review, monitor, and analyze the data as it comes in. This enables you to ask follow-up questions while the original responses are fresh in a participant’s mind.
Imagine that a participant sends a voice memo, stating her frustration with a new feature in your app. You can follow up shortly thereafter to understand her perception of what caused the inconvenience. What screen was she on? Was she still able to complete the task? If you waited until the end of the study to review the diary entries, participants would be liable to forget their earlier responses. When you use screen- and session-recording tools such as Hotjar in tandem with diary studies, they can help record user behaviors and detect rage-clicks, which can be valuable indications of user frustration. Together with the subjective feedback that you can capture in diary entries, session recordings can paint a vivid picture that helps product teams identify problems and devise solutions for them.
Broad and Targeted Diary Studies
Diary studies can be broad or targeted. If your team were working on a meditation app and taking a targeted approach, you would ask users to complete their diary entries based on their app usage alone. But, in the case of a broader study, you would also encourage them to make entries that relate to their having mindful moments when they’re not meditating using the app.
Why Choose a Diary Study?
Diary studies aim to understand what users do, think, and feel when completing a task or having an experience. They provide rich, contextual information about the customer journey, either as it is happening or shortly thereafter. These insights can unlock key information about the end-to-end user experience that, in turn, enable teams to improve the user experience, differentiate it from those of competitors, and gather data that influences the creation of or adds context to a customer-journey map (CJM). If you are trying to understand what users do and why they’re doing it, diary studies are a great option.
What Does a Diary Study Let You Learn?
By conducting a diary study, you can learn the following about users:
Habits and usage:
When are participants using the product?
What prompted them to use the product? (Think of circumstances such as their receiving notifications or because of the specific time of day.)
How long are they using the product?
How often are they using the product?
What leads to or inhibits specific behaviors such as purchasing a product?
What motivating factors led users to perform the tasks?
What did users feel throughout the journey and when they completed the tasks?
What were users thinking during the journey and when they completed the tasks?
How users’ attitudes or sentiments change over time:
Do customers feel loyalty to the brand or product?
Are they pleased with the product? Why or why not?
Did they like a new feature or did it turn them off?
Are they able to learn how to use a new feature?
When I reflect back on the printable yoga and mindfulness product line that I created, I see in hindsight that diary studies could have been instrumental in my learning how people were using the products and how actual users perceived them.
A diary study could have followed teachers and other users as they utilized the content in their classes and given us opportunities to ask questions such as the following:
How did they use it?
When did they use it?
What was their sentiment about their outcomes?
What prompted them to use the materials?
Were there any special moments to share about students applying the learning?
Although I collected much of this data informally, my knowing how to conduct a diary study could have yielded data that would have helped me learn more about my users’ frustrations, wins, and motivations. Now, let’s look at how to conduct a diary study.
How to Conduct a Diary Study
Before you start a diary study, it’s important to be prepared. Diary studies depend on your having a well-defined list of questions whose aim is answering your research questions.
Remember, your participants are human, so respect their time. Your diary study must be compatible with the way people use your product and should occur within the context of a use case for your product. I suggest your encouraging participants to create diary entries either at regular intervals such as once daily, in the morning or evening, or after each use of the product.
Use a variety of question types.
Open-ended questions prompt participants to expound on their thoughts, experiences, feelings, and motivations in a way that multiple-choice questions cannot.
Use questionnaires or prompts to ask specific questions whose aim is to help discover where there is room for improvement. Participants share the actions they took, reactions to these steps, and their assessments or thoughts about their experience.
Voice memos, participants’ videos, and calls with researchers can enable participants to provide their more in-depth thoughts and help researchers to see or hear participants’ emotions. Sometimes diary studies include the option to provide responses that answer more than one type of question.
Schedule participants effectively.
Chasing participants at random intervals won’t yield the data you need. Ensure that you establish a schedule and that participants understand and consent to follow it so they can comply with the study requirements. Plan to use alerts, reminders, texts, email messages, or notifications to prompt participants to make their diary entries.
A Step-by-Step Guide to Conducting Diary Studies
To conduct a diary study, follow these steps:
Planning and preparation—During the planning phase, write your research questions and participant instructions, provide resource materials, and establish how you’ll collect data and prompt participants to make diary entries. At this time, consider whether to recruit participants from current or target users.
Participant recruitment—Recruiting and vetting participants is a critical step before your study can begin. Recruit participants based on the target audience for the specific features of your product that you want to test or evaluate. Recruiting from a high-quality candidate pool that comprises members of your niche audience is imperative because it guarantees that the feedback your team receives comes from individuals who match your research requirements. Participant recruitment and management platforms streamline this process.
Prestudy brief—When doing your prestudy brief, schedule a call with participants to set expectations and provide instructions. Participants have the opportunity to ask you questions and get to know you and your research team. Participants should also know when and how to submit their responses.
Logging period—This is the designated period for collecting diary entries from participants. Something to consider: Strike a balance between the duration of your study or the number of repeat uses of your product that it requires and the amount of time you expect participants to be willing to commit to the study. If you design a study to be too long, you may experience attrition or participants’ filling the diary with inadequately thoughtful responses. If a study is too short, you might miss out on important data about changes in participants’ attitudes over time. A good rule of thumb for encouraging participants to write diary entries over time is making sure that the incentive you offer demonstrates value for the participants’ time. Remember to monitor insights as you receive them so you know when there is a need to ask clarifying questions. Types of logs:
Text logs provide written content such as unprompted feedback about negative experiences that can help you improve the product.
Video or voice memos provide rich data. Participants talk you through their use of the product and explain their thoughts and feelings.
Photos let you see the user’s context.
Post-study interviews—During this interview, you ask participants questions about their overall experience with the product. This is the time for researchers to ask additional questions to clarify what participants thought, felt, and did when completing their assigned tasks.
Analysis of your findings—You can pitch UX design improvements and product iterations to stakeholders based on the key points of friction that participants experienced as they completed the activities that you defined in step one.
Benefits of Diary Studies
Conducting diary studies provides the following benefits:
These longitudinal studies let you collect information over time.
They reveal details about the user’s thoughts, feelings, desires, motivations, and painpoints in a detailed way that other user-research methods cannot.
They provide flexible options for recording entries, including voice memos, email messages, texts, videos, and more. Technology has made dairy studies easier than in the pen-and-paper days.
There are many ways to prompt users, including text reminders, alerts, check-in calls, questionnaires, and the participant-recruitment process. Plus, management platforms can handle automated outreach.
Subject to participant quality, this method is very inexpensive given the richness and reliability of the data.
Because diary studies occur in the user’s home or office, they reduce lab expenses. Plus, being at home encourages users’ more natural behaviors.
Remote facilitation of diary studies allows many to participate concurrently, typically on their own devices.
Limitations of Diary Studies
Diary studies have the following limitations:
It can be challenging to ensure participants are compliant with the study’s requirements. They must consistently complete their diary entries over time.
Communication struggles such as challenges with the legibility of participants’ handwriting for handwritten diary entries, difficulty interpreting people’s strong accents when reviewing voice memos or videos, or entries from people who lack strong written communication skills. You can address these issues during participant recruitment—for example, through appropriate screening for good articulation.
Because of this research method’s emphasis on participants’ experiences and feelings, it is not the ideal method for triangulating on flaws in designs. Ideally, you should use this method in conjunction with other user-research methods.
You can’t always trust what participants say. They are not always honest and sometimes say what they think researchers want to hear or whatever is quicker.
Parsing and analyzing the diary entries takes time and effort.
The keys to successfully recruiting high-quality participants for a diary study depends on finding relevant, articulate, authentic candidates. Do the participants’ traits and experiences match important requirements? Do they have the ability to communicate at an acceptable level and show an interest in helping improve the user experience?
Participant recruitment and management can vary in complexity, depending on how specific your research needs are and how accessible your niche audience is. There are use cases that would allow UX researchers to recruit qualitative-research participants from a user pool to which they have easy access. There are other instances in which researchers would need a specific kind of user to participate—for example, banking executives who would complete a specific action using a specific feature of an app with a certain amount of regularity. In such instances, doing your recruiting using a user-research recruitment platform is cost-effective and can save your research team time, bandwidth, and energy to focus on the actual research.
Convincing Stakeholders That User Research Matters
Rather than simply restate what I wrote earlier, I scoured online research findings for meaningful metrics from thought leaders in User Experience that you can use to help you argue the case for user research as a valid investment, as follows:
“Businesses that invest close to 10% of their total budget on UX are likely to see tremendous ROI. User research improves user experience, and it must be done right.”—Amazon AWS
“The cost of fixing an error after development is 100x that of fixing it before development.”—“The RIO of User Experience,” by Dr. Susan Weinschenk
“UX-centric organizations need fewer support calls, have increased customer satisfaction, reduced development waste, and lower risk of developing the wrong idea.” The UXSchool by Career Foundry
I hope this article has helped paint a clear picture of the use cases, best practices, and operational processes for diary studies. Diary studies are a powerful research method that provides rich, reliable insights into consumer behaviors and sentiment. As UX researchers, isn’t that exactly what we want?!
Lara is an author and content strategist who focuses on the topics of user research, user experience, technology, yoga, and mindfulness. She headed an education company for ten years where she learned about the influence of user research on UX design and the iterative design process. She has a passion for interviewing subject-matter experts and learning about research methods for use cases that influence how businesses make product decisions. Read More