The Role of Iterative Usability Evaluation in Agile Development: A Case Study
Published: August 18, 2014
The agile approach to software development has significant impacts on the practice of user-centered design (UCD), including usability evaluation. To better understand the role of iterative usability evaluation during agile development, we recently conducted a study whose focus was the usability evaluation of a personal health–management system. The complexities of healthcare systems require thoughtful and well-structured usability evaluations—especially when the design process occurs within the context of an agile development process.
Our study identified three different stages of the usability-evaluation process. Usability experts, system developers, and users participated at different stages of this process, which occurred iteratively during each two-week sprint. Our research also offered insights into how usability experts perceive their roles during rapid, iterative collaboration with system developers and users. We learned that usability experts serve as an essential bridge connecting system developers and users.
Introduction to Usability Evaluation in an Agile Software Development Process
The emergence of the agile software development manifesto has changed the practice of user-centered design (UCD).  In contrast to traditional approaches to usability evaluation, iterative usability evaluations within an agile context should be rapid and efficient, but must still reveal any problems that newly added features have introduced. Software engineers have rapidly adopted the agile approach, and embedding usability evaluations in an agile software development process provides timely feedback from users and results in intensive collaboration between usability experts and software developers. [2, 3] In particular, it facilitates a collaborative approach to usability testing.  However, while agile development encourages creative teamwork and fosters effectiveness and maneuverability, the underlying iterative processes are sometimes confusing and hard to distinguish because of the rapidity of the ongoing development process. [5, 6, 7]
While software engineering employs various software-development processes, focusing on usability to different degrees, prior research has rarely disclosed explicit usability-evaluation processes that are integrated into the agile development process. [8, 9] In particular, there has been a lack of research on usability evaluations’ ill-defined iterations within the context of the agile development of personal health–management systems. In contrast with the development of systems other than healthcare systems, the usability evaluation of healthcare systems is more complicated.  With the greater emphasis on iterative, formative evaluation in agile development, there is demand for closely integrating usability evaluation within the development process in engineering healthcare systems. [11, 12] Therefore, our study aimed to describe the explicit process of iterative usability evaluation in the agile development of a personal health–management system.
Because an agile development approach emphasizes the value of collaboration and the rapid feedback that iterative usability evaluation can provide, its effectiveness typically relies more on the involvement of usability experts. This helps to reduce the investment of time and money in developers’ decision making and the delivery user feedback.  However, previous studies have not identified a distinct role for usability experts working with designers. [14, 15]
In examining the role perceptions of usability experts who were collaborating with developers and users in the agile development process, our study was founded on the theory of Role Identity Salience,  which provides a conceptual bridge linking the individual to the larger social structure that emphasizes self-definition, behaviors, and social relations. The concept of role-identity salience plays a critical part in many contemporary discussions of the concept of the self.
The Purpose of Our Study and Its Findings
Our research aimed to discover the explicit processes that constitute iterative usability evaluation in agile development, as well as to increase our understanding the roles usability experts play in an agile context. Two research questions guided our study of the agile development of a personal health–management system:
- What constitutes an iterative usability evaluation process in agile development?
- How do usability experts perceive their roles in an agile development process?
What Constitutes an Iterative Usability-Evaluation Process in Agile Development?
In the case we studied, the development of a personal health–management system, the usability-evaluation process extended across three stages within each two-week sprint. Usability experts, system developers, and users participated during different stages of the usability-evaluation process, as Figure 1 shows.
Figure 1—Three stages of usability evaluation within each two-week sprint
The system developers and the usability experts resided in geographically dispersed locations. Therefore, during the first week of each sprint, the development team and the usability team participated in a conference call to decide what parts of the application they would be testing—in the form of either paper prototypes or live code. The usability team familiarized themselves with the application, brainstormed ideas, identified their usability-evaluation goals, developed a think-aloud protocol for test sessions, as well as a protocol for user interviews, and recruited participants for the study.
During the second week of each sprint, the usability team conducted usability testing with the participants they had recruited. To facilitate and speed up the delivery of feedback to the team, a facilitator led each usability test session, and a note taker captured users’ comments during each session. Typically, five to eight users participated in each cycle of usability testing.
At the end of each two-week sprint, the usability team synthesized their findings and the participants’ feedback and reported the results of the study to the development team, providing an Excel spreadsheet and video clips that documented the test sessions, as well as their recommendations for improving the application. Then the two teams participated in another conference call to discuss the findings, provide clear answers to any questions, and address any concerns.
How Do Usability Experts Perceive Their Role in Agile Development?
During each two-week sprint’s cycle of usability evaluation, three to five usability experts were assigned as members of the agile development team to plan and facilitate the test sessions. A member of the usability team served as a facilitator, who led the usability testing; a note taker, who recorded users’ feedback; and a team leader, who synthesized users’ feedback. A team leader could also be a facilitator or a note taker. The usability experts who facilitated the user interviews and conducted usability testing described their specific tasks as follows:
- Meet with the system developers to discuss and set the objectives of the usability evaluation.
- Develop protocols for user interviews and usability testing.
- Conduct usability testing to discover usability issues and collect users’ feedback.
- Analyze the findings from usability testing and the user feedback that was collected during interviews.
- Report the findings and user feedback, and make suggestions for improving the software to the system developers.
After examining the usability professionals’ perceptions of their experiences conducting usability evaluations in an agile context, we found that the usability experts played significant roles during each two-week sprint throughout the entire development process, as shown in Figure 2.
Figure 2—Usability experts’ perceptions of their roles in agile development
The usability experts’ most important role was serving as a bridge connecting system developers and users. In each sprint throughout system development, the usability experts tested developers’ assumptions in their usability-evaluation sessions with users. During the usability evaluations, the usability experts played a vital role in obtaining users’ feedback. Finally, they were responsible for reporting users’ feedback to system developers and provided appropriate recommendations for the system’s improvement.
The Need for Usability Evaluation of Healthcare Systems
Many new healthcare applications—particularly personal health–management systems—require continual, iterative usability evaluations while they are under development. Our research identified and provided an example of the explicit processes of iterative usability evaluation and its implementation in an agile development environment. The findings from a previous study by Ammenwerth and others  stated that, considering the different stakeholders who are involved in the development of a healthcare system, it is necessary to plan for iterative usability evaluation over the long term, allowing a development team working in an agile environment to integrate the new findings and user feedback into their implementation.
The principal strategy for usability evaluation should be to take into consideration the complexity of healthcare systems, while the consequence of usability evaluation should be decreased complexity for users.  Our study provided evidence that the rapid development and iterative prototyping that characterize an agile approach  should rely on usability evaluation to provide designers and developers with the necessary inputs to enable them to improve the usability of complex healthcare software. Because agile development emphasizes collaboration and communication among developers, usability experts, and users, it can lead to a blending of these roles.  However, usability experts can serve as a bridge connecting users and system developers—enabling them to understand the needs of the users of a personal health–management system.
Our Recommendations to Usability Professionals
The integration of usability evaluation into an agile development approach for a healthcare system requires a high degree of adaptability to change, savvy collaboration between the system developers and the usability team, and timely planning for the involvement of users. Therefore, usability experts need to work with system developers on sprint planning and tracking. Long-term agile development requires continuous cycles of design, usability testing, and making design improvements throughout all development iterations.
Based on our study, here are a few tips for usability professionals conducting usability evaluations in an agile development environment:
- To ensure the satisfaction of multiple user personas working with health-information technology, conduct usability evaluations on a regular basis throughout the agile product development lifecycle.
- Identify the design changes that have been made to an application since the last development cycle, and ensure that the system developers understand the usability-evaluation goals by maintaining effective communication with them—especially when the design changes are subtle.
- Prior to doing usability testing, conduct a pilot study, then develop an appropriate interview protocol.
- In comparison with non-agile development processes, agile approaches require intensive collaboration between system developers and usability professionals and short cycle times, which makes the recruitment of usability test participants a highly demanding process.
The Methodology for Our Study
This study explored a specific case: the agile development of a personal health–management system. The development team worked in collaboration with a usability testing laboratory at a large Midwestern university in the United States. The usability testing laboratory conducted 14 cycles of usability evaluations over a period of 12 months, in 2011 and 2012. A total of 59 users participated in the usability evaluation sessions, during which the usability team asked each participant to perform a list of predefined tasks in a controlled lab setting. The usability team asked participants to verbalize their thoughts and actions, or think aloud, while performing the tasks. Each usability test session lasted approximately 45–60 minutes. The usability team recorded each session, using Morae 3.2.1.
Data Collection and Analysis
To answer our first research question—What constitutes an iterative usability-evaluation process in agile development?—we analyzed all 14 cycles of usability evaluation. The data that we analyzed included meeting minutes, documents on the development team’s expectations, explanations of the applications being tested, task lists, the usability team’s notes, and their final reports. Our analysis focused primarily on the activities that were part of each cycle, the people who were involved in product development and usability evaluation, and their responsibilities. We also analyzed data about each cycle of usability testing, including the duration of each cycle, the number of participants, the richness of the feedback that the participants provided, and the development team’s satisfaction with the results of the usability evaluations.
To answer our second research question—How do usability experts perceive their roles in an agile development process?—we examined the role perceptions of the usability professionals who participated in the usability evaluations. According to Biddle,  most versions of role theory assume that people’s expectations are the primary generators of their perceptions about roles that they learn through experience, and people are aware of the expectations that they hold. Therefore, we conducted semi-structured interviews with five of the usability professionals who were actively involved in multiple cycles of usability evaluation of the personal health–management system. We open coded the data that we collected from the interviews and identified common themes.
Our future research might examine the role perceptions of other stakeholders who were involved in the usability-evaluation process—for example, the system developers—to allow us to build a better, more comprehensive understanding of iterative usability evaluation in the agile development of a healthcare system.
We would like to express our deepest gratitude to the people who helped and supported us throughout our study, especially to our colleagues Josipa Basic and Ben Richardson for their insightful comments.
 Singh, Mona. “U-SCRUM: An Agile Methodology for Promoting Usability.” AGILE ’08 Conference. New York: IEEE, 2008.
 De Vreede, Gert-Jan, Ann Fruhling, and Anita Chakrapani. “A Repeatable Collaboration Process for Usability Testing.” HICSS ’05. Proceedings of the 38th Annual Hawaii International Conference on System Sciences. New York: IEEE, 2005.
 Sy, Desirée. “Adapting Usability Investigations for Agile User-Centered Design.” Journal of Usability Studies, May 2007.
 Ferreira, Jennifer, James Noble, and Robert Biddle. “Agile Development Iterations and UI Design.” Agile Conference (AGILE), 2007. New York: IEEE, 2007.
 Highsmith, Jim, and Alistair Cockburn. Agile Software Development: The Business of Innovation. New York: IEEE Computer Society, 2001.
 Paetsch, Frauke, Armin Eberlein, and Frank Maurer. “Requirements Engineering and Agile Software Development.” IEEE 21st International Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises. New York: IEEE Computer Society, 2003.
 Nawaz, Ahsan, and Kashif Masood Malik. Software Testing Process in Agile Development. Computer Science Master Thesis, Blekinge Institute of Technology, 2008.
 Göransson, Bengt, Jan Gulliksen, and Inger Boivie. The Usability Design Process: Integrating User‐Centered Systems Design in the Software Development Process. San Francisco: Wiley Online Library, 2003.
 Eklund, John, and Ciaran Levingston. “Usability in Agile development.” UX Research, 2008.
 Ammenwerth, Elske, Stefan Gräber, Gabriele Herrmann, Thomas Bürkle, and Jochem König. “Evaluation of Health Information Systems: Problems and Challenges.” International Journal of Medical Informatics, 2003.
 Kushniruk, Andre W., and Vimla L. Patel. “Cognitive Evaluation of Decision Making Processes and Assessment of Information Technology in Medicine.” International Journal of Medical Informatics, 1998.
 Kushniruk, Andre. “Evaluation in the Design of Health Information Systems: Application of Approaches Emerging from Usability Engineering.” Computers in Biology and Medicine, 2002.
 Cockburn, Alistair, and Jim Highsmith. “Agile Software Development: The People Factor.” Computer, November, 2001.
 Wusteman, Judith. OJAX: A Case Study in Agile Web 2.0 Open Source Development. Bingley, UK: Emerald Group Publishing Limited, 2009.
 Kollmann, Johanna, Helen Sharp, and Ann Blandford. “The Importance of Identity and Vision to User Experience Designers on Agile Projects.” AGILE ’09. New York: IEEE, 2009.
 Callero, Peter L. “Role Identity Salience.” Social Psychology Quarterly, September, 1985.
 Rouse, William B. Health Care as a Complex Adaptive System: Implications for Design and Management. Washington, DC: Bridge-Washington-National Academy of Engineering, 2008.
 Biddle, Bruce J. “Recent Development in Role Theory.” Annual Review of Sociology, 1986.