Creating a Universal Usability Agenda
Published: November 6, 2006
Disclaimer: The views in this article are those of the author and do not necessarily represent the views of the Elections Assistance Commission or the National Institute of Standards and Technology.
How do you keep usability, accessibility, and user experience requirements on track while developing standards? It is part of the very nature of standards to focus on details—and in the process, to sometimes lose sight of the real goals. This is especially true when a standards-making process goes on for a long time, a situation is highly political, or most people are focused on technology issues. For over two years, I’ve worked in just such a situation as part of the Technical Guidelines Development Committee (TGDC) creating federal standards for voting systems in the United States.
The Help America Vote Act of 2002 (HAVA) mandated the TGDC and its members—who include election officials, members of the US Access Board, and other experts, working with scientists from the National Institute of Standards and Technology (NIST). The TGDC is an advisory committee, so we don’t actually create regulations or standards, but offer our advice to the Elections Assistance Commission (EAC).
For most people, the mention of voting systems conjures up one of two thoughts: either the hanging chads of a terrible usability disaster during the 2000 presidential election or the need for paper audit trails and the importance of security in voting systems. With everything that people have written and said about security, transparency, trust, and the necessity of our being able to accurately recount ballots, it’s easy for something like usability to seem like a trivial “nice to have.”
But it does matter, and HAVA called for improved standards for voting systems and required that they allow individuals with disabilities to vote “in a manner that provides the same opportunity for access and participation (including privacy and independence) as for other voters.” (HAVA 301(a)(3)). The subcommittee on Human Factors and Privacy is responsible for drafting guidelines for usability and accessibility, according to the provisions of this law.
This article looks at how we created an agenda to guide our work, including decisions about how to create the new requirements. Our approach is also applicable in other situations, such as creating usability guidelines for a product or focusing corporate attention on user experience.
Standards are all about details, and it’s only too easy to lose sight of the big picture while writing and debating specific requirements. A committee of appointed members not only brings together different perspectives, but means there will likely be process, political, and administrative overhead. Sound familiar?
To combat this entropy, we created a set of guiding principles and voted to adopt them as official resolutions. Viewed as a whole, the following resolutions created an agenda for a universal approach to the usability and accessibility of voting systems:
- By agreeing on basic principles in advance, the committee could focus on the details of a complex standard with a shared understanding of its goals.
- By defining relationships and important dependencies at the beginning of the project, longer-term work could start immediately, so it would be ready when needed.
- The resolutions could span the development of several versions of the standard.
These resolutions provide both a high-level view of our usability and accessibility goals, as well as specific directives for how to organize the work. This was important for a committee that included many stakeholders, not just user experience and human factors experts. Most importantly, the resolutions define the full scope of the project, and we can use them as a measure of success for the completed standard.
The Human Factors and Privacy Resolutions
Accessible Voting Systems—The first resolution acknowledged the HAVA requirement for accessible voting systems and called for the guidelines to both incorporate the latest available accessible technology and draw on existing regulations for accessibility. This allowed us to build on the work of the Access Board, W3C, and other international standards bodies.
Human Factors and Privacy Requirements for Capturing Indication of a Voter’s Choice—This companion to the first resolution focused on how more usable systems could support voters’ ability to be accurate and efficient in casting their ballots.
Human Factors and Privacy of Voting Systems at the Polling Place—US elections use a secret ballot, and this resolution made the point that usability, accessibility, and privacy are functions of both the system we use to vote and the environment of the polling place. Although our guidelines cover only the system itself, we believed there was an opportunity to provide guidance on good election procedures to support the equipment requirements.
Human Performance-Based Standards and Usability Testing—This was one of the most forward-looking resolutions. It called for standards for performance that are measurable with usability testing. Performance standards allow innovative solutions to the challenges of creating a voting system and avoid narrow design requirements, while still requiring systems to be usable and accessible.
Accommodating a Wide Range of Human Abilities—The voting population includes not only people with specifically identified disabilities, but also an aging population, language minorities, and people with other special needs. This resolution set out the principle that all voting systems—not just so-called accessible systems—should accommodate a wide range of abilities. This universal usability approach to the guidelines will make voting systems more usable for all.
Usability Guidance for Instructions, Ballot Design, and Error Messages—Instructions for voters and poll workers, ballot design, error messages, and Help are important components of any system, so this resolution called for improving the usability of instructions, error messages, and Help through plain language and good ballot design, in all necessary formats.
General Voting System Human Factors and Privacy Considerations—This resolution ensured that the committee would consider the usability and accessibility of all aspects of the system, including security features and other functional requirements, and review all guidelines for their implications for human factors.
Usability of the Standards—Anyone who has read a standard knows that they are often written in arcane language. We wanted to express the requirements in clear, plain language that is understandable to experts in usability testing, voting officials—who may not be experts in human factors—and advocates who are interested in voting systems.
Availability of Voting Machines for Validating Benchmarks and Conformance Test Protocols—The last resolution covered important housekeeping. Creating a standard usability test meant that the staff needed access to a variety of real voting systems. This resolution enabled NIST to set up a program under which vendors could make their systems available to the team developing and testing these guidelines.
How Well Did It Work?
No standard is perfect—especially one that is created in the political hot-house environment of special interests in elections and voting systems. The standard, called the Voluntary Voting Systems Guidelines (VVSG2005) includes the first usability requirements for voting systems and enhanced accessibility requirements for voters with disabilities. There are specific requirements covering partial vision, blindness, dexterity, mobility, hearing, and cognition. See the sidebar for a timeline for the creation of the VVSG2005.
During the process of drafting the VVSG2005 and an update that is currently in progress, the resolutions have allowed us to keep the usability and accessibility work on track:
- We’ve used the resolution on universal usability to support our approach of creating general usability guidelines that make all systems as accessible as possible.
- The resolution on performance guidelines provides the basis for creating ground-breaking benchmarks and has enabled us to start the research and pilot testing necessary to write performance-based requirements and create a repeatable test for conformance.
- Our insistence that the standards themselves be usable has resulted in a prolonged tug-of-war as we have tried to both write the individual requirements clearly and structure the whole document in a way that reduces its complexity and the number of nested levels.
One of the most interesting things that happened occurred during the comment and revision process: Some of our guidelines were upgraded. For example, the draft version only recommended usability testing as part of the certification testing. However, public comments—such as one from Senator Christopher Dodd, of Connecticut, who called for “requiring … not merely encouraging them, to conduct summative usability tests”—supported the idea strongly enough that the final version of the standards requires usability testing—both with the general public and people with disabilities. It may be the only federal regulation to do so.
Human Factors and Privacy subcommittee members—Whitney Quesenbery (Chair), Philip Pearce (US Access Board), John Gale (Secretary of State, Nebraska), Alice Miller (Director of Elections-District of Columbia), and Sharon Turner Buie (Director of Elections-Kansas City, Missouri)
NIST staff for Human Factors and Privacy—Sharon Laskowski and John Cugini
Past members include James Elekes (US Access Board) and Donetta Davidson (Secretary of State, Colorado), now on the Elections Assistance Commission.
If you would like to learn more about the effort to create federal standards for usable, accessible voting systems in the United States, here are some online resources:
- The Voluntary Voting System Guidelines (VVSG2005) are on the Election Assistance Commission Web site.
- Information about the Technical Guidelines Development Committee—including reports, hearings and meeting notes—are on the vote.nist.gov Web site.
- Janice (Ginny) Redish and Dana Chisnell are doing research on instructions and messages. Their initial report comparing best practices to actual ballots is “Plain Language for Instructions and Messages Research.” They also presented a paper titled “Making Ballot Language Understandable to Voters” at EVT06.
- Bill Killam and User-Centered Design are working to create a benchmarking usability test for voting systems. Their initial report on Usability Conformance Tests Research (Word document) outlines the protocol for conducting such tests in preparation for pilot testing.
- All of this work on defining usability tests for voting systems builds on a report from a UPA 2005 workshop on the usability of voting systems, “Defining a Summative Usability Test for Voting Systems.”