Designing internally facing systems differs from designing externally facing systems in two important and interrelated ways:
- The UX design can take into account the potential for training users on a new user interface.
- It is possible to test the acceptability of a new system.
It is the latter of these two differences that is the subject of this article.
What Is Acceptability Testing and What Is Its Rationale?
In the 1970s, many studies of large-scale, internally focused information systems found that their stakeholders rated only about 20% of them as successful and, in around 40% of cases, there was either just a marginal organizational gain or users completely rejected the new system! Despite the massive technical advances in information technology (IT), the situation remained poor throughout the ’90s.
Such findings led many senior researchers to conclude that, of the many tasks necessary for a successful, internally focused IS initiative, one of the most crucial is the early assessment of the expected organizational impact of implementing a new information system—that is, the actual impact, not the ideal or desired impact.
Typically, an organization does not fully appreciate the impact of an IS implementation on its employees until development work is well advanced—when it may be difficult to adjust the development process—or, worst still, only after the implementation of the new information system is complete. This is unfortunate because the more accurately and expediently an organization can assess the system’s actual organizational impact, the greater the potential there is for adjusting the information system design during development and planning any organizational changes that are necessary to make the implementation a success.
IT human factors can make such an assessment by doing acceptability testing, which in my experience, is an approach that is little known in the UX community, but has great importance when working on internally focused IS projects. To understand this approach better, we must first comprehend the broader scope of socio-technical system design.
Socio-technical System Design
Over the last two decades, the majority of researchers and consultants who have addressed issues of the organizational impact of new, large-scale IS systems have employed socio-technical system design thinking to better understand the key forces at play. This has helped them to understand the interplay between the IT system, business processes, and an organization’s social system. Business people now typically have a good understanding of the key concepts of technical systems and business processes, and most large-scale IS projects include many experts in these areas. However, the key concepts of social systems are still not widely understood.
What is a Social System?
As the Organisation Development Centre and many others have explained, people in organizations necessarily work together—for example, in departments, teams, and workgroups. The social sciences refer to these as social systems. A social system functions according to a set of underlying rules, a working culture, facts, myths, shared values, power relationships, and a generally shared perception of organizational icons—for example, “the boss.” People express these things both formally and informally. Similarly, they understand them both consciously and unconsciously.
Social Systems and Resistance to Organizational Change
As the Organisation Development Centre also points out, if an organization’s social system neither experiences nor anticipates change, it appears to be relatively stable. Individuals are familiar with the variety of norms and rules guiding the system, and they have developed means of exploiting these to best meet their own particular needs.
Lewin, a famous social psychologist, referred to this situation as quasi-stationary equilibrium, in which a social system only appears to be stable. There is always movement because of the vast number of tacit, or unconscious, negotiations that allow the system to serve the interests of its individuals, but the totality of all movement equals zero—that is, the system remains in what appears to be a state of balance. Another way of describing these negotiations is as forces whose aim is to stabilize the system—that is, forces that resist changes in the organization.
While it can be useful to consider a social system as a single entity, ultimately, social systems comprise individuals, and organizational psychologists have proven that this quasi-stationary equilibrium serves important psychological functions for these individuals—even if the system is neither satisfactory for everyone nor completely satisfactory for anyone. Further, the work of Macefield and Mellor and others on psychodynamics in organizational contexts explains how some individuals may perceive changes in this equilibrium as a very serious threat, extending even to their physical survival. Importantly, they note that, while individuals are often conscious of such threats—which they may experience as increased work stress—they are unlikely to be consciously aware of the full severity of the perceived threat and are even less likely to express this overtly.
This means that individuals within social systems are typically willing to commit considerable mental resources in mounting robust defenses against changes in social systems. Often, they will not verbalize their defenses—possibly not even in their own conscious thoughts. Thus, the key point here is that resistance to change in social systems can be, and often is, extremely potent, tacit in nature, and difficult to predict!
Socio-technical Impact Patterns
Based on extensive research, Eason, a professor of cognitive ergonomics and a leading socio-technical design thinker, identified a pattern of actions and reactions to the way a new technical (IS) system, business process, or social system tends to play out in situations where there is business-driven change. This is illustrated in Figure 1.
Key and Explanations
A. New business drivers result in a new set of strategic business goals. These translate into a requirements specification for the new technical information system.
B. Once implemented, the new technical system immediately affects the business processes—quite possibly as the organization intended. The immediacy of this reaction is such that, in many cases, the organization designs A and B to act simultaneously, within an approach known as ISE-BPR (Information Systems Enabled Business Process Re-engineering.
C. Soon after, the technical system begins to affect the social system by altering power relationships and levels of individual discretion.
D. A further assault on the social system follows, resulting from associated changes in business processes.
E. There is then a backlash from the social system that affects, or resists, both the technical system and the business process, with the intent of restoring the original state of the social system—that is, to restore the original power relationships and levels of individual discretion.
The intensity and nature of the backlash (E) varies across both individuals and organizations as a whole, but the extremities range from physical sabotage of IT equipment and widespread industrial action through mostly harmless expressions of dissatisfaction and relatively minor reductions in work effort. Common reactions include
- withholding help and encouragement in using the new information system and associated business processes—This is essentially a passive-aggressive approach—that is, aggression that results from not doing something—and often occurs in the interrelationships between front-line workers and their immediate, junior management.
- disrupting the power base—For example, working to gain or regain control or passing on inaccurate or misleading information—either about the system or via the system—through management and the supply chain.
- operating dual systems—This often means keeping a personal system as well as using the new, official system, with the intent of regaining power and discretionary control in the work environment.
Ignoring the Backlash
Understandably, many business professionals get seduced into thinking that they can easily address this backlash by taking an approach that is typified by such comments from project sponsors as these: “We’ll just make them use the new system.”, “Make them do things the new way.” And “We’ll just sack anyone who fails to comply.” Business people making such comments typically justify this approach by citing the theoretical business benefits of using the new system and doing things the new way. Unfortunately, while their thoughts about the potential benefits are often accurate, unlike business processes and the information systems that serve them, a desire for business benefits does not typically drive social systems—nor are they necessarily rational in nature!
It is true that some organizations have successfully addressed this issue by essentially dismantling the social system—usually along with the obsolete business processes. This typically involves radical steps such as breaking up the business and re-employing both new and original staff members, in new or different roles, and on new types of projects—along with a deliberate re-engineering of the corporate culture. We can find some examples of this approach within former, public-sector monopolies that have moved to privatization—for example, British Telecom in the UK. However, this approach typically carries considerable business risk and requires vast cash reserves and highly skilled change agents. To summarize, it is sometimes possible, indeed necessary, to bludgeon your way through a social system, but this is not a viable option in many instances.
Perhaps more importantly, it is naive to believe that an individual can be made to use an information system. This assumption implies that system usage or misusage is a binary condition. In contrast, modern information systems are complex, subtle, and powerful in nature. Therefore, an individual can be using the system, but doing so in an inefficient, slow, or inaccurate way—for example, entering information that is subtly incorrect.
Organizations can, and should, use modern IT systems creatively to exploit new business advantages. Therefore, simply failing to apply such creativity can constitute misusage of the system. In summary, it is usually difficult to conclude definitively that an individual is misusing a system, even more difficult to prove that misuse is taking place, and more difficult still to prove that this is a conscious, deliberate act on an individual’s part.
Addressing the Backlash
Given that some kind of backlash from the social system is inevitable in most cases, it is easy to argue that the best way forward is to acknowledge, accept, and work with the backlash rather than just ignoring or fighting it. This presents two challenges:
- Assessing the nature and intensity of the backlash as early and accurately as possible.
- Using this feedback to adjust the technical development and take deliberate steps to influence the social system in a way that minimizes the negative aspects of the backlash and, indeed, even maximizes any unforeseen opportunities that the disruption in the social system may bring.
Assessing the Backlash
Clearly, assessing such potential backlashes at any point prior to implementation is a challenging problem, but as Eason pointed out, a useful starting point is to consider that a new information system impacts the social system at different levels, as shown in Figure 2.
Figure 2 illustrates that there is a point of contact between acceptability engineering and usability engineering at the interface between the technical system and the user. However, it is important to recognize that acceptability and usability are qualitatively different things that we should not confuse. A system may be highly usable, but totally unacceptable!
Using this framework, Eason developed a technique now known as Eason’s User-Cost Benefit Analysis, for the purpose of assessing the potential backlash from a social system upon the implementation of a new information system.
Eason’s User-Cost Benefit Analysis
Eason’s User-Cost Benefit Analysis consists of a process comprising 3 stages.
Stage 1: Defining the Changes
Based on the new information system’s intended functionality and the associated business-process changes, we can define what we believe will be the key changes that will occur for key individuals, in terms of the following hierarchy:
- Tasks that a particular IS initiative defines:
- Key task 1
- Key task 2
- Key task 3 …
- Job security
- Security within the organization
- Employability outside the organization
- Job content
- Task variety
- Effort necessary
- New skills necessary
- Old skills lost
- Work pace and deadlines
- Discretion and autonomy
- Standardization and formality
- Personnel policies
- Basic pay
- Other rewards
- Career prospects
Stage 2: Presenting the Impact
We can then present these changes to the individuals who are concerned. There are several key points to note at this point:
- The presentation needs to take a form that is suitable to the audience. Typically, this means it must be free of esoteric formalisms and notations. Demonstrations of functional prototypes can be greatly beneficial.
- Different presentations may be necessary for different audiences, which we can often define in terms of the different personas that we used during the system’s design.
- Eason’s research in applying this technique indicates that the way in which an organization presents these changes is often a key influencer of how well the social system reacts to the changes. Thus, this is a classic case of how the act of measuring something can also affect what is being measured.
The delivery of the presentation to individuals is typically viable only as a group exercise. Sampling individuals who are representative of each persona is often a practical requirement. Similarly, it may also be necessary to abstract those personas that we predict will be key to the success of an initiative and subject to the greatest change resulting from the introduction of the new information system.