Chapter 11: A Pattern Language for Talking About Interactions
Epistemic actions are … a mechanism for spreading cognitive processes across brain, body, and the world. They allow us to see wildly different behaviors as serving the same cognitive purpose. …
In this chapter, we are going to describe a suite of epistemic interactions that people use to create meaning, solve problems, make decisions, establish plans, analyze information, and do other cognitively complex tasks. Taken together, they serve as a versatile vocabulary for describing how people figure things out, … an invaluable framework for pinpointing the underlying cognitive work that drives the understanding process. …
Our interaction vocabulary … provides a way to separate the visual trappings of our digital world—windows, menus, and scrollbars—from the essence of how we interact to figure things out. Moreover, these interactions are not tied, in any way, to a particular technology. They can also be used to describe how we create understanding with paper-based technologies, or virtual reality, or any other technology—even ones that have yet to be invented. …
Four Interaction Themes
Our interaction vocabulary comprises fifteen epistemic actions. They are divided across four major themes and, taken as a whole, these nineteen terms—that is, four themes and fifteen interactions—constitute a broad vocabulary to describe how we create understanding from information, as shown in Figure 11.1. The four themes are:
Foraging: Locating resources that will lead to understanding
Tuning: Adjusting resources to align with desired understanding
Externalizing: Moving resources out of the head and into the world
Constructing: Forming new knowledge structures in the world
The interactions are organized into general themes, rather than narrow categories, because each interaction can fall under multiple themes. For example, rearranging refers to changing the spatial position and orientation of a resource. When rearranging is directed at small scale change—sorting search results to make the cheapest flights appear at the top—it is more about tuning. When rearranging creates substantial change and plays a central role in understanding, such as when a hiring manager divides resumes into different piles to reflect a preliminary assessment of job candidates, it fits better under constructing.
Note also the word resource instead of information. We described information as a resource in the opening chapter. While we have tended to use the word information here, in this and subsequent chapters, we are going to lean more heavily on this notion of information as a resource. This is because embodiment forces us to consider the physical world and, especially, our habit of recruiting objects as props and scaffolds to power thinking. That’s what happened in the restaurant, where sugar packets and coffee cups were used to explain the complexity of a niche area of the American healthcare and pharmaceutical industry. The sugar packet was just a sugar packet, until it was co-opted into the understanding process and took on a representational role. The term information tends to make us think of words, numbers, and documents. Representation makes us think of images and pictures. The word resource, however, reminds us that anything in the world, including what’s in our heads, can help with understanding.
Foraging for Resources
When you hear the word foraging, you likely picture someone scrounging around the forest for berries, firewood, or mushrooms. Today, we forage for information more than berries or anything else. We are always on the lookout for information. It’s the essential resource of civilization. If aliens visited earth, their anthropologists would conclude that information foraging is what humans do most. Even our own scientists have long described our species as informavores—biological organisms that consume information. There is even a theory of information foraging, which posits that how early humans learned to find food was the evolutionary basis for how modern humans look for information.
We forage for information everywhere. We tap on our glowing screens, select avocados in the grocery store, and scan the parking lot for empty spots. Foraging happens anytime we need information from the world, in any form, to accomplish a task. We can distinguish between at least four types of foraging interactions: searching, probing, animating, and collecting.
Let’s start with the prototypical foraging interaction: searching. Searching is what happens when you set out to find resources in the world. You might search for a known fact, such as how many times Robert Redford won an Oscar for his acting work, or something more vague, such as when you need to fix a leaking dishwasher. No matter what you’re looking for, searching happens when you need information and believe it exists somewhere.
While the web has forged an association between searching and typing words into a box, this is just one way to search. Searching means to look for and locate information in the world. On our screens, we search by typing words, scanning results, and clicking through options. But we also search by scanning our screens for facts and figures and tweets and buttons to press. Beyond our screens, we routinely search with our body. When you need a replacement light bulb, you search the hardware store for clues that will guide you to the lighting section and then, once there, you search by adjusting your body to scan the shelves for the right kind of bulb. You don’t need Google to search.
Searching, no matter what form it takes, depends on how we articulate our information needs. With the search box, we use words. With our body, we rely on spatial positioning and orientation. When it comes to visual representations, we scan with our eyes to locate important features, elements, and patterns. All searching.
Probing is a constant companion to search, and is summed up by the phrase “tell me more.” You probe when you follow links to the next page, zoom in for a closer look, and scroll down to the next paragraph. Probing happens when the information you have isn’t quite enough. You are probing when you take the next step, move to the next level, and obtain more salient specifics. Probing isn’t about switching direction or following a tangent. It’s about drilling down and saying to the world “show, explain, and reveal more about this.”
Probing is a relentless feature of everyday life. If, while reading this book, you pause to read a footnote, you’re probing. If you’re listening to some music and ask Alexa to name the song, that’s probing, too. If your child asks why the sky is blue, they are searching for information, and when they question you on the finer points of your explanation, they are probing you. Searching locates information; probing pursues that information one step further.
As technology delivers more and more information to our fingertips, it feels as if we should be able to probe anything. The technology writer Kevin Kelly tells the story of a mother in a grocery store. When she stopped to examine a product label, her son suggested she “just click on it.” The kid had the right idea. When you point your smartphone camera at a soup can or desk lamp, image recognition apps—Google Lens, for example—will display details about the object including brand, price, and purchase options. These apps can identify artwork, plants, buildings, people, and eventually almost any person or thing. The technology feels magical, yet as an interaction for gaining knowledge, it’s probing all the way down.
We probe because we suspect there is useful information just around the corner. It happens when the information at hand isn’t enough, and we go looking to evaporate our dissatisfaction. But probing give us more than info-nuggets. We can probe to reveal new patterns, structures, and relationships. It brings to light new information that helps us reconsider what we already know and, from that, we can build more robust mental representations. …
Do we forage in ways other than searching and probing? Consider a study where people learned to tie knots from video clips instead of diagrams. For some knots, people could only watch the videos from beginning to end, although they could watch it as many times as they wanted. For other knots, they could control the video—pause, rewind, slow it down, and so forth. When people had this type of control, they learned to tie the knot in less time and needed less practice.
Thanks to the controls, the learner was able to focus their attention on troublesome and confusing parts of the knot. It let them “adapt the pace and density of the visual information presentation to their cognitive processing needs.” The controls made it possible for each person to focus on what mattered to them. When learning the bowline, for example, some people paused on the opening step while others slowed down the latter stages where the rope passed through a loop, around the main line, and back through the loop.
This is animating. It means to initiate and control motion in an information resource. When you learn knots from a book, you follow a sequence of images. You need to infer what happens between each step and how to twist and turn the rope. Animating brings this information to the surface and, as a result, it’s a kind of foraging. Animating is similar to probing, but where probing increases the level of detail, animating makes information more comprehensible by showing how it changes over time. Every interaction happens over time, by definition, but the benefits of animating exist in relation to time. This is especially helpful for understanding transitions and transformations.
Movement can clarify, and it can also confuse. If static images are good, then dynamic, moving, animated ones must be better. But this assumption is not born out by the research: animating is beneficial in some situations, but not all. While each successive frame reveals new information, it also removes the previous frame. This puts a new burden on working memory. Pausing, rewinding, changing speed—this reduces the problem. Other times, it’s better to show information as a series of individual images so they can be examined carefully, a method known as small multiples. But when time and motion are central to understanding, animating can be powerful.
Foraging creates a new problem: too much information. Information appears on the scene, and then we forage for more and the prior information flutters away. Our brain-bound memory is rapidly overwhelmed. Collecting is how we gather foraged information and tuck it away for future use. Until we invented writing, this meant storing information in our head. Since the days of clay tablets, we have sought to invent ever new ways of amassing information like squirrels gathering nuts for the winter.
If your web browser has a lot of open tabs or windows, each showing a different page, that is collecting. If your desk has piles of articles or magazines or other papers, that is also collecting. Have items in your Amazon shopping cart that have been lingering there for weeks or months? Again, collecting.
In the era before the web, electronic search systems were designed entirely around keyword searching. The information scholar Marcia Bates proposed an alternative model that she called berrypicking. While her metaphor evokes our broader notion of foraging, it also raises a critical question: What happens to the “berries” that are picked? You gulp down some of them on the spot, but quite a few more are collected.
We collect and clip for all kinds of reasons, including as a trigger for future work, to capture and evoke certain memories, and to share with others. Collecting is rampant and essential in a world of cheap, abundant, easily accessible information. Apps such as Evernote, Pocket, OneNote, and many others all exist to solve this problem. Even so, our homes and cubicles and kitchens are filled with scraps of information captured on sticky notes, scribbled in notebooks, and stuck on the fridge door. We collect incessantly.
Foraging Interactions, Summarized
Let’s summarize our foraging interactions with some concise definitions.
Searching: To look for, or locate the position of, resources in the world
Probing: To acquire more detailed information from the world
Animating: To initiate, and optionally, control motion in a resource
Collecting: To gather resources for future use
Understanding problems often begin with foraging. We search for data, then analyze. We spot a new flavor of granola on the grocery shelf, then flip over the box and probe for the calorie count. We collect items in the cart, then compare. Once we have information, then other interactions come to the fore. Foraging never disappears, but as understanding proceeds, it often down-shifts to take on a supporting role. Probing, in particular, is something that happens so frequently it’s hard to put in just one category. But when the situation shifts from researching to analyzing, exploring to sense-making, and preparation to problem-solving, then foraging goes from being the melody to the bass line.
Tuning the World
Tuning modifies, adjusts, and tweaks the resources we use to understand. Tuning happens whenever you adapt a resource to your own needs. You tune when you snip an article from a magazine, or when you search for a new sweater and filter out the cashmere ones. Tuning means tinkering with information. Foraging treats the world as an environment with resources to aid understanding; the job is to uncover them. Tuning is how you align foraged information with your needs and goals.
There are three primary tuning interactions: cutting, cloning, and filtering. Collecting can also work as turning, not just foraging, and we’ll consider why that’s so. Tuning interactions tend to result in modest, temporary, or reversible changes. Think of a musician preparing to play. Tuning doesn’t change the guitar into a saxophone or a basketball. It brings the guitar into a harmonious state, literally in tune, which is a precursor for making music. Tuning information is similar. The idea is to tweak, alter, and customize a cognitive resource to support the understanding we seek.
Foraging leads to clutter, and cutting reduces clutter by trimming, pruning, or slicing away some parts to focus on other parts. No matter the metaphor, the primary work of cutting is to hack off what’s in the way of the essential, useful, and interesting.
When you think of cutting, you probably think of the cut-and-paste feature on your laptop. But cutting is more than a mechanical operation. When researchers have studied how people use paper documents, for example, they routinely find cutting is pervasive. Cutting is a way of saying “this matters” and “I need this part, but not the rest.” Summarizing more than a decade of research on how people interact with paper and digital materials, the scholar Catherine Marshall concluded that three interactions were pivotal above all else: cutting, collecting, and annotating—the latter an interaction we will encounter under externalizing.
Most of the time, cutting is directed at the information we’re interested in, such as when we clip an article from a magazine or when we grab an image from a web page. In other situations, cutting is the inverse: removing parts we don’t want because they’re hiding the parts we do. When approached this way, cutting is like peeling away the skin of an orange and discarding it to get at the fruit underneath. With paper, we use scissors to literally cut away what we don’t want or clip out what we do. Think back to our light bulb example. We picked up an individual bulb and, upon realizing it could not be dimmed, put it back on the shelf. The bulb won't work, so we physically return it and mentally cut it away, blotting it from memory, even as it remains visible.
A close cousin to cutting is cloning, an interaction so pedestrian that it hardly seems worth mentioning, and so pragmatic that it hardly seems essential to understanding. Yet cloning supports every kind of cognitive work. Whatever name it goes by—copying or duplicating or replicating—cloning lets us take information from one situation and use it in another while leaving the original intact. Scholars of the ancient world, to cite just one example, have come to depend on precise digital clones of antique vases, clay tablets, and illuminated manuscripts. This not only preserves the originals but also allows for new kinds of analysis.
But set aside the scholar, at her desk, scrutinizing ancient runes, and imagine your life without cloning. What would your day be like, just today, if it were impossible to clone information. No photocopies, no cut-and-paste, no downloading files or printing reports. To share that great New York Times piece with a friend, you couldn’t copy the link, and would have to resort to typing it out by hand. But even that involves cloning: you read the link, store each letter in your memory, then type it out—that’s copying, too, from the screen to your brain and back through the keyboard.
Cloning used to be expensive and time-consuming. Digital tools make it cheap, a recurring theme with epistemic actions, not to mention perfect. But cloning also paves the way for generating new variations from information, comparing different scenarios, and testing hypotheses. And among the most transformative inventions in history, who among us would not rank the printing press in the top ten, if not first. We are now so used to cloning, so dependent on its everyday nature, that we can miss how it drives the desire to understand.
Collecting, as a Tuning Action
Now that cutting and cloning is so quick and easy, it accelerates collecting, and that creates clutter. Life is filled with stacks of papers, tweets flagged as favorites, and files tagged #readlater. We previously encountered collecting as a foraging interaction, but we also collect as we tune our information environment. It’s an interaction that is at home in both roles, and it’s not alone. Every interaction can take on dimensions of all four themes, depending on context. Our interaction vocabulary organizes interactions into themes—foraging, tuning, externalizing, and constructing—to capture their primary role, which makes it easier to grasp their similarities and differences. But these themes are not hard categories, and each interaction carries some traits from other themes.
As part of tuning, collecting is largely a choice between piling and filing. Piles are a feature of every home and every office. We have piles of books, magazines, and letters. Piles are the cockroaches of the information world: a structure that you find everywhere and that technology never eliminates. Adding to a pile requires almost no effort, so piles proliferate. Filing, on the other hand, requires more work, first to establish the categories, and then to decide which categories apply. Both paths lead to new arrangements of information and so collecting, because it produces new kinds of structures, is also an interaction for tuning our information landscape.
Where does cloning end—or cutting—and collecting begin? The distinction is clear enough in theory, but when you watch someone working with information, it’s not always obvious how to separate these interactions.
Suppose that your washing machine is broken. The washer starts to rumble but then gives a thunk, stops, and displays an error code. You write down the code along with the model number and head for your laptop. Let’s pause there: Was writing down the model number an example of cloning or collecting? It’s cloning, because the information is now in two places. It’s also collecting since you needed it for typing into Google. You might think we have only dodged the question, but consider the idea that it’s both at the same time.
Think of Interactions Like Atoms
As an analogy, think of each interaction in this vocabulary like an atom. Atoms are the building blocks of the physical universe, yet they rarely exist independently. Most of the time, they bond with other atoms to form molecules. Hydrogen is the most abundant element in the universe, but here on earth, hydrogen gas is quite rare. It exists mostly as a component of hydrocarbons and water. While hydrogen is definitely an atom, unique and distinct from other atoms, it is almost always linked to other atoms.
The interactions in this chapter work in much the same way. We can define, describe, and conceptualize an interaction called collecting. We can list its cognitive benefits and give examples. But when it comes to how collecting happens in the wild, it routinely co-occurs with other interactions. Often, as with the washing machine, they are difficult to tease apart. Observe someone making a photo album from their summer vacation, and you might, now and then, be able to unambiguously observe discrete interactions: some searching, then probing, a bit of collecting, and some rearranging. But other times, these interactions will occur so fused together that it makes little sense to worry about where one ends and another begins.
This is a feature, not a bug, of our vocabulary. In chemistry, the periodic table defines each individual chemical element and this leads to a massive number of molecular combinations. These interactions work the same way. Human interaction is extraordinarily complex and subtle and takes on an immense number of forms. By condensing this infinite variety down to a handful of interactions that combine, and recombine, and take on different forms, we can begin to describe interaction for understanding with a smaller and more manageable set of terms.
Moving forward, let’s consider another example to illustrate how our vocabulary can be used to break down an everyday understanding problem. Imagine you’re at a restaurant, flipping through a menu that runs a dozen pages, trying to decide among hundreds of choices. A salad might be good, but you’re allergic to walnuts, which means finding the ones to avoid (searching) and removing them (cutting), though only in your mind’s eye. Perhaps a burger instead, so you flip through the menu (searching), scan the offerings (searching again), and read the descriptions for anything that sounds especially delicious (probing). You run your finger down the list (searching). The blue cheese burger looks amazing, as you tap your finger on the description (annotating)—we’ll get to this soon enough, but you keep reading (searching and probing) and identify two more burgers for your short list (collecting).
All this interaction, yet note that none of it changes the menu. You can’t physically pull out scissors and cut out any dish with walnuts, or alternately, snip out three different burgers and collect them in a pile. The cognitive work has to rely on memory and brain-bound processes. But you can also see all the foraging and tuning, which leads to possibilities for doing some of the work in the world, as with Tetris.
Tuning simplifies perception and choice. Once we locate the information we need, we often discover that we don’t need all of it. So we cut, clone, and collect, simplifying the world to the information we do need, right now, and setting aside the rest. There is, however, another important interaction for tuning: filtering.
Filtering reduces complexity, like cutting, by reducing visual clutter to expose salient details. But where cutting is directed at a specific element—remove this object from the picture, delete that paragraph—filtering applies to anything that matches specified criteria. Take online shopping. Search for winter gloves, and you get options for hiding the expensive ones and, of those remaining, deciding to show only the black ones from Patagonia and North Face. From there, it’s all about adding a filter, removing a filter, probing for details, and so forth. Filtering is expedient and not strictly necessary. You could sift through all the gloves, one by one, pulling out any that seem promising, either collecting them in brain-bound memory, or more likely, saving them to a shopping cart or opening up a new tab. But filtering speeds this up dramatically: of all the options, just show gloves made of leather, size large, in black, from these brands.
Of the interactions we’ve encountered thus far, filtering is perhaps the most dramatic example of how technology enhances our perceptual powers. In the hardware store, you have to mentally filter out the bulbs that don’t match. There is no way to physically change the shelf to only show 2700K bulbs, with E12 sockets, providing at least 400 lumens. In the online store, we expect this capability. One promise of augmented reality is making this kind of filtering a feature of everyday life. If Google Lens can recognize real-world objects, like a glove, and let us probe for details, the next step is having it recognize all the gloves in a store, provide on-screen filters, and then highlight the most likely matches. In a world overflowing with options, the possibilities are enticing.
Tuning Interactions, Summarized
Tuning interactions cover a wide range of small changes to the world, and we can define them as follows:
Cloning: To create an identical copy of a resource
Collecting: To gather resources for future use
Cutting: To remove unwanted resources
Filtering: To expose, conceal, or transform parts of a resource that have certain characteristic
Tuning happens throughout the understanding process. Quite often, we dismiss these interactions as trivial and insignificant since each change tends to be small. But think of the chef, sipping the soup, then adding a pinch of salt, or the woodworker who sands the chair just a wee bit more in the quest for perfection. The differences may be small, but they’re not insignificant. The benefits of each adjustment takes us, step by small step, closer to understanding.
Externalizing actions are how we move information from our head into the world. We have already explored externalizing earlier in our discussion of external representations. But there, the emphasis was on the representations themselves. Here, we will revisit the topic from the perspective of externalizing.
The point of externalizing, as an interaction, is to add information to the world. In contrast, foraging and tuning deal with the world in situ. As a result, externalizing is a more direct means of shifting cognitive work from brain-based processes to perceptual ones. Consider calendars and day planners. In theory, you could remember to pick up the dry cleaning on Friday afternoon. In practice, it’s easy to forget. Hence the calendar, the day planner, the checklist, and the task management apps.
Externalizing is fundamental to any kind of cognitive work. Writers dump words on the page. Mathematicians scribble equations onto whiteboards. We write on sticky notes, in notebooks, and highlight important passages. We have a vast repertoire of ways to externalize our thoughts. This is not just about the markings we make in the world. It’s why we create these marks that matters: to adjust the balance between internal and external representations.
We can identify three main types of externalizing actions: annotating, generating, and linking. Annotating is adding meta-information to the world, like comments or labels. The new information augments the information that already exists. Generating, on the other hand, happens when we add completely new information to the world. It’s pure creation. The third form of externalizing is linking, which establishes the relationship between different pieces of information.
The term externalizing conveys the characteristic feature of these interactions: information flowing from the head into the world. Other words that could describe this interaction theme include articulating, expressing, or offloading, among others. While these are all in the same spirit, externalizing best captures the underlying cognitive imperative: get this outta my head.
Annotating is how we add information to the world, extra information to augment and amplify what is already there. Think of the red ink on your freshman history essay or the forceful note, scrawled in bold letters, beside a plate of freshly baked cookies, reading “Do not eat! This means YOU!” Annotations are informative marks, clarifying or enhancing or directing attention to other information.
While the forms that annotations can take are vast, the most familiar forms are scribbles, highlights, and other markings. When people annotate for themselves, highlighting passages in a textbook for example, they are “often embarrassed by [the annotations] and what they reveal about their understanding of the text,” even when they also say the annotations helped them understand. The mark may not be useful once on the page, in other words, but the act of creating the mark is essential. It also matters how the marks get made. Annotating works best as a fluid interaction, as effortless as possible, and without diverting your attention. Annotation is a detour, flagging the important or notable, while staying on course to the primary destination.
Paper remains a fabulous annotation medium. It’s simple and flexible, adaptable to almost any situation. At the same time, working on paper means some tasks, such as editing and revising, are tedious. Jane Austen wrote her books longhand, almost a century before the typewriter, and curtailed the drudgery of revision by co-opting an everyday object to speed things along: straight pins. Faced with a full page of script, and no space for anything but the smallest changes, Austen would scribble out her changes on paper scraps, attaching them to the draft with straight pins.
Today, we use other techniques for introducing small, useful bits of information to the world, techniques that are faster and easier. The straight pin has given way to paper clips, sticky notes, and highlighters. We have digital stickers and stamps, and merrily flag items with hearts, stars, and tiny thumbs pointing up or down. Austen could have done the revisions in her head, or written out a new draft, but she developed her annotation strategy because it afforded rapid tinkering. Making things easier reflects a common theme in epistemic actions: reducing the interaction cost. In annotation and other interactions, it’s important for things to be fast, easy, and fluid.
Another way we externalize is by linking, an interaction for connecting bits and pieces together. While the idea can be traced to 1945, the term hyperlink was only coined in the mid 1960s, and hyperlinks didn’t appear in their modern form until the early 1980s. Today, hyperlinks are like oxygen, so familiar that to remember a world without them is like imagining a world without light bulbs—primitive and undesirable. And yet, connecting one piece of information to another, whether we make the link explicit or not, gives us undeniable epistemic benefits. Consider the footnotes on this page, which give you the title of a research paper, cite more research, and so on. The hyperlink didn’t make the World Wide Web possible so much as made the invisible connections and joints of the information world into something visual, perceptual, and tangible. The links have always been there. The hyperlink just made it easier to follow the thread.
Mindmapping tools, such as MindJet or The Brain, are built on linking. It’s more than a feature. It’s the pivotal interaction. While other tools provide ways to externalize thoughts and ideas, mindmapping tools emphasize how to create links between different ideas. But whether it’s about drawing a line between two items on a whiteboard, or building a massive mindmap, linking is how we formally express patterns, relationships, and connections. By putting that connection in the world, the relationship can be referenced visually and no longer needs to be remembered.
Generating is the most generic externalizing action. Annotating and linking create new information, always relative to information already in the world. Generating is different. It introduces new information into the world, on its own, without needing any clear connection to what was previously there.
Imagine finishing the first draft of a report for your boss. You prefer to revise on paper, so you print it out and grab a pen. You read through, striking some words, adding others, and marking where one paragraph should be split. This is annotating since each mark is spawned as a reaction to the original text. In another paragraph, you circle a sentence, then draw a line to the start of the paragraph, indicating where you plan on moving the sentence when you get back to the keyboard. This is linking—you’re defining a relationship. But then, midway through the report, you have an idea, and jot some notes about a new paragraph that will clarify your point. This is generating: adding new information, but without an explicit connection to what is already present.
A key benefit of generating, as with all externalizing actions, concerns memory. Shifting information into the world means not having to keep it in the head. Generating, however, is rooted in the acknowledgment that thoughts and ideas come to mind and, when we recognize them as valuable, we want to do something with them. If you have ever tried meditation or mindfulness techniques, you have first-hand experience with how the brain is always generating ideas, always producing new thoughts, even when you don’t want them. Focusing attention and quieting those thoughts is difficult. Where mindfulness methods are designed to calm the mind and let go of thoughts, generating is how we capture them.
But more than this, generating is also about the benefits of articulating our thoughts. We have all had the experience of having an idea and then, upon writing it down, realizing that it wasn’t quite right, and so we had to revise and elaborate. The act of writing changes ideas. Writing techniques that emphasize stream of consciousness, such as free writing and morning pages, derive their power by acknowledging that ideas in the head are partial. We develop them by writing them down, sketching them out, and speaking them aloud. Externalizing gives us distance and perspective and new ways to explore, shape, and modify the ideas. Generating is a precondition for understanding.
Externalizing Interactions, Summarized
To summarize, we can define our externalizing interactions in this way:
Annotating: To add useful markings and meta-information to a resource
Linking: To establish relationships between resources
Generating: To create new information structures in the world
Young children love to share. They burst with insights and ideas and need, often desperate, to talk about it. Even as adults, we can feel a similar drive to share. Externalizing actions, in a narrower way, describes three major ways in which we satisfy this drive, by moving information into the world where it can find fertile ground. Then we can organize, break it apart, and use it to create new structures. This is the essence of our next and final theme, constructing interactions.
The shape of information matters. Constructing is how we assemble new shapes, using the information at hand to fashion meaningful structure. Constructing fills the gap between what we know and what we want to know, between what we can see and what we want to understand. Any number of words, not just constructing, could describe this process: assembling, forming, building, making, and more. We’ll use constructing because the world offers so many ways to combine information, bit by bit, piece by piece, like LEGO bricks, into new shapes.
Constructing is tuning on a more ambitious scale. Where tuning makes small tweaks, constructing goes further, leading to substantial change, treating resources as building blocks, props, and scaffolds for higher level thinking. When your browser has dozens of tabs, tuning happens when you close the useless one (cutting) or save them all into a folder (collecting). Constructing happens when you work through the tabs, culling the irrelevant items, dividing the rest into groups, giving the groups memorable labels, and introducing other meaningful structure. In this, the final theme in our vocabulary of interactions for understanding, we will discuss six interactions: chunking, composing, collecting—again; it’s versatile—fragmenting, rearranging, and repicturing.
Let’s begin by differentiating between three similar interactions: collecting, chunking, and composing. We have encountered collecting twice before, first for how it helps us forage, and again for how we use it to tune information to our needs. Collecting naturally leads to new structures. We put files into folders, add papers to piles, and move sticky notes into clusters. This often produces a spatial arrangement and, as we saw in the chapter on spatial representations, the way we structure information in space has special powers.
Chunking groups independent yet related information together. It’s the strategy our brains use to extend the severe limits of working memory, similar to when we remember ten-digit phone numbers by creating chunks of three digits, three more digits, and then four digits. We also create chunks in the world. Chunking explains why we can read a sequence of ten digits as a large sum of money when chunked as triples (5,557,283,118), and as a phone number when the same numbers are chunked in a three-three-four pattern (555-728-3118). As an interaction, chunking is the epistemic process of grouping independent yet related information. No matter where the chunked information is found, head or world, it’s all chunking.
Chunking and collecting have strong similarities. They both take different pieces of information and pull them together. But before we tease out the distinction, let’s consider a third interaction. Composing produces a new, separate structure that has its own meaning and purpose. Where collecting and chunking take separate pieces of information and group them together, they don’t introduce deep structure. Composing does. You compose an email. You don’t just collect words from a dictionary or arrange those words into chunks. You assemble the words into a coherent sequence, complete with punctuation and spacing. Composing produces information with a high degree of organization and meaning.
Here’s a simple way to grasp the difference between collecting, chunking, and composing. Suppose that you want to build a house from LEGO bricks. Collecting happens when you forage for useful bricks and set them aside. Chunking happens when you organize the collected bricks into useful groups: these for the walls, those for the roof, and windows in a separate pile. Composing happens when you take all of the bricks and build the house.
Fragmenting runs in the opposite direction, taking information and breaking it apart. You might read books, or watch videos, to understand how a combustion engine works, but those are weak substitutes for disassembling your lawnmower or getting grease under your fingernails. Information is the same. When scholars read scientific papers, for example, they don’t gloss their eyes over the words. They decompose the paper into pieces “to access and manipulate individual components of a document, such as its figures, conclusions, or references.” Breaking information apart, and stitching it together in new ways, is how knowledge is born.
Fragmenting serves two major purposes. The obvious reason is gaining access to the individual pieces—a movie divided into scenes; a book partitioned into chapters; a recipe into ingredients. The other purpose is figuring out how the parts are related, identifying patterns, and grasping the overall structure. Fragmenting is less about breaking and more about creating opportunities for more interaction, bit by bit, piece by piece. Viewed as breaking, fragmenting is the opposite of composing and chunking, and in the opposite sense, fragmenting enables such interactions, and more besides.
Rearranging sounds simple enough: to adjust the spatial position of a resource or the elements within it. Simple, yet with a tremendous boon to understanding. Rearranging is what made it easier to count coins and play Tetris. Or consider a study of Scrabble, which found that players who rearranged their letter tiles found more word combinations, in less time, than players who could only look at the tiles. Putting two letters beside each other was an effective way of triggering new words, and more effective than doing a brain-bound word search.
If arranging is the art of creating order, rearranging is the art of creating meaningful order for ourselves. Libraries are highly organized and librarians are experts in organizing information. The work of librarians has been called “order-making in the large”: structuring vast swaths of information so that other people could find what they needed. When it comes to how we create understanding for ourselves, arranging is key: it’s order-making in the small. While all these interactions, from the beginning of this chapter through the end, play a role in understanding, there is a strong case to be made that rearranging is the essential one—which is why we dedicated an entire chapter to spatial understanding. There is an enormous amount to be gained by taking matters into our own hands, quite literally, and moving things around to explore possibilities, reveal new patterns, and simplify problems.
Repicturing, the last of our fifteen interactions, changes the way that information is represented without changing the information itself. Think back to the numbers game, where the goal was to pick numbers until they summed to fifteen, and how it became vastly easier when we switched from numbers to the 3x3 magic grid. That’s repicturing. The game was a chore when represented as numbers, and a breeze once repictured as the tic-tac-toe grid. Similarly, with the revised diabetes chart, information was neither added nor removed in the visual transformation. The revision was more understandable, thanks to repicturing.
Repicturing, like any of these interactions, isn’t just for diabetes charts and everyday information in everyday life. Richard Feynman, the Nobel Prize winning physicist, developed diagrams that visually describe the interaction between different subatomic particles. These interactions are complex and, even for theoretical physicists, difficult to understand when represented in abstract mathematical equations. Known today as Feynman diagrams, they carry the same information as the equations, but are so much easier to use in their repictured form that they “have revolutionized nearly every aspect of theoretical physics.”
Collaborative work often uses repicturing to create understanding. You’re at work, in a conference room, and the discussion is going nowhere. Then someone goes to the whiteboard, diagrams the options, converts words into pictures, and reframes the debate. Sometimes, changes to the representation are all it takes to trigger insight, agreement, and move the conversation forward. Sometimes, a different way of representing information is the key to understanding something as complicated as the subatomic world. Repicturing can also occur in a nonvisual sense, such as the conceptual frame or narrative that allows someone to see something in a different way.
Constructing Interactions, Summarized
Tuning interactions fiddle at the edges. Constructing is directed at the center, leading to new structures, putting information into new forms. It happens whenever understanding is driven by the need to create, build, and assemble.
Chunking: To group independent yet related resources into a unified structure
Composing: To create a new resource by assembling other resources into a meaningful structure
Fragmenting:To dismantle a resource into its component parts
Rearranging: To alter the position of a resource or the elements within
Repicturing: To convert a resource from one form, or shape, into another
By this point, after four themes and fifteen interactions, you may be wondering how this all works in practice. As our numerous examples suggest, these interactions can be somewhat slippery, overlapping in places, making them tricky to separate. Yet they also describe a wide range of ways that interaction aids understanding, and not just with digital information. So let’s take another step and try using these interactions, the full arsenal, to describe a more complete scenario.…
Stephen is a design leader who focuses on workforce learning and organizational development——and he’s on a mission: to make learning the hard stuff fun, by creating things to think with and spaces for generative play. By creating custom toolkits, providing on-site training, and through The Mighty Minds Club, Stephen helps product teams work through their most difficult situations. As a keynote speaker, he continues to challenge and inspire audiences as he exposes the quirky connections between games, play, learning, interactive visualizations, and other exciting topics. Stephen is most well-known as the man behind the Mental Notes card deck—a tool that product teams can use to apply psychology to interaction design. He is also the author of Seductive Interaction Design, which answers the question: How do we get people to fall in love with our applications? Stephen is also coauthor of the Rosenfeld Media book Figure It Out: Getting From Information to Understanding.
At Normative, a software design studio in Toronto, Karl works on design problems for a world with infinite information and cheap computation. His focus is on how interacting with information is critical to the way people solve problems, make decisions, develop insights, and construct meaning. As a former professor of User Experience Design at Kent State University, Karl was instrumental in building one of the first online UX graduate programs. He has a PhD in library and information science from the University of Western Ontario, where he studied the cognitive benefits of interacting with visualizations. He has taught graduate courses on such topics as information visualization, information architecture, human-information interaction, and user research. Karl has been writing and speaking about information-rich design problems and the digital future since the late 1990s. He has been an invited speaker at events across North America, as well as in Europe and Australia. He is also a founding member of the Information Architecture Institute. Karl is also coauthor of the Rosenfeld Media book Figure It Out: Getting From Information to Understanding.