Anyone whose work and recreation takes place primarily on a computer will find that their daily repertoire of gestures–or the motions made by their hand and body to communicate and accomplish tasks–has shrunk considerably from what it would have been in the pre-digital age. A near-uniform set of hand gestures–point, click, scroll, swipe, tap, and zoom–are used to accomplish virtually every task across every device with a screen in the 2D virtual world (Beugnet 2020). The acts of wandering, searching, finding, choosing, examining, making, communing, and spectating–actions that once required a variety of different motions, gestures, and touches–have been condensed to what is allowable by a keyboard, trackpad, mouse, and screen.
An example of the flattening of gesture is the change that has taken place in film editing techniques. In the days of analog film, editing meant physically cutting, joining, winding and unwinding rolls of film stock. When watching footage of analog film editors at work, one can see that each editor has their own way of handling the medium and their tools, manifested in subtle ways: a unique bend to the body or a characteristic flick of the hands (Manufacturing Intellect 2017). In contrast, in the 21st century a digital film can be edited solely with a single computer. Within the span of a decade, everyone’s gestures on these apparatuses have become nearly identical, whether they are cutting, joining, exporting the footage, or working with audio. No one’s fingers will physically touch the image at any point in a digital project.
Within Natural User Interface (NUI) design practice, the flattening and condensing of gesture into a handful of streamlined motions is intentional. NUI is intended to design a universal set of “natural” or “playful” interactions between a so-called standard human hand and a computer (Cooley 2014, 28). However, despite NUI design’s self-characterization as natural, many NUI gestures (tapping, scrolling, swiping, pinching, and prolonged sitting or looking down) are actually quite unnatural, even for a so-called standard (abled, young, financially stable) human hand, to the point where repetition of the gestures over time will cause musculoskeletal disorders including hand and arm strain (Griffith, Mackey, and Adamson 2007), computer vision problems (Blehm et al. 2005), and diseases related to a sedentary lifestyle.
So how natural is NUI anyway? Its gestures do not have any one-to-one relationship with any gesture in the real world. In the real (physical) world, one does not click on a stranger’s face to find out their personal history; one does not scroll through nature and cityscapes as if they were webpages; one does not make decisions by swiping through them. The relationship between digital gestures and their function is not concrete, but abstract. One may mentally “wander” (scroll, swipe) the digital world and then “choose” (point, click) to go down a particular rabbit hole of information or interactions. Inherent in the language of digital gestures are acts of servitude: the ringing of a bell that calls an attendant, or the rubbing of a lamp that summons a genie; Siri and Alexa’s (feminine-gendered) presences are the digital helpmeets, or mechanical brides, that facilitate our desires. Drawing on feminist science and technology scholarship, Wagman and Parks (2021) posit machines as “social machines” that replicate and perpetuate social dynamics, a problematic phenomenon when the underlying assumptions treat place machines in the categories of tools, human companions, animals or creatures, or slaves (10).
Cultivating a wide repertoire of gestures is often considered recreational, at least for those normally reliant on computers. A digital worker can learn, say, to rock climb, play the banjo, or throw pottery, but these gestures are wholly extraneous to the physical requirements of their work. It would be very inefficient to conduct their paid labor with extraneous gestures–for example, doing something as silly as waving their hands wildly or tap dancing every time they tap the keyboard. Those extraneous gestures would not fit within the system of capital that governs the nature of their work and dictates their physical workflow. NUI design, therefore, is not simply based on the pursuit of naturalism or comfort, but is also driven by efficiency–even at the expense of healthy bodies and minds.
To be sure, many people who do not primarily rely on computers for their work still require a specialized and unique set of gestures for their craft. For example, an experienced construction worker still should know, on a physical level, the best way to excavate different types of terrain with a specialized set of tools, and they may share this knowledge with others of their profession. Although these gestures may become mundane to that particular person over time, they are still specialized motions used by a small number of people, to accomplish specific tasks–in other words, they remain gestures rooted in a craft. But anyone who uses a phone, computer, or any digital machine will experience, to varying degrees, the same gestures, and the same flattening of gestures. Even a construction worker likely owns a smartphone: according to the most recent U.S. census data from a 2018 survey, 92 percent of households in the United States had at least one type of computer; smartphones were present in 84 percent of households, while 78 percent of households owned a desktop or laptop (Martin 2021).
Undeniably, many NUI gestures that govern human-computer interaction can feel novel and playful–this is why technology companies, like Apple, invite so many consumers to touch, play with, and marvel at their machines in brick-and-mortar stores. It can feel miraculous to commune with the computer, like a puppeteer with invisible strings, and feel as though it is responding to you through gestures alone. It is pleasant to experience a machine executing what its designers intended. And over time, a user feels an affinity for their machine; although it is identical to millions of other devices, repeated interactions with the object engenders familiarity and affection (Frykman and Frykman 2016).
Through repetition, gestures become mundane and integrated into the everyday as if there were no other alternative, in the same way that the “thumbs up” sign, descended from obscure origins, now feels like it has always been a part of humanity’s gestural lexicon (Fabry 2017). Digital gestures are no different: before the “swipe to pass,” however did we move from one choice to the next? When did the concept of “pointing” become so important to making decisions and navigating in digital space? Instead of aimless wandering, moving from one virtual space to another requires a concrete decision-making process before the movement even happens.
Human evolutionary anthropologists speculate that the use of tools, particularly those that required the dexterous use of opposable thumbs (Kivell 2021), shaped the ability and behavior of early humans. How we interact with digital tools will shape human consciousness and development just as profoundly. A funny turn of events: an ever more thumb-centric culture seems to be at hand. As ancient tools drove early humans to rely on the thumb’s maneuverability, so did the invention of the typewriter keyboard introduce humans to typing with fingers and thumb. Now, digital media interfaces require the thumb to execute (rather than simply support) tasks like typing and swiping and other functions (Ramati 2021).
NUI gestures are here to stay. They may evolve with the development of virtual reality (VR), which generally calls into action more appendages, for example, headsets and motion trackers that track movements of the arms, head, and eyes. On a surface level, it seems that VR will increase the repertoire of gestures for human-machine interaction, and this may be true. But what VR gestures currently share with commonplace screen-based gestures is that each gesture triggers one (or a handful of) prescribed responses. In a virtual world–say, on the popular virtual world platform VRChat–there are limited and standardized ways to pick up a pencil, hit a tennis ball, or shake hands with another person/avatar. In the non-VR world, humans have personalized handwriting, signature ways of spinning a tennis ball across the court, and unique handshakes–distinctions that are lost when they enter the virtual realm.
Perhaps human-machine interfaces will soon become advanced enough to replicate the individual traces of each human body and its unique gestures. Until then, cost-efficiency and convention continue to point towards the generalization of gestures, especially in regards to touch-input based tools. In his essay “The Question Concerning Technology,” Heidegger writes that enframing technology reveals the mystery–and power and danger–of its essence. It is worth questioning whether the technology that governs contemporary gestures is natural or inevitable? Do we want it? Should we disrupt the path that has been determined by the limited set of gestures allowable by current NUI, and if so, how? Writing of this path, Mizuko Ito posits that handheld devices generate a new kind of consciousness, by encouraging new habits of “sharing,” a “persistent alertness,” or a “new kind of personal awareness” (76) that is “[a]ctive always unfolding, in-relation assumes the rhythm and time of a present that is always becoming, never still” (43). This persistent personal awareness climaxes in small actions, i.e., “the impulse to text, image, or post an update” (76). From an embryonic quagmire of urges, emerges the pre-circumscribed gestures to carry them out. A swathe of humanity has come to be governed by these fragmented urges and gestures, acting on impulses to ceaselessly update, browse, and purchase. The ease and homogeneity of gestures gives rise to the ease and homogeneity of modern-day consumption of images, services, and goods.
It is difficult to swim against this stream. I am typing this essay on a computer, repeatedly scrolling up and down and zooming in and out to better see the text. On other tabs, I am tempted to browse social media and do online shopping. It is a distraction from the labor of sustained thought–it prevents the labor of sustained thought. And yet (beyond writing the manuscript by hand), there is no other way. These are the most efficient tools available to me. In an attempt to swim at least in a crosscurrent, there is a double task to be completed: one, to point out the homogenization of human gestures as human-machine interaction becomes commonplace; and two, to identify gestures that remain unique and/or uncommon, whether those are the dying gestures of a trade, or gestures that protest against oppression around the world.