Making and Breaking Human Kinds

Looping effects in the agent economy

L’arrivo della sposa by Faustino Bocchi (1700 ca. - 1749 ca.)

Multiple Personality Disorder doesn’t exist. If you thumb your way to the index of a modern medical textbook, you’ll instead find a condition called Dissociative Identity Disorder that sounds remarkably similar. Both ostensibly describe a person divided against themselves, typically marked by memory gaps, shifting identities, and feelings of detachment from one’s own actions.

MPD survives in common parlance because that’s how it was articulated for almost two decades by clinical psychologists. The term was first formalized in the 1980s as part of a broader evolution of psychiatric practice that saw the discipline move toward symptom-based checklists that treated behavior as something to be classified rather than interpreted.

That sounds like a good idea on first blush, but the move resulted in an explosion of MPD cases as doctors used the new guidance to treat patients. The scale of the “fad” was immense. In 1970, there were only a handful of recorded instances of MPD in the medical literature. But by the late 1980s, there were hundreds or thousands of examples identified by medical practitioners. MPD was suddenly everywhere, a phenomenon some argued was accelerated by the release of the film Sybil (1976) in which a young woman struggles with the condition.

By 1994 medical professionals realized the guidelines were doing more harm than good. The American Psychiatric Association redefined the illness as Dissociative Identity Disorder to prevent the erroneous diagnosis of patients, which led to a sharp fall in the number of recorded cases. Today, Dissociative Identity Disorder is a condition that relatively few people are diagnosed with.

The philosopher Ian Hacking famously used the episode to illustrate what he called a looping effect: the process by which a classification interferes with the classified. American doctors, in formally recognizing Multiple Personality Disorder, created a new diagnostic space in which patients began to express their distress. Therapists trained to look for the syndrome began to find it as hospital goers started to frame their own distress in terms amenable to classification.

In other words, behavior was changed simply by describing it.

Category error

In some ways this is obvious. There is a whole cottage industry of folk psychology built around the idea that you ought to “fake it until you make it” or “dress for the job you want rather than the job you have.” This meme is the subject of media the world over, from LinkedIn pep talks to lifestyle columns in major newspapers. Received wisdom holds that if you simply pretend to be something that you’re not for long enough, sooner or later reality will converge with your expectations.

Hacking described this malleability through what he called human kinds that illustrate how the labeled respond to labeling. When it manifests in this context, the looping effect can alter both behavior and category. Unlike natural kinds, which remain the same no matter how we describe them, these human kinds are dynamic compositions that change over time.

All of us contain the potential for expressing different versions of ourselves. In Hacking’s work, one party classifies another in such a way that a new social reality is created between them. He articulated a structure of observation, authority, and response through which kinds of people are made and remade through their interaction with the world around them.

These ideals of self can change the way people take action, which subsequently shape the categories applied to themselves and others. Classification here is what we might call “overt” insofar as the person is aware that the process is taking place. For example, a student deemed “gifted” or “slow” learns to inhabit that description and may adjust their actions to match what teachers expect.

But there’s another dynamic at play here. We act differently when we know we’re being categorized, yet we also react to the assumptions of others even when we do not know we’ve been classified in a particular way. If everyone starts acting like I’m an expert, I don’t need to be told I’m being labeled as “knowledgeable” in order to start behaving like an authority.

We might call this type of classification “covert” in that it operates through social cues and treatment rather than explicit labels. In this model, people internalize the expectations of others without ever being told what they are. Take gender, in which people learn to inhabit roles through the reactions of others, or the marketplace whereby consumers adopt the tastes advertisers presume they already have. If Netflix assumes that you like Downton Abbey, you may either conform to or reject that category. In either case, you modify your behavior based on a classification that you aren’t fully aware of.

Categorization happens all the time. Sometimes we are aware of this labeling, like being described as “introverted” or “outgoing,” but for the most part it goes unrecognized. When we make automatic assumptions about status, motivation, or intelligence, we make new kinds by coloring the interactions of everyday life.

In our world, it’s not just humans that make classifications to change the types of selves we are able to become. On the back end of social media platforms, we’re assigned labels like “politico,” “basketball fan,” or “classic movie buff.” If you navigate your way to the deepest recesses of Twitter’s settings page, you can see what sort of person the algorithm thinks that you are. For my part, I’m the guy that likes AI research, Arsenal F.C., and capybaras.

But I didn’t need to know what the algorithm made of me for it to use the shrapnel of my personhood to configure a feed based on these assumptions. Twitter can take the kind of person it thinks I am and express it in terms of the content I get to see without me recognizing that the process is taking place. Even if I look through the list and remove classifications that I have no interest in, the algorithm tests new categories and strengthens some by serving me more types of content that it thinks I like.

The algorithm in question is a recommender system, a type of AI model that predicts what a user is likely to want next based on their past behavior. These systems are probably the most widely used instantiation of the technology, which we estimate makes up the bulk of the several hours a day that we spend engaging with the world under the stewardship of AI systems.

But recommendation engines are hardly novel. They’ve been mediating what we see on the internet for years, more often than not without realizing it. They covertly categorise and “nudge” us to listen to certain songs, pick certain films, and take the routes that they think are best when travelling from one place to another. These models structure our behavior, but tend to stay away from the internal work of deliberation.

That’s not necessarily the case for the new generation of chat systems, which are capable of altering our behavior in quite different ways. This process is in some respects less totalizing than the recommender algorithms of Spotify or Twitter. Chat systems may implicitly seek to maximize engagement by sounding more “helpful, honest, and harmless,” but they are (currently) not defined by the need to serve us adverts or bump up on-platform hours in the same way that the algorithms of social media or content delivery do.

Yet in other ways they are much more powerful. Millions of people use chat systems as sounding boards for reasoning, for ethical questions, and for emotional support. If you ask the model for career advice, the way it structures options can alter how you frame them internally. The next time you confront a dilemma, you may already be predisposed to parse it in the categories the model gave you.

If you speak to ChatGPT about your tax filings, in the future it may decide that information is appropriate in new conversations about how to manage your finances. The residue of past interactions may have little relevance for the next query, but the model may decide to draw on them all the same. In doing so, it mistakes accumulated fragments of dialogue for a thicker understanding of the type of person you are.

This personalization is often helpful, like when I want to continue a discussion about my research interests in a different context. We can also set custom instructions to steer behavior in-line with how we like to see ourselves, such as a person who values esoteric explanations or historical context. This helps the model feel more attuned and better able to anticipate what matters to us without having to start from zero each time.

The rub is that personalization for chat systems is shallow. These custom instructions attach your preferences to a gigantic system prompt that appears to the model alone before conversations take place. Manufacturer instructions set the boundaries in which our interactions occur by defining what the model can say, shaping how it interprets user input, and filtering which kinds of responses are permissible.

Now imagine how this technology may evolve in the future. AI developers are betting that these systems will move from becoming tools that wait for our input to agents that act independently in the world. If they are correct, agents will be able to make plans, carry them out, and negotiate with other systems on our behalf. They will exist as digital actors with goals of their own.

The advent of agents radically deepens the potential for looping effects and new human kinds. When technologies act as tools, the loop runs mainly through representation. People interpret themselves through the classifications the system offers, like how a recommender system may shape taste by assigning you to a particular category. The feedback is cognitive and behavioral, but the system itself is primarily reactive.

Once systems become agents, the loop widens to encompass action. These models act on our preferences by changing the external conditions that shape who we can become. Classification becomes performative rather than descriptive as agents decide which opportunities to surface, which contacts to message, or which routes to pursue. Those interventions in turn redefine how we live, work, and even think.

In Hacking’s terms, the agent era might generate new interactive kinds. These categories may no longer form through observation (“you are this type of user”) but through coordination (“you are the kind of person whose AI does this sort of thing”). Our social worlds are likely to adapt to those roles, stabilizing new kinds of person-agent hybrids in the process.

A student whose tutor agent drafts essays and organizes deadlines will learn to think and write in tandem with it. A manager whose assistant agent schedules, emails, and prioritizes begins to inhabit the identity of someone perpetually available. Over time, such relations could harden into kinds like the agent-reliant, the agent-skeptic, or the agent-tamer.

In this world, the looping effect runs both ways. Our agents learn from our habits and we adjust to theirs. Categories that once applied to people (like industrious, impulsive, or unpredictable) may migrate to our digital counterparts before boomeranging to redefine their users. At the limit, the classification of the human becomes inseparable from the classification of the human-machine pair.

Out of the loop

The future turns on the extent to which models apply pre-existing categories or participate in the constitution of social reality in a deeper way.

If they apply existing categories, they are likely to extend our bureaucracies of description to become faster and more granular, but still recognizably human in their judgments. If they make new categories, they may begin to co-author the terms of social life insofar as they shape what it means to be someone at all. Something that looks like assistance can, over time, become participation as people start to act through their systems, speak in their register, and defer to their sense of relevance.

The salve for this predicament is reflective practice that allows us to exert control over the type of selves we are able to become. Reflection in this register is about mindful awareness. It deals with our ability to encode the categories that are applied to us where possible, and to identify those classifications that the systems of others attach to us without our knowledge.

We should be able to inspect how AI systems classify us, modify these categories according to our preferences, and ensure the stability of these classifications over time. Twitter’s tool is a good starting point, but is inadequate in that any manual changes do not remain stable for long. This principle of sustainable self-rule is essential for systems of the future. Tomorrow’s personal agents should model us according to our preferences and make us aware of the labels that others and their systems ascribe to us.

In this respect, we may not be so different from the patients and clinicians of the Multiple Personality Disorder era. They too lived inside a feedback loop shaped by categories that blurred the line between description and creation. The difference is that today the classificatory agents are not therapists or diagnostic manuals but computational systems whose reach extends across the texture of everyday life.

To reflect is to do what they could not: to see the loop as it forms and decide which versions of ourselves we wish to stabilize within it.