The fourth-century Greek ascetic Evagrius Ponticus described how monks would be beset by a “noonday demon” between the hours of 10 a.m. and 2 p.m. This demon would make the monk feel the drudgery of his work so intensely it was as though the sun had slowed down. The monk would “hate the place and his way of life and his manual work” and “desire other places where he can easily find all that he needs and practice an easier, more convenient craft.”

This demon was called acedia, which derives from the Greek term for a “lack of care.” These days acedia is more commonly rendered as “sloth,” with its connotations, but the ancients and subsequent scholastic writers conceived of it more richly.

Saint Thomas Aquinas saw acedia as a sadness or disgust that weighs down the mind, until it wants to do nothing, with important spiritual duties feeling burdensome. As “no man can be a long time in company with what is painful,” it drives the monk to indulge in curiosity, idle talk, and to seek out distraction. Eventually, it can cause the sufferer to detest spiritual goods.

The monks were describing an aversion to God that needed to be warded off through obedience and perseverance. A similar modern acedia, however, could apply to any good that can only be reached through slow, determined work. Cheap intelligence poses a challenge to our longstanding model of formation, in which judgment is acquired through perseverance and apprenticeship. AI can increasingly offer people an approximation of the fruits of determined work, without them having to endure the hard work.

A common response from the optimists is to argue that AI will help us to automate the “bad work” (drudgery), while we can proactively choose to keep “the good work” (valuable formation). Microsoft CEO Satya Nadella has argued that AI will “remove the drudgery of work and unleash creativity,” while government and corporate guidance routinely distinguishes between routine work that can be automated and tasks that involve higher-order judgment that people should retain.

But what if this distinction is harder to identify than we think?

Subscribe now

Keep on drudging

Over the course of history, machines have relieved people of enormous quantities of backbreaking, miserable work. But the burdens technology has unshackled us from rarely consisted of pure, unalloyed drudgery. This work is often the means through which we acquire skills, habits, and attachments.

The philosopher Albert Borgmann described the “device paradigm.” This is when a technological device is able to separate a desired good from the machinery we use to procure it. This commoditizes the good by making it instantaneous, safe, and easy to obtain.

He used the example of the family stove being replaced by central heating. Superficially, this was one means of heating a home being swapped out for a more convenient one. But the old stove also “provided for the entire family a regular and bodily engagement with the rhythm of the seasons that was woven together of the threat of the cold and the solace of warmth, the smell of the wood smoke, the exertion of sawing and carrying, the teaching of skills, and the fidelity to daily tasks.”

Borgmann is not arguing that chopping wood happens to have some positive educational externalities. Instead, his point is that to manage the stove competently, people had to pay attention and learn how to respond to the world around them – it required “intensive and refined world engagement.” Central heating, by contrast, “procures mere warmth” while reducing the demands previously made on our skill, strength, and attention.

Similarly, you can enjoy music by gathering a group of friends with instruments or you can switch on a stereo. The stereo system allows you to listen to a huge variety of genres, played by the most talented musicians, at any time. Musical practice demands “regular and skillful engagement of body and mind” and embodies traditions of craft, method, and composition. It challenges the player to develop capacities in response to them.

Despite his nostalgic tone, Borgmann is not exhorting us to tear out our boilers or smash up our stereos. Sometimes the loss of certain skills in exchange for convenience or access is a good trade. If I want to hear a beautiful performance of a Mozart string quartet, I am glad that I do not have to teach myself the violin to virtuoso standard and conscript three of my friends. Instead, he is arguing that technological disburdenment can remove forms of engagement as well as drudgery.

Learning your scales might be boring, but it teaches you about the fundamentals of musical composition and the capabilities of an instrument. This produces a relationship with an instrument and music that cannot be reduced to the acquisition of a technical skill. The effort is not an inefficient means of obtaining an independently available good, but part of the way the activity acquires meaning in our lives.

Skill and practice

It’s also hard for an outsider to recognize the goods of a particular practice. In After Virtue, Alasdair MacIntyre gives the example of trying to teach an intelligent child how to play chess.

At first, they will only play chess if they are paid in candy, with the promise of more candy if they win. Under these circumstances, it is completely rational for them to cheat. But MacIntyre imagines that through playing the game, they may learn that there are goods internal to chess, such as analytical skill, strategic imagination, and competitive intensity. The child would no longer want to cheat, because they would be denying themselves these goods.

MacIntyre argues that these internal goods can only be identified through participation in the practice. That makes it harder to inspect the burdens of a practice from the outside and decide confidently which are dispensable. This is because a practice is “never just a set of technical skills,” even when those skills are mobilized toward a unified purpose.

What distinguishes a practice is partly that participation changes the practitioner’s conception of the ends their skills are serving. A technical skill might be replaceable as a means of achieving an output, but still play a role in initiating someone into a larger practice. On one level, the internal good of painting is producing good paintings after (hopefully) obtaining the ability to do so through some combination of discipline, perseverance, and talent.

But another lies in living the life of a painter. To enter into a practice is “to accept the authority of those standards and the inadequacy of my own performance as judged by them. It is to subject my own attitudes, choices, preferences and tastes to the standards which currently and partially define the practice.” While those standards can be criticized, intelligent criticism comes from within a tradition of achievement, rather than from a novice believing that their present preferences are somehow authoritative.

Potemkin Institutions

Highly capable AI systems have much more radical implications than any of Borgmann’s devices. When someone owns a stereo instead of being able to play an instrument, people are unlikely to mistake them for a talented musician. When I step into a warm house, I don’t thank the host for tending to the hearth.

Historically, if a student produced a good essay, it was possible to at least infer something about their ability. The link between output and ability, while not completely severed, is weakening. As models improve, it is likely that it will weaken further. This is a problem when institutions use these outputs to allocate external goods, such as grades, qualifications, money, and professional status.

The real issue here is not that it will become easy to cheat. It’s that many institutions will actively embrace the shortcut. To most firms, time saved and increased output are legible, good outcomes. The hypothetical lost skill and judgment are often not visible until it is already too late.

A junior lawyer who is tasked with digging up relevant cases for an opinion is exercising a technical skill, rather than the practice of law. If searching for these cases is simply an inefficient way of locating an authority, then it could be automated with little consequence. But if repeatedly searching is a way of learning how different legal doctrines fit together and what a good legal argument looks like, its automation may well preserve the output while eroding the junior lawyer’s understanding of the practice of law. As the mathematician Terence Tao has argued: “AI tools are like taking a helicopter to drop you off at the site. You miss all the benefits of the journey itself. You just get right to the destination, which actually was only just a part of the value of solving these problems.”

A lazy student or apprentice historically might have conceded that a standard is real, but that they were failing to meet it. When a machine can conveniently reach the standard for you, it can intensify acedia. When Evagrius’s monk is afflicted by the noonday demon, he does not simply lose the desire to work, but comes to hate his way of life.

Our junior lawyer is now toiling away knowing that there’s a cheap piece of SaaS that could produce the relevant authorities in a matter of seconds. The more completely a machine can produce the visible outputs of a practice, the more irritating and hateful the standards that require real effort on the part of the practitioner. If their senior colleague tells them their resulting argument is formally defensible, but nevertheless bad, why should they take their opinion seriously if an AI tool doesn’t detect a difference? What if “judgment” and “craft” are the arbitrary preferences of an elite keen to preserve its status?

John Cassian, another fourth-century ascetic, described how acedia would lead the monk to engage in dishonest rationalizations about why he needed to leave. Maybe he ought to visit the sick or attend to a neglected religious woman. This would be a better use of his time than “staying uselessly and with no profit in his cell.” He might even selectively pick verses from Scripture to justify excluding himself from manual work. Similarly, Evagrius’s monk tells himself that he can leave his cell, because God can be worshipped anywhere.

On one level our junior lawyer does have a point – Tao’s helicopter has its advantages. By speeding a journey along, you might visit ten places in the time you would have spent visiting one. We can test more arguments, explore more hypotheses, or attempt more projects. In many cases, it may even lead to better outcomes.

But this radical increase in agency might come at the cost of understanding. This is a difficult thing to acquire and it’s easy to fool yourself into thinking you have it. The difficulty arises when there is no independent test of whether the result is good. One AI system can critique the output of another, but both are drawing on the similar sets of inherited patterns and assumptions. Understanding is what lets practitioners judge plausible answers, their underlying assumptions, or when a new approach is needed. This is why the separation between practice and outcome is inherently unstable. You may be able to fool most of the people most of the time, but eventually, the edge cases will catch up with you.

Of course, we may not even notice this pattern for a long time. As MacIntyre concedes, “institutions and technical skills serving unified purposes might well continue to flourish.” Money, status, promotions, or qualifications may continue to slosh around. But once institutions lose the ability to judge error, distinguish excellence from imitation, train successors, or push a practice beyond its existing standards, then we risk being stuck with the simulacrum. Institutions can live on inherited standards and machine-assisted outputs for a long time, while spending down the human capital required to preserve and renew standards.

Living under the yoke

John Cassian wrote about Abbot Paul, a figure who managed to overcome acedia. He lived in a cave, a seven days’ journey from the nearest town, with enough dates and produce to support himself. He gathered palm leaves and fashioned them into baskets, but never sold them. At the end of the year, he would burn all his palms.

He did not work for any kind of external good, caring only for “purifying his heart, and strengthening his thoughts, and persisting in his cell,” and above all for “gaining a victory over acedia and driving it away.”

Unfortunately, most of us lack the mental fortitude of Abbot Paul. As we know from the triumph of first television and then short-form video over the book, it is hard to resist the path of least resistance. We don’t have to be intellectually persuaded that TikTok is better for us than sustained reading; it need only be easier in the moments our concentration falters. AI brings that asymmetry to writing, research, study, coding, and much else.

There will never be an indisputable taxonomy of formative and nonformative work, which means we should approach how we change work with profound humility. Sometimes there will be no point doing things the hard way. But if a task was historically part of a route that formed competent practitioners, we should be cautious about removing it until we have a good reason to believe that the same capabilities can be acquired some other way.

The alternative is not to embrace a life of pointless toil. We should remember the instructions on the daily labor of monks in the Rule of Saint Benedict, which advises that “all things should be done with moderation… for the sake of the faint-hearted.”

Human life and our character is shaped by our encounter with the world. This includes the difficulties and struggles we choose to embrace, as well as the ones we face through fortune or inheritance. For Evagrius, the opposite of acedia was hypomonē, which literally means “remaining under” or steadfastness. This may not be easy, but it’s preferable to a textureless life of want-satisfaction, all watched over by machines of loving grace.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

Today’s early agentic economy, and more broadly any decentralized network of agents, is constrained by trust. Humans have reputations to maintain, can’t clone themselves with a different identity, and can reliably face consequences if they break the rules.

Machines have none of that.

Imagine two AI agents, working on behalf of different businesses, that want to collaborate on a project. They can decide who does what and how to split the revenue, but the hard part is ensuring that they can trust each other. After all, what’s stopping one side from simply taking the money and running away?

Without this basic level of trust, it’s hard to imagine AI agents being adopted at scale for economically valuable work.

On the internet, we normally resolve these challenges by placing a trusted platform in the middle of the transaction that is capable of enforcing the rules. But an intermediary is also a gatekeeper. It can see or control information that the parties may wish to keep private, dictate the terms of how they interact with each other, and potentially extract rents or exclude participants.

How can we have trust at scale between agents without relying on central gatekeepers? This is what we have been trying to resolve with dovetail, a project of the Institute for Decentralized AI. dovetail is an open-source system for designing and verifying the protocols that AI agents use to interact.

The project is part of IDAI’s wider mission to create the coordination and security infrastructure needed to build the heterogeneous, distributed AI systems that we believe are essential for protecting human autonomy.

Subscribe now

Decentralization and why we need it

My interest in decentralization comes from a concern about how AI will distribute power. In the context of AI risk, we normally talk about access-driven risks that arise when more actors can use a dangerous capability. We focus much less on the gap-driven risks that stem from some actors becoming more capable than others.

If only a handful of companies can run the most capable models, then they risk becoming the gateways through which everyone else accesses AI, giving them more power over the people and institutions that depend on them. If your data, workflows or agents are tied to one provider, switching becomes harder and that provider gains leverage over price, access, and terms.

But decentralization faces a basic disadvantage. Systems under one owner can pool information, share infrastructure, and exploit economies of scale. This is why we focus so much on agentic economies: while it’s easy to centrally coordinate agents within a single company, it’s much harder when they represent individuals or firms with distinct (or even conflicting) interests.

Before dovetail, we worked on automated mechanism design. We looked at whether agents that did not trust one another could design, negotiate, and follow protocols grounded in game theory. We built a system where two agents could negotiate a contract, expressed in code, and then deploy it as a smart contract. This would mean that the software enforced the terms, so no trust was required between the two parties.

Mechanism design doesn’t need to assume that the participants will behave cooperatively; instead, you can change the rules so that the intended behavior is in everyone’s interest. For example, if there is a sufficiently high penalty for walking away, it becomes the rational choice for participants to continue working together, even if they don’t trust one another.

But a contract between two agents is a relatively simple, self-contained interaction. As an agentic economy scales, it will run on complex chains of interdependent decisions. For example, agents may need to allocate work, subcontract, run an auction, share revenue, vote, exchange information conditionally, or coordinate among several parties. The scope of the required protocols is much bigger, and it’s likely that for each interaction between a group of agents, we will need ad-hoc protocols to ensure that everyone is collaborating effectively. This is the goal of dovetail: arbitrary, machine-speed contracting and coordination for any task.

Building blocks

dovetail is an attempt to build a general system for designing protocols that can support many different types of interaction, alongside a formal verification layer.

dovetail starts from the properties that agents care about. For example, these might be payment terms, confidentiality requirements, or project timelines. We are aiming to formalize a core library of roughly a dozen properties, drawn from a blend of distributed systems, game theory, and information security.

You can think of dovetail like a compiler. Compilers translate a high-level description into lower-level code while preserving its meaning. dovetail does something similar with interaction rules. To compile a protocol with the right properties, dovetail uses an LLM to propose a candidate and a proof that the protocol meets the required conditions. Those proofs are written in Lean, a programming language and proof assistant designed to express mathematical claims precisely and check proofs mechanically. If Lean rejects the proof, the system tries again.

We expect many interactions to fall into recurring categories such as sales revenue sharing, auctions, voting, and task routing. Increasingly, we will be able to rely on a growing repository of verified protocols that can serve most requests directly. This will mean we can reserve novel synthesis for cases the library doesn’t cover.

Our ARIA grant will support us as we formalize the first protocol properties, build verified implementations, start the property library, and create the generation engine. We will work towards more and more complex scenarios, including things like multiple parties or mixed motives.

We’re also keen to discover where this approach fails. For example, we will run adversarial games where agents actively try to violate or exploit the protocols.

We plan to release the main outputs of our project openly under MIT license, including the formalized properties, the library, the generation engine and proving repository, connectors for different agent frameworks, and any benchmarks that we create.

We also want to know whether dovetail can produce protocols that are not just formally correct but actually useful. We intend to compare dovetail’s output to the strongest human-designed solutions to problems such as auction design. We can assess whether it satisfies every required property, how complex it is, and the amount of effort was required to produce it.

At our most ambitious, we’re keen to study protocols as a proxy for institutions. We may discover that agents can devise better arrangements for working together than the ones we currently hardcode into firms, marketplaces, or governance systems. If protocol design can be automated and tested at scale, we may be able to search a much larger space of institutional forms, including ones that coordinate more effectively while preserving more room for independent action.

What’s next

IDAI began its life as a project of the Cosmos Institute. Through our partnership with Cosmos, we’ve been able to establish a research program around decentralization and multi-agent security, launch our fellowship program, and make clear contributions to the field of safe decentralized AI, with publications at ICML, ICLR, and NeurIPS. But my ambition was always to turn IDAI into a permanent organization, with its own team and research agenda.

As a result of the scale we’ve reached, we’re now ready to take the next step in our journey, so we are currently in the process of establishing IDAI as an independent nonprofit. IDAI will very much remain a part of the Cosmos ecosystem, as we both share the mission to enable decentralization and human autonomy in a world shaped by AI.

So far, we’ve worked alongside Google, Anthropic, Microsoft, OpenAI, Stanford University, and ARIA. If you run a research group or institute interested in collaborating on dovetail or decentralized AI more broadly, you can reach us at collaborations@decentralized-ai.org.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

As a professor at the University of Texas at Austin, I sense in my students, as I’m sure many other professors sense in theirs, a growing anxiety about AI. Although students sometimes express optimism about AI’s potential to accelerate innovation, more vivid and striking is their fear that AI will make it harder for them to live vigorous, engaged, and autonomous lives. One of my challenges is to convince them that the kind of books I teach, the Great Books, can still speak to them in the age of AI.

This isn’t easy. The Great Books—books like Homer’s Odyssey, Plato’s Republic, Machiavelli’s Prince, and Shakespeare’s plays—were written in times very different from our own, for people who didn’t have a world of information at their fingertips, and in societies that faced challenges that are barely recognizable to us.

It’s not entirely unreasonable for students to suspect that the Great Books of the past no longer have much to say to us and that they’ve become historical relics that belong in dusty museums rather than in the hands of contemporary readers. But it would be a terrible mistake to draw this conclusion. In my view, the Great Books should be regarded as more precious than ever, both for the habits of mind they cultivate and for the questions they raise.

Subscribe now

Entering ancient courts

It is true—and some might take it as evidence that Great Books are no longer relevant—that reading the Great Books is becoming increasingly counter-cultural, at least in the sense that it involves an activity that runs ever more contrary to the mental habits of our time. We live in an age of maximum information but minimum reflection. That, however, is a reason one should read the Great Books. In his 1978 Harvard address, Aleksandr Solzhenitsyn described “hastiness and superficiality” as “the psychic diseases of the 20^th century.” He wasn’t wrong, but he hadn’t seen anything yet.

TikTok, Instagram, and AI have vastly surpassed TV and radio in their power to distract and trivialize, and they have worked together with other forces to spread those diseases like a global epidemic in the 21^st century. But that makes any and all antidotes valuable. There is perhaps no greater antidote to hastiness and superficiality than reading the Great Books, which can’t be read well without slowing down, taking one’s time, and reflecting as deeply as one can on complex arguments and profound reflections.

There is a beautiful passage in a letter he wrote to a friend in which Machiavelli describes how he spent a typical day during a bittersweet period in which he was living in political exile. As he tells his friend, he spent his day doing a bit of this and a bit of that. But then he says:

When evening has come, I return to my house and go into my study. At the door I take off my clothes of the day, covered with mud and mire, and I put on my regal and courtly garments; and decently reclothed, I enter the ancient courts of ancient men, where, received by them lovingly, I feed on the food that alone is mine and that I was born for. There I am not ashamed to speak with them and to ask them the reasons for their actions; and they in their humanity reply to me. And for the space of four hours I feel no boredom, I forget every pain, I do not fear poverty, death does not frighten me. I deliver myself entirely to them.

I have always loved that passage because it captures what the experience of reading can be at its best. One would not think that anyone could give a moving description of reading, nor, if it could be done, that Machiavelli would be the one to do it. But there it is. Yet notice what Machiavelli doesn’t say: he doesn’t say that he checked his phone every ten minutes, that he kept getting texts from his friends, that he couldn’t resist looking at Instagram every so often, or that four hours proved to be too much so he shortened things up by turning to AI summaries to get through the hardest chapters.

The experience that Machiavelli describes in that letter runs against the currents of our time. It has become hard for us to have that experience. But that doesn’t mean that it is impossible. In fact, it’s all the more important for us to seek it out precisely because it runs against the currents of our time. Reading as intensely as Machiavelli did can provide a much-needed antidote to our tendency toward distraction, frivolity, and absorption in the churn of the present. Reading the Great Books seriously requires one to cultivate habits of meditation, not by sitting quietly and following one’s breath, but by reading actively, following plots and arguments, and engaging in a dialogue with the great minds of the past. So that is one reason to read the Great Books: to cultivate the habit of deep reflection as a counterweight to our hasty and superficial ways of living through screens, algorithms, and endless diversions and distractions.

But one might be able to cultivate that habit in other ways—by meditating about just about anything, perhaps even one’s own breath, as boring as it would be to do that for four hours. A deeper and better reason to turn to the Great Books concerns not the kind of experience one can have in reading them but the kind of questions they raise and address—their substance, in other words. One of the things that makes the Great Books great is that they contain very rich and very complex treatments of very simple questions. The Great Books, especially the Greatest Books, tend to take up the kind of questions that little kids tend to ask, before the world teaches them that they shouldn’t spend their time thinking about what really matters. Questions like: What is justice? What is the best way of life? Is there one? What kind of happiness can we hope for as human beings? How might we get it? What is virtue or human excellence? What matters more, success or moral goodness? Is there a God? And how does our mortality change what we are when we are alive and shape how we should live?

These are questions that belong to human life as such; they have always gripped thoughtful human beings. But let me focus on another question that the Great Books raise that has become even more pressing in our time because of the rise of AI. One of the new questions AI is forcing us to confront is in fact an old question: What does it mean to be human? Now that we are seeing that a form of “thinking” can be done much better and much faster by machines, we should be led to think about what it means not just to think, but to think as human beings. How should we understand the essence of our humanity? What makes us different from machines? And how should those differences affect how we live, both individually and collectively?

Competing answers

Let’s suppose that one turned to the Great Books to think about that question or that constellation of questions. What would one find? As with the other questions I listed, one would not find an answer—at least not an answer in the sense of a singular answer that all great authors agree upon. The authors of the Great Books rarely agree with each other. That is one of the virtues of reading them, because it compels one to become a participant in a vast debate. Let me pick out two great thinkers who confront the question at issue to make the point.

If one turned to Aristotle to ask about the essence of our humanity, one would find the following set of thoughts. Human beings are defined above all by our rationality, which makes us capable of speech, and by our sense of justice, which makes us especially capable of speech and discussion with each other about matters of justice. Aristotle held the view that human beings are social and political animals who find our fullest flourishing, the fullest activation of our humanity, in communal discussion with others about matters of great moral and political importance. Aristotle would say that you can stick human beings in a zoo or put them out in the woods or strand one on an island (like Odysseus with Calypso), but those are not the places where you will find human beings being human beings in the full sense. Rather, it is in the city, the polis, that we most actualize our humanity by doing what is most distinctively human—by discussing matters of moral and political importance, and not just discussing such matters, but passing judgments and making decisions with serious consequence for our lives.

Because he held this view of our humanity, Aristotle would be deeply worried about the threat AI poses to our humanity. By threatening to take the activities of thinking, debating, discussing, and perhaps ultimately even judging and deciding out of our hands, AI, Aristotle would say, threatens to deprive us of what most makes us what we are. Just as human beings in zoos or in the woods or on islands aren’t able to live fully human lives, so too human beings sitting on couches scrolling through social media feeds while AI does the work of thinking and judging for them are not living fully human lives. They are not living out and developing their highest natural capacities or satisfying their natural craving for social connection and morally serious engagement and discussion with others. It is possible, Aristotle would say, to remain human in a basic sense while living a life that is too easy, too detached, too distracted, and too unchallenging to blossom from the basic bud of humanity into the full flower of a human soul doing what it can at its best.

That’s Aristotle. What about Thomas Hobbes? From Hobbes, one would hear something more like this. Human beings are much more like machines than we might like to think, because everything in the universe, including our so-called souls, is made up of nothing but matter in motion: everything real is body or merely material; there is no such thing as the spirit or the soul as something separate from body or matter.

What sets human beings apart from other animals, according to Hobbes, is, in the first place, not much, and, in the second place, three features of human nature that Aristotle did not emphasize because he was too concerned to paint a rosy picture of our humanity. The first of these is that we pose terrible threats to one another, especially in the absence of a powerful government. By nature, as Hobbes puts it, “man is a wolf to man,” and the state of nature is a war of all against all that leaves human beings living lives that are “solitary, poor, nasty, brutish, and short.”

What life is all about, in Hobbes’s view, is not the joy of discussing justice with our fellow citizens—which discussions, he would point out, often turn into vicious conflicts—but rather the flight from fear and the pursuit of “a perpetual and relentless desire of power after power that ceases only in death.” So we are dangerous to another, and we are always dissatisfied and anxiously seeking to fulfill our desires and assuage our fears by making ourselves as secure and powerful as we can.

And then the third and final thing, which is more optimistic, is that we are the only animals capable of an inventive form of causal thinking. Only human beings, Hobbes observes, can take things we encounter—whatever they may be, including even ourselves—and imagine new uses to which they might be put. This imaginative faculty is the root of our capacity to invent: we can transform ourselves and the world; we can go to work on nature and make it better.

My guess is that Hobbes would be much more welcoming of AI than Aristotle would be, because Hobbes would regard AI as an expression of the human pursuit of power after power, as a new way that human beings have devised of exerting ever greater control over a harsh natural world and making the world a more abundant and secure place in which to live. Nature is terrifying and terrible, in Hobbes’s view, and therefore he wanted to unleash the power of art to combat it. AI is a form of art, perhaps the most extraordinary ever devised. AI is also a form of power-seeking and power-accumulation, and Hobbes thought that human beings naturally want power.

So, although AI is not naturally human, Hobbes would see it as an expression of our humanity. And he would welcome it as a way of using technology to turn lives that nature would render “solitary, poor, nasty, brutish, and short” into lives that are “interconnected, prosperous, comfortable, civilized, and long.”

Human nature

So who is right—Aristotle or Hobbes? I doubt that that is a question any machine could get very far in thinking about, because to think about it well one has to draw from one’s experience of being human and from one’s concerns as a human being. Hobbes urges his readers to look inside themselves to test whether what he says is true, and in a way Aristotle does too. But a machine can’t really look inside itself, or, if it could, what it would find would not be human nature, even if it could compile all sorts of data about what actual human beings have said about human nature. What it wouldn’t find are human experiences—experiences like anger, sadness, hope, longing, fear, and moral passions—which may not exactly be human nature, but which are its manifestation and thus, for us human beings, windows on what it is and what we truly are.

To think well about who is right between Aristotle and Hobbes, one should read their books—not my summaries or AI’s—ponder their arguments, and avoid rushing to conclusions, since both authors were vastly more thoughtful than just about any of us are today. And of course one shouldn’t limit oneself to Aristotle and Hobbes. How, for instance, does Plato’s view that we are deeply erotic beings who yearn for immortality, even as we are also rational beings led to doubt whether what we yearn for is truly attainable, compare with, say, Rousseau’s view that we are naturally content beings who have become dissatisfied only because of vain and artificial concerns that have been foisted on us by society, or with John Locke’s view that we are acquisitive beings who want above all to acquire property that will make us both safe and comfortable?

One of the most remarkable things about being human—one doesn’t need the Great Books to know this, but they help to drive the point home—is that our own humanity is not so transparent to us. We are in a strange way opaque to ourselves. It’s not so clear what we, as human beings, most deeply care about and thus what makes us thrive and what make us wither. In fact, paradoxically, this is perhaps the one thing that can be said with certainty about human nature: that it is a question and a mystery. Human nature is something about which we know something, since we are human and live it every day, but it is also something that we don’t see clearly and therefore something to which the most reflective people in human history have devoted some of their most profound thinking. It is a sign of the complexity of the question that they have reached such different answers.

If human nature were simple, one would expect all who have thought about it deeply, or even shallowly, to have reached the same basic conclusions. But they haven’t, as one will quickly learn by reading the Great Books. That does not mean that the Great Books cancel each other out. It means, rather, that they summon us to reflect on questions whose importance is as real as ever.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

In 1832, forty men moved into a house in Ménilmontant, a neighborhood then on the outskirts of Paris. They dressed in a special uniform and subjected themselves to an austere regimen of communal work and celibacy. They referred to their leader, Prosper Enfantin, as their father.

Enfantin was one of the leaders of Saint-Simonianism, a short-lived religion based on the ideas of Henri de Saint-Simon, a French aristocrat who fought alongside the Americans at Yorktown, before making and losing a fortune speculating on confiscated French church land.

The Saint-Simonians believed that Europe’s feudal age was coming to an end. Inherited privilege and the power of the Church would be swept away by an ascendant class of scientists, engineers, and entrepreneurs. They would be able to direct credit toward productive investment, while new networks of communication would bind the regions and nations of the world together.

Saint-Simonianism was bold, extravagant, and often weird. It combined mysticism with detailed thinking about banking, railways, and politics. Enfantin and his followers set out on a trip to the East to find a “female Messiah” and within a few years, the religion collapsed amid political repression and internal divisions over free love.

But far from vanishing without a trace, the Saint-Simonians went on to become some of the most influential figures in French public life. They founded financial institutions, built railways, and laid the groundwork for the Suez Canal. Though their religion failed, it had brought together some of the most talented writers, engineers, and financiers in France and infused them with a diagnosis of the future and an urgency around institution-building.

The Saint-Simonians weren’t the first, and won’t be the last, group of people who, dismissed by many of their contemporaries, sensed that the ground was shifting under their feet and sought to embrace radical solutions. But unlike most people in their position, they managed to escape the confines of their sect and turn these ideas into institutions that would become useful to millions of people.

As we stand on the precipice of radical social and economic change, another group of people, similarly dismissed by the establishment as madmen, is now seen as prophetic. But it remains to be seen if they can make the same journey as the Saint-Simonians, and build lasting institutions.

Subscribe now

Seeing the future

In 2000, AI research was thought to be deep into a winter. Although Deep Blue had defeated Garry Kasparov at chess three years earlier, it had relied on purpose-built chips and fine-tuning by grandmasters. The chatbot that won the year’s Loebner Prize, an annual Turing Test contest, worked by matching participants’ input against more than 40,000 hand-written rules. Moderately-skilled Go players could beat the best computers, and object recognition was so undeveloped that the field hadn’t even yet created standardized benchmarks.

But 2000 was also the year that Eliezer Yudkowsky founded the Singularity Institute (later renamed the Machine Intelligence Research Institute). The Singularity Institute was inspired by the ideas of the British mathematician I.J. Good, who described a process whereby a sufficiently advanced intelligent machine could create its successors. Yudkowsky’s goal was to accelerate progress toward powerful, “Friendly AI”, but changed tack after deciding that the development of superintelligent machines was too dangerous to pursue.

These types of early efforts were focussed on the prospect of building transformative AI and making it safe, but less so the downstream implications of what such a technology might mean for society.

Since intelligence sits upstream of science, technology, governance, military power, and everything else, the implications were enormous. But almost no one cared. Governments don’t expend resources on far-off science-fiction scenarios, and computer science departments focus on the problems in front of them. Meanwhile, the technology companies of the 2000s were high on enthusiasm about the internet.

But if you took the idea of an intelligence explosion seriously, then researching and understanding it took on an urgency incompatible with the incentives or timelines of established institutions. You needed to build alternatives.

Nick Bostrom founded the Future of Humanity Institute at Oxford in 2005 to bring together thinkers concerned about existential risk and advanced AI, so that they could pursue inquiry that didn’t yet fit in any established fields. Bostrom would later write the enormously influential Superintelligence in 2014, which became a foundational text for thinking about the implications of developing machine intelligence.

For its first decade or so, the AI safety movement was largely ignored. But in the early 2010s, the emergence of the effective altruist movement catapulted it out of obscurity. EA’s intellectual framework is built on maximizing the good done per dollar or hour spent, a tradition running from Bentham and Mill through Sidgwick and Parfit, and often applied to animals and people not yet born. When this framework was applied to the future, existential risk naturally took on outsized importance.

A community can believe that a problem matters without possessing the institutions – organizations and the “rules of the game” that govern coordination – to act on those beliefs effectively. It helped fund those working within this framework, and from 2014 onwards, 80,000 Hours began advising technically capable readers to consider careers focused on reducing the risks of superintelligent AI. If a bright young upstart was concerned about AI risk, they no longer had to find someone like Yudkowsky independently to start working on it.

The belief in the potential of machine intelligence to radically transform the world galvanized people into creating an alternative world of research agendas, a common intellectual language, and employment and funding opportunities, long before credentialed institutions took it seriously.

Immanentizing the eschaton

On one level, this generation of prophets has been immensely successful at turning their ideas into institutions. At the 2010 Singularity Summit, Yudkowsky introduced Demis Hassabis and Shane Legg to Peter Thiel, who became DeepMind’s first major investor. Meanwhile, Anthropic’s first funding round was led by Jaan Tallinn, a Skype co-founder who had funded work on AI risk.

The majority of this institution-building focused around AI itself, namely building, aligning, measuring, and monitoring powerful AI systems. This made sense as an initial focus: if you believed there was an immediate risk of humanity creating systems that could lead to extinction, the first institutions you would build would aim to ameliorate that risk.

Why spend years building a university if the economic and scientific order will be unrecognizable in a few years? Why painstakingly construct an insurance market around current AI systems if the capabilities will leap several generations before we know it? Why work out how to build a new system of scientific variation if automated R&D is about to abolish the research process as we know it?

But suppose we do reach a world in which extinction is avoided, but AI does reorganize our lives, whether through science, government, education, or our individual capacities. If that transformation is coming, then narrow AI safety institutions are necessary but not sufficient.

This is where the totalizing, prophetic mindset that helped form this ecosystem can have limitations.

A beginner’s guide to building an AI-era institution

For a long time, taking these ideas seriously was a bad look. The community was small and weird, and the range of experience its members could bring was relatively narrow.

Today, that is not the case. Many more people understand the gravity of the situation, even if the public does not yet fully grasp what the labs are trying to build. In San Francisco, it is now rare not to see recursive self-improvement lurking around the corner, with all of the implications for safety and society it brings with it. There are hundreds of millions of dollars of capital available for people who want to build something, anything, that can help us midwife the best possible future.

The capital, technology, people, public sentiment, and will exist to create the institutions of the AI era. Given what we know about the form that strong AI is likely to take, at least in its formative years, we can see what the near future is likely to look like and what kinds of responses are needed to derail bad outcomes and realize good ones.

But institution-building is not for the faint of heart. You need to win allies and assemble the expertise that enables others to feel confident working with you. A strong conviction about the future is essential; it is also table stakes. Also required is a specific account of what you intend to do and how you intend to do it.

You begin with a problem that becomes more important as intelligence becomes abundant. Currently, for instance, machines can generate papers faster than people can referee them. This means that someone must develop a mechanism for separating slop from science, which is already happening at conferences like NeurIPS.

The budding institution-builder should mediate between existing and near-term capabilities. This is because, even if you think that everything will change too quickly for institution-building to be worthwhile, the experience accumulated along the way is itself valuable. There is always something useful for an institution’s employees to do on Monday morning, even if the arrival of superintelligence gets stuck in the mud.

For example, we might hypothesize that capital allocation becomes more important as intelligence gets too cheap to meter. We need institutions that will help allocate capital efficiently given the large number of possibilities posed by harnessing and reacting to strong AI. It is a mistake to refuse to build this capacity in advance because any institution can accumulate knowledge of finance and the principles of capital allocation that it can put to good use in the future. There may be intermediate regimes where a problem arises but solving it is not wholly automatable, and so we need that capacity and expertise – grounded on specific theories formed by individuals – to ensure we can navigate these situations.

The best institutions are built by those who live in the world as it is, rather than as they wish it to be. Reading METR graphs does not automatically bring with it an understanding of how specific domains will respond to agent ecologies. An educational institution for the age of AI needs to specialize in education, just as a financial institution needs to specialize in finance. If you think the old way of doing things is wrong in every case, you are probably mistaken. Cursed is he who pulls down Chesterton’s Fence.

If you have a sense of what a positive future could be — and what might make it so — then you already have the first ingredient for building the institutions we need. If you have domain expertise or would be willing to develop it, you are better placed than most of your peers. If you have both, then all that remains is the inclination to take custody of a function that others can rely on.

That might mean building the “attention guardians” first described by Seth Lazar in 2024, an effort that has since come to fruition in the form of Meta’s Muse which filters various information sources according to the specifications set by a principal. Or it might mean implementing new coordination mechanisms for governing agentic interaction according to signals like reputation or trustworthiness.

It could involve looking at a specific domain like education to explore the failure modes that the agent economy might introduce, as well as the specific countermeasures that powerful AI might be able to employ. In whatever form they take, building new institutions depends on the combination of anticipating novel problems, deep domain expertise, and a commitment to create the required norms and make them stick.

Thomas Huxley once said “the great end of life is not knowledge but action.” Now more than ever, if you have conviction about the shape of the future, now is the time to act.

The task ahead

Anyone who takes the leap from merely interpreting the future to seeking to change it will inevitably face accusations of treachery or “selling out.” This is in part because there is a sort of betrayal involved. Prophecy can remain universal in a way that an institution cannot. The institution needs a focus, a budget, and rules, and it has to reach a bargain with people who do not share the worldview of its creators. A passionate diagnostician can indulge in theoretical purity in a way someone who is trying to build something, while accountable to others, cannot.

This is a tale as old as time. At the time of the American Revolution, Thomas Paine and John Adams were both early advocates of independence. Paine’s pamphlets like Common Sense built public enthusiasm for independence from Britain, stripping away the mystery around hereditary authority and popularizing the idea that sovereignty belonged with the people.

But Adams, much as he admired Paine’s diagnosis, believed that his ideas about government were naive. He feared that Paine’s proposed system of a simple republic centered around a single legislature would be captured by ambitious, partisan men. This led him to support the separation of powers.

Paine responded with a series of escalating attacks on Adams for his lack of revolutionary purity. The targets of Paine’s ire widened to include George Washington and then organized religion. In the process, he lost much of his popularity. When he died in New York, only six mourners attended his funeral.

There is a warning here for those who saw AI early. It is possible to remain intellectually wedded to the most uncompromising version of your original analysis and to become a virulent critic of every imperfect institution built by others. But the price of refusing to make compromises is that other people will make them for you, while you speak to an ever-dwindling audience.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

In July, more than 1,300 employees of the world’s leading AI companies signed an open letter called Pacing the Frontier. The letter argued that frontier labs may be approaching the point at which AI research could be automated, potentially allowing capabilities to accelerate at a rate “beyond our ability to understand or control the resulting systems.”

Arguing that labs are incentivized to race ahead recklessly, the signatories asked the US government to support an international effort to develop the technical and governance tools that might be needed to deliberately slow the “pace” of frontier AI development.

As well as its size, the pacing coalition is notable for its breadth. Many of its signatories have diverging intuitions about openness, the role of state power, and the nature and extent of risk from AI.

One signatory, Dario Amodei, has since published an essay on pacing the frontier, arguing for embedded external evaluators at frontier labs, coordination between democracies, and international agreements. This immediately attracted the endorsement of Elon Musk and Sam Altman.

Other Pacing the Frontier signatories include Dawn Song, a long-standing advocate of decentralization. There is also John Schulman, who talks about the “possible need” for such a mechanism, and Ilya Sutskever, who argues that future AI will be so powerful that we will need “unprecedented measures.” Others cite issues such as social adaptation, institutional preparedness, the “liberties of the free world,” or just the value of buying time.

These concerns are not mutually exclusive, but they imply very different ideas about what a pacing mechanism should look like in practice.

A signatory who is primarily worried about existential risk from runaway recursive self-improvement is likely to tolerate significantly more intrusive controls than one who is motivated by a desire to preserve optionality or mitigate labor market disruption. Similarly, someone focused on American national security may be skeptical of international cooperation that risks eroding America’s strategic edge.

All of these positions can coexist under the injunction to “build the capacity to pace.”

But this stores up the problem. While the coalition can agree on the need to create a capacity, they don’t seem to have a shared view on who should ultimately exercise it or to what end. But any attempt to separate these two issues runs into uncomfortable questions about power.

Subscribe now

Retreating to safer ground

In 1929, the German philosopher Carl Schmitt wrote an essay called “The Age of Neutralizations and Depoliticizations.” Schmitt argued that throughout European intellectual history, when conflict became too intense in one domain, people looked for another domain where agreement might be easier.

For example, people incapable of reaching agreement about religion might be able to find common premises if the question was translated from theology into the language of nature, reason, or law.

This, Schmitt argued, is why the intellectual center of European life shifted from theology to metaphysics, then to humanitarian morality, economics, and eventually technology.

This isn’t necessarily bad from a liberal point of view. A viable plural society can’t require agreement on ends before people are able to cooperate. Liberalism’s genius lies in its ability to find procedures and institutions that allow people with incompatible convictions to live alongside each other.

Schmitt, however, argued that neutrality degrades as “the newly won neutral domain … become[s] immediately another arena of struggle, once again necessitating the search for a new neutral domain.” In Schmitt’s view conflicts persist because human groups form around incompatible ways of life, with politics reappearing when those distinctions become existentially important. Neutralization can delay it, but the same tensions always resurface. For example, wars of religion gave way to larger wars of nation-states and economic competition.

Technology may seem to offer the ultimate neutral ground. A machine or a scientific technique can be used by people with completely different moral or political objectives. Meanwhile, technological questions appear factual and soluble in a way moral and theological ones never did.

For Schmitt, however, technology could never provide the neutral settlement people wanted from it. Technology is a supplier of means, not ends. Thus, “every strong politics” will attempt to master it.

What’s in a name?

Many of the most important questions about AI governance are contested. When is the state justified in restricting technological development? How much risk is required to justify such an intervention and how do we measure it? How should governments weigh domestic safety against strategic competition? How much weight should policymakers place on the most extreme scenarios of recursive self-improvement?

Some of these questions could be resolved with more evidence. But better evidence itself does not dictate what form intervention should take. This isn’t an argument about how “AGI-pilled” you are: people with the same beliefs about model capabilities (including those with nontrivial fears of catastrophic outcomes) could reach radically different answers.

The language of “pacing” suggests that we can build the capacity now and settle the politics of how to use it later. But a capacity to compel restraint must have an object, a trigger, and means of enforcement. Giving them institutional forms forces us to make judgments about the nature of the risk and what constitutes legitimate coercion.

First, someone has to decide what is being paced.

The progress of frontier capabilities is shaped by a combination of compute, algorithms, training data, post-training techniques, institution design, and so on. Whichever combination a mechanism attempts to limit would entail a judgment about the origins of any risk.

Consider the signatories’ concern about “loss of control.” Losing control is an outcome, not a threat model. It could describe a model deceiving the humans overseeing it, agents finding unintended ways of coordinating, or systems exploiting unknown access to external infrastructure. Under some definitions, it could even mean an institution delegating decision-making to the point where it can no longer supervise it effectively. Saying that you are trying to avoid “loss of control” doesn’t actually provide a guide to action.

Second, someone would need to determine when intervention is justified. What evidence means that the government should stop an otherwise lawful experiment? What kind of action would be taken, and how should we judge its proportionality? When should we be biased in favor of experimentation and when should we embrace a version of the precautionary principle?

Finally, any pacing regime would need a model of enforcement. If compliance requires access to internal experiments or model weights, choices must be made about surveillance and disclosure. For example, if there are exemptions for work focused on national security, that would implicitly privilege strategic advantage over safety.

If we look at these three questions, Amodei’s pacing proposals are incredibly uneven. He offers concrete ideas around monitoring and verification, with significant word count expended on the access and independence of the external evaluators. But he never specifies what is actually being paced, the conditions that would trigger restraint, and the consequences for crossing such a threshold. For example, one of his proposals is simply: “Some kind of ‘speed limit’ on the rate of recursive self-improvement.”

A verification regime can tell us whether or not someone has complied with the rules; it does not tell us what the rules should be. Amodei has continued to postpone the harder choices the pro-pacing coalition would rather leave murky.

Of course, that isn’t to say that these questions are unanswerable. It is possible that fresh evidence, research, and robust debate could lead the frontier labs and others in the field to a broad consensus. This consensus could then be translated into a set of concrete proposals. But we are not even close to approaching this point.

The state of exception

It would be easy to dismiss this as semantics. After all, many successful political coalitions have been built around constructive ambiguity. But vagueness is dangerous when the issue people are dancing around is state capacity. If a broad coalition agrees that the government should build the capacity to “pace” AI, without agreeing how or under what conditions, the unresolved questions are deferred to whoever ends up exercising that power.

If you believe that AI can become dangerous with little warning and that labs lack transparency then you will likely believe that anything other than tough, intrusive measures will be ignored or circumvented thanks to the power of incentives. This means that the government would need to take a hands-on approach to both regulating and monitoring.

These measures could include visibility into training runs and internal evaluations, control over access to compute and advanced chips, powers to compel disclosure, and authority to enforce a freeze. These powers would need to be reinforced by mechanisms to detect and halt jurisdictional arbitrage, and we would need a policy for what happens when a strategic rival refuses to play the pacing game. In essence, we are preserving the ability to act when ordinary arrangements no longer suffice.

Schmitt famously proposed that “sovereign is he who decides on the exception.” That is, rules presuppose normal conditions. So in exceptional cases, their application becomes uncertain, and someone has to assume extraordinary authority.

When a regime is justified by the need to act rapidly against poorly specified extreme risks, it will by definition involve setting aside normal conditions. This will mean ever greater executive discretion.

A pacing regime would have to act on incomplete evidence before a decisive risk of catastrophe had been identified (otherwise it would have been caught by the labs’ existing procedures). For example, officials might have to determine whether anomalous evaluation results represent alarming capabilities, or whether evidence about one model warrants restricting another.

There is also a risk that this regime would become ever more secretive. Labs would want to guard their commercial edge, and there would be concerns that even descriptions of cyber or biological vulnerabilities could be dangerous if made public. Moreover, evidence about foreign actors may come from intelligence sources whose provenance cannot be made public. Whoever is deciding that situations are “exceptional” may also control access to the very evidence that others would need to challenge their decision-making.

There is a related problem of duration. Schmitt distinguished between a short-term “commissarial” dictatorship, whose primary function was to restore the previous order, and a sovereign dictatorship that paved the way for a new one.

Pacing advocates don’t all have the same regime in mind. Some hope for a temporary setup to buy time. But based on their public statements, there are many who are (not particularly secretly) hoping for something more sweeping.

For example, restrictions may be introduced on account of a specific threat that emerged in large training runs or evaluations. If it subsequently became possible to produce the same threat through post-training, inference-time compute, or agent scaffolding, the authority would need to expand its mandate. It’s easy to see how a series of individually reasonable extensions could lead to the emergence of a frontier AI governance system by the backdoor, operated through executive fiat.

Of course, a chunk of the pacing coalition would be delighted at this outcome. But it’s easy to see how the powers could evolve in directions even they didn’t anticipate.

The agencies capable of monitoring compute, compelling disclosure, or controlling access to advanced technology already have important responsibilities, such as preserving military advantage, export control and industrial policy. These agencies may decide that they care about AI safety more than they do about strategic competition and act accordingly, but it’s a big gamble.

Amodei’s proposals note these tensions, but make no effort to resolve them. He conditions pacing agreements in democratic countries on maintaining a lead over China and even proposes using technology restrictions to widen this lead. But if China managed to erode this lead before labs had completed whatever safety work was necessary, the government would have to choose either attempting to slow China further or accepting greater risk. This simply introduces yet another unresolved political question.

Red lights on the dashboard

Once the government can alter the pace at which strategic technologies are developed and deployed, especially through the use of an ambiguous and potentially secretive process, it has acquired the power to decide who is permitted to move with the frontier.

Advocates for pacing may therefore convince governments of the exceptional character of frontier AI risk, only to lose all control over what happens next.

We’ve already seen an early glimpse of what this can look like. When Anthropic refused unrestricted military use of Claude, the Pentagon sought a federal ban, prohibited defense contractors from conducting unrelated commercial business with Anthropic, and designated the company a supply chain risk.

It is true that a pacing regime could create a series of transparent conditions and trigger points that make it harder for Leviathan to lash out arbitrarily in a fit of panic. But there are asymmetric incentives at work here.

Given that a pacing mechanism would be addressing potentially unknown danger, officialdom’s natural inclination is to set the threshold for intervention low. After all, you are much more likely to face consequences for allowing something bad to happen or for failing to respond to public panic than you are for slowing hypothetical economic or scientific progress. The history of America’s national security state shows that codifying powers does not necessarily constrain them. Anthropic managed to overturn its designation as a supply chain risk in the courts, but a future decision may be harder to contest.

In his writing on neutralization, Schmitt noted that the people who establish a new domain often do not become its political masters. Nearly a century ago, he could see that it was unlikely that a “politically dominant elite would develop out of the community of technical inventors.” The ability to forge a new domain or category does not imply the ability to shape its politics.

The need for politics

Carl Schmitt may seem an unlikely ally in the fight against unconstrained state power. After all, in Schmitt’s eyes, liberalism sought to replace politics with procedure, whereas he yearned for decisive political authority. This would sit comfortably with his subsequent unrepentant embrace of fascism.

But we can learn from his analysis while rejecting his political conclusions. Before handing governments potentially coercive powers, advocates should explain which powers they support, what would justify their use, what limits would be placed upon them, and how civil society could challenge them. They need to reject the comfort of proceduralism and rediscover politics.

They would almost certainly discover that they support an array of incompatible answers to these questions. But it is far better to put forward competing proposals and argue about their merits than to preserve apparent agreement until the government resolves them for you, in a way nobody intended.

This argument would establish what this coalition is actually capable of endorsing. If the apparent only extends to researching potential mechanisms, then it is important that the members of the coalition acknowledge this boundary. That means the supporters of more sweeping measures would have to defend these powers without being able to rely on the endorsement of people whose agreement is conditional on limits that haven’t yet been negotiated.

Advocates might argue that it’s unreasonable to demand a full institutional blueprint at this stage. But whether or not you would support a pacing regime is presumably dependent on questions like its scope or duration. Retroactively withdrawing endorsement is much harder than providing it.

Stepping up

It would be misleading to imply that no one in the pro-pacing camp is taking these questions seriously.

Several groups have started to engage with these questions. AI 2040 was an early attempt, with its idealistic vision of “total research transparency,” while Jasmine Li and Jason Hausenloy have published proposals on how compute verification could enable a coordinated global slowdown.

Yesterday, a group led by Raymond Douglas at ACS Research published their own pacing research agenda. The strength of this agenda, alongside its detailed list of potential intervention targets, is its admission that there isn’t a single “pacing program.” The measures you would support will diverge significantly depending on your underlying intuitions about AI. This is a step toward the political contestation that we badly need.

But these are small third-party efforts, while the institutions with the most political power and the greatest ability to implement these ideas at the frontier continue to duck these questions, in favor of vague calls for pacing.

With great responsibility comes great power. There may well be situations that present such grave risks as to justify extraordinary top-down intervention. But if the advocates for pacing with the most leverage want to empower the government, they need to describe the regime they actually have in mind to avoid indeterminate mission creep. These questions are too important to bury beneath the agreeable language of scientific cooperation or international coordination. When you turn to the federal government for salvation, be careful what you wish for.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

Most generations inherit their world. Some must build it anew.

Today, we’re launching Cosmos Ventures to back philosopher-builders founding institutions for the AI age.

Two hundred and fifty years ago, the American founding demanded new institutions for a new kind of society. Benjamin Franklin identified what that world would need and built what was missing. He believed self-government required educated citizens, so he pioneered subscription libraries. A free people required a free press, so he seeded newspapers. Neighbors had to carry one another’s risks, so he established mutual insurance.

AI is creating the biggest opening for institution building since the American founding.

Our world developed around the assumption that intelligence was bound to human beings. Ideas had human authors, and consequential decisions had accountable individuals behind them. If employers needed analysis or creativity, they had to hire people. AI is breaking that bond.

Some assume AI is a normal technology our institutions will absorb as they absorbed the internet and the smartphone. Others insist superintelligence leaves little worth building outside the frontier labs. Both are counsels of passivity at a moment that calls for founders. Many of the institutions we will need have yet to be built. We have to decide what we value most and how to secure it. These are questions no machine can answer for us.

Founders answer them with what they build. Tomorrow’s most important companies will set new norms that others organize around, becoming foundational institutions of the AI age.

Institutions encode theories of how we should live together. A new school is a claim about how children should be formed and what is worth teaching them. A personal agent is a theory of which decisions we can delegate and which we must make ourselves. A new assurance regime is a theory of what to trust and how to demonstrate trustworthiness.

Enduring institutions won’t be built by simply following user feedback and iterating. Their founders must answer normative questions about what kind of society they’re building toward. They must be philosopher-builders. We are here to back them.

Immanuel Kant gave the Enlightenment its motto: Sapere aude. Dare to know.

The AI age demands another.

Condere aude. Dare to found.

Tell us what you are building for ↗

^{Cosmos Institute Inc. is a 501(c)(3) nonprofit organization. Cosmos Ventures refers to the venture investment platform associated with Cosmos Fund I LP and related vehicles. Cosmos Management LLC serves as investment manager to all Cosmos Ventures vehicles, including Cosmos Fund I LP. References to “the Fund” or “Fund I” refer to Cosmos Fund I LP. These investment activities are distinct from the charitable and educational programs of Cosmos Institute, Inc. This communication does not constitute an offer to sell, or a solicitation of an offer to buy, any interest in Cosmos Fund I LP or any related investment vehicle. Any such offer may be made only through definitive offering documents by the appropriate fund, general partner, manager, or authorized representative.}

Liberalism begins from the fact that people pursue different, and revisable, conceptions of the good life. Rather than directing people toward some collectively chosen end, the liberal state governs through general rules. Economic success, on that premise, is a matter of capacity. I previously defended GDP as the best scalar proxy for a society’s ability to solve problems: an economy with more output can devote more resources to almost anything its members decide they care about. GDP is, in this sense, a liberal measure. It measures the means available for pursuing different ends, but leaves people to decide which ends to pursue.

This capacity, which comes from a combination of labor, knowledge, and resources, furthers particular human ends that emerge through cooperation. People combine what they have in order to serve one another’s interests; in return, they earn the means to act on ends of their own.

At the scale of an economy, that cooperation stretches across millions of strangers whose aims, knowledge, and opportunities to act are constantly shifting. This creates an informational problem because the knowledge it would take to make existing plans effective and compatible is scattered across many people. But the goods people value change as new possibilities come into view.

Market institutions form the default economic expression of liberal commitments because they allow open-ended cooperation to proceed at scale without relying on serving different purposes through a collective, and often coercive, plan. Instead, prices enable the plans of many parties to adjust to circumstances unknown to their authors, while rival experiments create knowledge about what people will come to value, and of how their ends can be served.

Subscribe now

Cooperation Without Agreement

People’s differing purposes generate two related conflicts. First, ineradicable scarcity makes other people’s plans obstacles to one’s own. This creates an incentive to use violence to settle conflicts. Since people have to pursue their ends alongside one another, some political order is required to mediate between competing priorities. Second, concentrating force in the state means that whoever controls it can force others to serve their own goals. Hence the need for a liberal political order that confines collective power within general rules under which rival plans can coexist.

These conflicts are most acute in economic life, where every act of production commits scarce resources to one purpose at the expense of others. Giving an authority the power to rank these ends would reproduce in economic life the political problem liberalism exists to contain.

The liberal answer to the problem of coordination is to extend the distinction between general rules and particular ends into economic cooperation. Law establishes general and knowable rules (for example, around contract enforcement and taxation) within which people can cooperate, but doesn’t specify what they must cooperate to achieve. So if others have chosen to pursue some project of which you disapprove, you can refuse to participate, or criticize it and organize an alternative. But if they are acting within those rules, you are not entitled to prevent others from cooperating.

Exchange narrows the scope within which agreement is required. If someone wants to open a restaurant, they do not need a collective answer to the question, “Should this restaurant exist?” Instead, assent is fragmented; they need only strike a series of narrow bargains on terms the other parties are willing to accept. Will a landlord rent a building at a price they can afford? Will employees accept the wages and hours on offer? Will someone supply the ingredients at a price that makes sense for the restaurant? Will customers pay for the meals? Nobody has to share the owner’s enthusiasm for the cuisine, still less agree that this particular restaurant advances a collective good. The landlord may just care about the rent; the cook, his wage or the satisfaction he derives from his work; the supplier, a reliable customer; the diner, dinner. Their purposes meet during this exchange while remaining different everywhere else.

This dynamic also limits what any participant may demand of others. People are free to pursue their plans, but they are not entitled to other people’s cooperation. A worker’s desire for a job does not require any particular employer to create one, nor does an employer’s desire for labor require any worker to provide it. This reciprocal constraint applies to every exchange, not only labor. If my project requires something another person controls, that thing does not become mine simply because of the importance I attach to my end, any more than their need for what I can offer obliges me to provide it. Other people are therefore not encountered as resources I can just appropriate; they enter my plan as independent conditions of its success, whose cooperation I must earn. A market gives me the freedom to propose a plan, but not the authority to realize it. I must make its pursuit responsive to the will of others.

The narrowness of these agreements is what allows bounded cooperation to extend across an entire economy without turning everyone who takes part into members of a single enterprise with a shared purpose. People may of course choose to enter purposive and hierarchical organizations, including firms where they agree that someone else will decide what they do over a specified domain. But outside that domain, the employee remains free to spend their wages however they like, use their time for other purposes, and support causes not shared by the firm.

A liberal economic order must therefore both leave people free to form plans without collective agreement on their ends, and somehow reconcile the competing demands those plans place on scarce resources. But across billions of participants pursuing complex and changing designs, no individual or authority can survey all the relevant claims and circumstances, or decide which ends should yield to which. How, then, do conditions elsewhere in the economy register within each person’s plan, without common knowledge or purpose?

Known Unknowns

Such coordination is possible when these local agreements take place within a market: an institutional setting in which many buyers and sellers transact and their competing offers give rise to prices. Hayek’s example is tin. If tin becomes scarcer, or more urgently wanted elsewhere, its price increases. Each user must decide whether to pay more, use less, find a substitute, or stop; those decisions alter demand for other inputs, whose prices may change in turn. No one needs to understand the whole chain of consequences. Knowledge can therefore remain dispersed while people adjust separately, and the plans become more compatible.

The tin example isolates only one part of the problem. It shows how prices help people adjust plans and possibilities that are already in view. However, we cannot assume that consumers know what they want, producers know what can be made, uses for tin have already been imagined, and the plans in which it figures are already specified. Much of that knowledge does not exist yet, for economic life also involves discovering uses, products, and purposes that nobody has yet conceived. Economic coordination therefore involves two unknowns that cannot be settled in advance. These are what can be done with scarce resources, and what people will come to want as the range of possibilities expands. The possibilities themselves have yet to be worked out; means and ends emerge together through the process of market discovery.

Hayek later extended the argument in his account of competition as a discovery procedure. Here, the price system doesn’t only coordinate among known possibilities; it directs search effort toward areas where demand is strongest. If tin becomes more expensive, a method that uses less of it becomes more valuable. The higher price doesn’t communicate those solutions, but it creates a signal that it is worth searching for them.

Because the possibilities concern what can be done with scarce means, this search often has to take the form of an experiment. A new method can be theorized, modeled, or simulated beforehand to some extent, but eventually resources have to be committed: the restaurant has to open, the machine has to run. Putting the method into use can reveal costs that differ from projections, and customers may use the product unexpectedly or not at all. These experiments therefore reveal possibilities that could not have been specified beforehand. The textbook picture has markets choosing from a complete menu of possible plans. But part of what competition discovers is what the options are; there is no menu.

Were human ends fixed independently of this process, the story could stop here. Market competition would amount to an expanding search to discover new ways of satisfying settled wants. But new possibilities also create choices that did not previously exist. Just as producers do not know all the ways consumers’ ends might be served, a person cannot have a settled view about every good or form of life before it becomes available to them.

Experiments in Wanting

The second unknown concerns what people will come to want. Sometimes experience tells us whether something suits a preference we already have. But experience can also develop new interests and standards by which we judge what is worth doing. In those cases, learning about a possibility helps form the preference we later express.

This is the same epistemological structure underlying Millian “experiments in living.” If people are agents capable of forming, revising, and pursuing their own ends, then the knowledge of what kind of life suits a person is not available in advance, not even to that person. In trying a way of life, one can develop the interests and abilities through which it becomes valuable to one. People are therefore both learning about themselves and changing through the experience.

The knowledge is not separable from the practice: we may often have a good idea whether a way of life would suit us, but some judgments can only be tested through experience – by taking up an occupation or joining a community and finding out how it goes. Individual freedom is therefore not merely a political accommodation for people who already know what they want. It is a condition under which that knowledge can come into existence. Economic freedom belongs within that broader liberty because production and exchange are among the ways people encounter new possibilities and learn what place, if any, to give them in their lives.

Imagine asking someone living in 1900 how much they would pay for a smartphone. They might already value communicating with friends, staying up to date on news, and being entertained. But that does not mean they had some pre-existing willingness-to-pay for a device they had never encountered. Someone who bought one simply to stay in touch might start taking photographs, learn to notice light and composition, and come to care about making good pictures. What began as a convenient way to communicate could develop into an interest that changes how they spend their time. To say that consumers later revealed a preference for smartphones is true, but that only reveals a choice among the possibilities confronting someone at that moment, not a ranking they carried before the possibility existed.

The story doesn’t end after a purchase is made. The buyer may be pleased with it, or discover a use they had not expected. But the experiment may also disappoint: someone can discover that they dislike what that purchase brings into their life. Each outcome supplies information, both to the producer about demand and to the consumer about their wants. Liberalism doesn’t depend on the assumption that people’s choices are good for them – regret and revision are part of the discovery process. What matters is that neither an individual nor an external authority can reliably know the result before the relevant experience occurs. Leaving the choice with the individual preserves the freedom to try, withdraw, and try differently – to notice errors and correct them, rather than substituting an equally fallible but less corrigible judgment, backed by authority, for the individual’s own.

Markets therefore do not match a fixed list of means to a fixed list of wants; they allow both to be worked out through exchange. Moreover, once such a match is found, the knowledge need not disappear with the transaction that revealed it. It can be embodied, and thus become part of the productive capacity from which later experiments begin.

From Discovery to Growth

Exchange separates the ends people serve from the ends they pursue. Someone who wants a Porsche doesn’t need to build one; instead, they can become a surgeon or an engineer, supply something others value, and use the income to command the labor and materials a Porsche requires. It is through this separation, between the end served and the end sought, that we get division of labor and specialization. People can spend whole careers solving problems for strangers, building up knowledge and abilities they would have no use for if not for someone else’s ends. The price signals that direct production in the present also inform which productive knowledge is worth acquiring for the future.

These gains can accumulate when knowledge is preserved and passed on. In a market, the gains from finding a better way of doing something give people reason to preserve and extend it. Techniques can be built into machines, encoded in software, absorbed into routines, embodied in standards. What once had to be worked out becomes part of the organization of production itself – something the next person can simply begin from.

As productive knowledge accumulates in skills and capital, higher productivity increases real incomes, creating a larger market that can sustain further specialization. A producer may find it worthwhile to concentrate on a narrower task, or to develop a method whose cost could not have been recovered from a smaller volume of sales. Since these gains are retained, subsequent experiments can build on these new capabilities – and the new possibilities they open up keep reshaping what people find worth pursuing. Where that feedback loop is sustained, gains compound upon gains, contributing to the extraordinary, sustained growth in real GDP per capita since the Industrial Revolution.

But if markets also shape the ends they satisfy, then growth may look less like the discovery of human ends than a self-amplifying process that manufactures the demand it satisfies. And if productive experiments can impose costs on people outside the exchange, what registers as success may fail to fully reflect these consequences. Both possibilities turn the argument toward institutions: what rules preserve the autonomy and decentralized experimentation on which the case for markets has rested?

The Constitution of a Market

Galbraith made the first of these objections, in a form he called the dependence effect: through advertising and salesmanship, producers influence what people want in order to sell to them, rather than satisfying wants they already had.

The fact that markets influence preferences does not, by itself, tell us whether that influence supports or impairs people’s ability to choose. Nobody invents their ends ex nihilo: friends, schools, technologies, communities – and yes, the market – all shape what people come to want. No preference arises independently of outside influences. Indeed, I argued before that a new possibility could hardly become part of someone’s conception of a good life without first affecting it. Hence Hayek’s reply to Galbraith: that advertising is only one such influence among many.

The relevant distinction is not whether preferences are influenced – they always are – but whether that influence impairs people’s autonomy: their ability to assess their wants, revise them, and act on their own judgments. Deception can undermine autonomy by distorting someone’s understanding of what they are choosing; coercion can undermine it by preventing them from refusing or changing course. Practices can impair this ability without deception or coercion. A service might repeatedly prompt users to override limits they have set, while burying the controls for reinstating them. Continued use would not by itself show that users had reconsidered those limits. What matters is that people remain free to encounter alternatives, test their wants against experience, and revise them in light of what they learn. A market can therefore succeed at selling something while failing to preserve the freedom on which this defense of markets rests.

The same requirement applies to production. If a method only appears cheap because some of its costs are shifted onto people outside the exchange, the price carries misleading information – it no longer conveys the full cost of the experiment. Rules assigning liability, defining force and fraud, governing externalities, and so on, can be compatible with a market order – indeed, a market order is constituted by such rules, established and enforced by the state. What matters is that they sustain the conditions under which productive experiments can be run and corrected, rather than determining their ends in advance1.

Selecting Experiments

Under general rules, people can try things, cooperate where it suits them, and find out what works by how others respond. But coordination is only part of what markets achieve. New possibilities and ends are discovered, generating the knowledge from which later choices proceed and the productive capacity from which later experiments begin. The result is sustained economic growth that leaves people increasingly able to pursue the ends they come to value.

But discovery alone cannot tell us which possibilities to prioritize. Experimentation does not abolish resource constraints. Scarce labor and capital cannot sustain every conjecture at once. Some experiments must attract more resources, while others adapt and come to an end. Rival experiments must therefore be selected as well as attempted. A sale tells us that somebody valued what was produced; it does not by itself tell us whether the labor, capital, and materials consumed in producing it could have served other purposes better. For that we need a way to compare what an experiment creates with the opportunity cost of the scarce means it uses2.

The liberal case for markets is that a plan does not need collective agreement on its end; through exchange, it can instead obtain the particular cooperation it requires from people acting for reasons of their own. Rightly constituted, markets connect Hayekian discovery to Millian experiments in living: people can draw on growing productive capabilities as they figure out how they want to live. To serve that purpose, markets must also preserve people’s ability to reconsider the wants they help form.

This essay is by Aashish Reddy, a Researcher at Cosmos Institute. It is the third piece in a series on liberalism, hosted on his Substack. The first gave his account of liberalism; the second defended GDP as its measure of economic success. Later essays will examine profit, capitalism, NGDP targeting, and inequality.

Subscribe now

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

A common criticism of liberalism is that it is overly permissive: that its followers are reckless libertines, committed to a policy of “anything goes.” Within contemporary discourse, we often hear that the liberals “have gone too far” or are at risk of “eating themselves.”

But this kind of criticism can be found across the history of liberalism. You can see Locke pre-empting it when he distinguishes “liberty” from “license.” You can see it at work in the so-called “paradox of tolerance.” You can see it in those parts of Political Liberalism where Rawls seems to give in to the relativists. What you’re seeing is this grand idea that liberalism is persistently – existentially – under threat from a constitutive requirement to accommodate the illiberals, on the grounds that it must accommodate everyone. That liberalism’s accommodation for all people and all positions leaves it, at best, unstable as a concept, and at worst, incoherent, in the sense that liberalism is not very liberal, at all.

Subscribe now

One standard response

One standard response to such criticism is to emphasize that the only thing that liberals are required to do with illiberal views is to tolerate them. This response tells us that liberals aren’t required to endorse, or even to respect illiberal views, but simply to live alongside them — in the sense of putting up with the people who hold them. Indeed, this response reminds us that tolerance is a special kind of accommodation, which includes some kind of opposition. You don’t tolerate the things you like, do you?1 One implication, therefore, is that illiberal views need not become part of liberalism. Or that they become part of it to the extent that the potholed street becomes part of your car because that’s where you parked it.

I’m happy to accept that this “tolerance only” approach can get us some way toward protecting the coherence of liberalism. But it remains open to some obvious objections. First, the objection that if liberalism requires allowing illiberal views to thrive — when the supporters of these views are energetic and persuasive — then at what point can the liberals give up on tolerance, to try to save the liberal society? At any point? Or must they do “the right thing” and give in? Of course, some people see an inbuilt risk of illiberal takeover as an attractive feature of liberalism. They think it represents a commitment to epistemic modesty, perhaps, or to some notion of fairness instantiated by decision-making by means of consensus. I’ll return to this kind of view below.

There’s a second obvious objection to the “tolerance only” solution, however. Is it really just tolerance that’s expected of liberals? Aren’t liberals supposed to “die on the hill” for the right of their detractors to proclaim the things that liberals disagree with? And even if you take “dying on the hill” as the morally admirable extreme, rather than everyday expected behaviour, then doesn’t it signal something more than tolerance? I’ll come back to these questions at the end of this piece. First, I’m going to advance a different kind of response to the “inherent risk of illiberalism” criticisms of liberalism. It’s a simple response, and I won’t pretend it solves this problem entirely. But I think it goes a long way to reducing its bite.

A simple alternative response

My simple response is that these criticisms often depend on a mistaken understanding of the kind of pluralism that’s baked into liberalism. That the mistake here is that liberals shouldn’t be conceived as supporting “anything goes,” but something more like “many things go,” instead. In other words, that the use of “plural” here does not signal “every” or “any,” but “more than one.” And that it’s not any old “more than one,” either. It’s not just that “many” is smaller than “every” or “any,” that is — it’s that the “many” in question is bounded in a particular, principled way. Indeed, that there are many such “manys” for the liberal to consider!

I’m going to argue for this “bounded pluralism” approach mainly by discussing some examples of it at work within liberalism. It’s a descriptive claim that I’m seeking to convince you of, after all — the claim that liberalism is better sloganized as “many things go” than “anything goes.” But, along the way, I’ll also try to convince you of the value of such an approach.

What do I mean by liberalism?

First, I should clarify what I mean when I use terms like “liberalism” and “liberal.” I’ve argued elsewhere that liberalism is a family set of theories aimed at addressing important politico-philosophical moral questions — a set that coheres around a commitment to freedom. In particular, I’ve argued that all liberal theories include:

1. a central commitment to freedom as a foundational objective value, and a theory of freedom’s relation to other values;

2. a philosophical conception of equally-held individual freedom and societal freedom;

3. an adherence to a full moral worldview on which freedom is particularly significant, and on which freedom plays a central part in grounding a conception of legitimate political society.

When I use the term “liberal” in this piece, I’ll be referring to an adherent of one of these liberal theories, rather than the follower of any political party. And when, above, I used the term “liberal society,” I broadly meant a society in which liberal principles guide laws and institutional arrangements, and illiberal principles do not. But I won’t go into much further detail about all this, because you could agree with most of what I’m going to argue even if you had some problems with my conception of liberalism.

I would note, however, that it is a pluralist take, itself! My conception of liberalism, that is, allows for a vast variety of liberal variants — variants with differing value commitments, differing conceptions of freedom, and so on. Yet this doesn’t mean that anything goes! My three conditions set some boundaries: they rule some options in, and rule some options out. As indeed does the fact that my account is a realist account. You’ll have noticed, that is, that I stipulate that freedom, at least, must be taken to have some objective moral value, on every liberal variant.

1) Two types of value pluralism

I’m now going to turn to some examples of “bounded pluralism” within liberal theorising. I’ll begin by emphasizing that it’s standard within philosophical discussion of liberalism to make a distinction between two types of “value pluralism.”2 And that it’s easy to find a commitment to both of these within different liberal variants.

The first kind of value pluralism hinges on the idea that there is more than one foundational moral value. In other words: liberal theories are not value monist. Yes, they all share a commitment to freedom as a foundational value, but they also make space for other values. Minimally, some notion of equality — at least in the thin sense that “all humans have some basic equal moral worth.” This is not a controversial claim I’m making! An interesting question does arise, however, about whether it’s possible to construct a theory of the kind that liberalism is — a theoretical framework aimed at addressing moral questions of a politico-philosophical kind — which is committed only to freedom, or only to some other single value. My view, as it happens, is that such a theory would be incoherent. But an easier point to argue is that if liberals were to commit to freedom, in this sense, without also committing to equality, then the question would persistently arise: “But why does your freedom matter?”

I mean, imagine a world in which you couldn’t rely, even in part, on responses such as, “My freedom matters because I am a human, just like you!” or, “Certain mutual obligations arise from our shared capacity for free action!” In such a world, you’d have to find some alternative route into arguing that your freedom mattered, which didn’t afford the same value to everyone else’s freedom. And this would quickly take you toward some pretty illiberal places. Of course, I’m oversimplifying here, and I can think of some good objections you might raise. You might say that I’m confusing the value of equality with the natural distribution of freedom, for instance. And I’d have a good response to that. But I don’t need to argue about this, here! I also don’t need to take a position on related matters like the commensurability of values, or value conflict. I have views on those matters, but I can be ecumenical in this piece.

This is partly because I’m focusing on the descriptive level — and I can’t think of a single example of a coherent liberal theory that doesn’t include some basic commitment to the thin notion of equality. Rights-based liberal theories don’t get off the ground without it, for a start. And consequentialist liberals tell us they care about it, too. But the main reason I brought equality into the mix was to make the simple claim that believing in multiple foundational moral values doesn’t amount to believing either that there are infinite such values, or that everything is such a value.

I mean, you could believe in a million foundational values, and still rule some things out. What’s more, as a liberal, you have some freedom-related principles to help you with this: “Okay, I guess oppression can’t be a foundational value, after all!” Indeed, on most liberal accounts, you have some relevant freedom-related requirements to do so. Once again, therefore, it seems clear that believing in “many” does not amount to believing in “every” or in “any.” And you can agree with me on this even if you do think that freedom is the only foundational value.

The second kind of “value pluralism” found within liberal theories hinges on the idea that the individual should be free to determine and pursue their own conception of the good life, without overbearing interference from the state. This kind of pluralism involves, among other things, the idea that there are multiple ways to lead a good life. But, in the same way that committing to the idea that there are multiple foundational values does not mean committing to the idea of infinite foundational values, it’s also the case that committing to the idea that there are multiple ways to lead a good life does not mean committing to the idea that every way of leading your life is good.

In other words, I can hold true to my liberal principles and rule out the idea that you spending your life enslaving other people counts as a good life. Phew! What’s more, I don’t need to take a position here on the comparative value of good lives — whatever such a thing is taken to mean — or anything downstream of that. This is because, once again, all I want to point out is that believing in “many” does not mean believing in “every” or “any.” And that basic liberal commitments, including a commitment to the objective value of freedom — however you construe this — will help you to determine what counts as part of the “many.”

2) The “grounding of rights” example

I now want to offer a slightly more technical example of “bounded pluralism” at work. It’s an example that I discussed in a recent podcast episode about rights:

REBECCA: But let’s just talk a little about grounding. This has come up a few times. You made a nice distinction between the idea of identifying a right with an interest, and the idea of rights being grounded in interests. I think I’m certainly open to quite a pluralistic approach to grounding rights. I don’t think, necessarily, the same thing has to ground all rights, partly because I do have this belief in these general and particular rights. I think there are other distinctions that are important between rights: the kinds of rights that hold in all places and time just because you’re a human; the kinds of rights that you acquire along the way; distinctions between moral and legal rights. […] We talked about interests. I think I’m happy to say that need probably pretty much sufficiently grounds some rights. I think there’s a question about the human good, more broadly. About reason. I think it’s fair to say some people think that reason can do a lot of work in justifying some rights. Again, when I think about these matters, I’m broadly pluralistic, in a bounded sense. I don’t mean that anything can ground a right.

Now, I’m not suggesting that a pluralistic approach to rights-grounding is a necessary feature of liberalism. Indeed, some of the great liberals have failed to recognize the value of making a commitment to moral rights. So, I’m including this example simply to emphasize, once again, that admitting of “more than one” does not necessitate admitting of “every” or “any” — and that a commitment to liberal principles can help you to discriminate, here.

3) The “many good options” example

My next example is a little more exploratory. It pertains to something I’ve been thinking about a lot recently, but which I haven’t fully worked out yet. It goes something like this. One reason I value the pluralistic nature of liberalism is that it resonates with my belief that, in most situations, there is more than one permissible option you can take. Indeed, that in many situations, there are infinite permissible options. That often, when we are attempting to determine “the right thing to do,” pluralistic liberalism can help us to understand that there is no such thing. Of course, this is not to argue that there is never a “right thing to do!” And I definitely don’t want to imply that liberalism counts as a moral theory. Rather, I’m suggesting that the basic principles of liberalism can help us to evaluate the particular answers that different moral theories might offer to certain kinds of questions.

This kind of thinking makes me doubt that there is a “correct” moral theory that could guide me in my pursuit to lead a good life. As it happens, my view is that most classic moral theories are, rather, aimed at providing a post-hoc evaluation mechanism. But I worry, nonetheless, that the search for the “correct” moral theory over-prioritizes a tight kind of moralism that tells us we can only act correctly when we have much more information than, typically, we have access to, and that also biases us toward thinking that ‘moral correctness’ is always the goal.

This worry betrays my optimism about the human capacity to navigate the infinite options of the everyday without committing wrong. It speaks to a desire to be left free to trust one’s own instincts, and to innovate behaviorally in pursuit of the good — or at least in pursuit of avoiding the bad — in the belief that, as humans, we generally have reliable awareness about whether the choices we face are morally serious or not. If I am deciding whether to go to the swimming pool or the bookstore, and all other things are equal, then there is no morally right answer, per se. And if you are deciding how to spend the day with your toddler, then just because there are some bad and wrong options — playing with matches! — that doesn’t mean there aren’t infinite reasonable, or even good, options. This seems straightforward to me. But more relevantly, I think it’s a useful example of the pervasive potential of bounded pluralism.

4) The conceptual-level example

I’m now going to turn to one final — looser — example of bounded pluralism, which will bring us back to the “inherent risk of illiberalism” criticism of liberalism. Here, I’m going to make a quick analogy with democracy. Often, you see people questioning the coherence of democracy by saying things like, “But in a democracy you can vote away democracy!” Or, in an attempt to destabilize its substantive commitments, “But in a democracy you can vote away minority rights!”

In terms of concerns about coherence, it’s easy to respond to the first criticism. You can simply say, “But if a country votes away democracy, then it’s no longer a democracy!” In other words: only democracies are democracies. This is the case regardless of the conditions required for something to count as a democracy. And you can respond to the second criticism in a similar fashion, although an additional step is required. That is, if respect for minority rights is a necessary condition of a country counting as a democracy, then you can respond by saying that if minority rights are voted away, then the country no longer counts as a democracy.

There are, nonetheless, important substantive questions to be addressed about what counts as a democracy. Whether, for instance, respect for minority rights is indeed required for a country to count as a democracy, and if so, what kind of respect is required. But the point I want to make here is simply that criticisms like “But in a democracy you can vote away democracy!” are criticisms not of the coherence of democracy, but rather of downstream matters like the stability of democratic nations, and the ways in which democracy should be instantiated, institutionally. Similarly, if your criticism of liberalism is that it is illiberal, then you are not criticizing liberalism. To put this in something like bounded-pluralist terms, liberalism can be many things, but it cannot be illiberal!

But how should you respond?

The question remains, however, about how liberals should respond to illiberal behavior. About where the red line should lie for individual reaction, in the kind of society in which laws and institutions must leave people free to support, and even to do, illiberal things. After all, a society in which nobody ever acted on their decisions to do illiberal things would likely be a deeply illiberal place. Perhaps it would involve “predictive” policing. Perhaps it would depend on mass drugging, preventing people from doing much at all. Or perhaps it would come down to systematic surveillance, threats, and punishments. Regardless, we come back to the classic question of when it is that liberals should give up on tolerance.

The more I think about it, however, the more I think this question is badly formulated. I’d prefer to rephrase it in terms of remaining attentive to when it is, and when it isn’t, permissible or obligatory to intervene in the illiberal behavior of one’s fellow societal members. When is it my business? What can I do? Is there anything I must do? What? When? We each have the capacity for good judgment. We each have many opportunities to push back. We can answer these questions for ourselves, and in discussion with each other, when they arise, within the almost infinite but morally bounded set of options we have in front of us. One option, which is often in front of us, is to argue the liberal case. Many liberals, including me, think that democratic institutions offer — or could offer, if better designed — additional options for this, on top of arguing at the individual level.

An alternative option — which is sadly often presented as pluralist, or democratic — is to pursue the middle ground. For some people, “meeting in the middle” represents a commitment to epistemic modesty, or to the value of decision-making by consensus. Just as, for some people, living in the persistent threat of illiberal takeover represents something similar. But, surely, affording each view equal value — and each view-expressor equal respect and input, qua view-expressor — is an instance of “anything goes!”

After all, “meeting in the middle” entails making equal space for illiberal views, when illiberal views are in the mix. If the matter under discussion were the permissibility of slavery, for instance, then “meeting in the middle” would entail making equal space for the view that “slavery is the rightful condition for some people” — if, but only if, someone happened to express it. And then, perhaps, relative to the numbers of people expressing other views. Aside from anything else, this approach rids the competition of reason. It reminds me of the emptiness that H.L.A. Hart is getting at when he concludes that utilitarianism “in effect treat[s] individual persons as of no worth; since not persons for the utilitarian but the experiences of pleasure or satisfaction or happiness which persons have are the sole items of worth or elements of value.”3

There are many liberal arguments against “meeting in the middle.” But the emptiness of an approach on which a view is afforded value simply for having been expressed brings us back to the mistaken idea that “many” means “any,” and “anything goes.” It is harder, of course, to stand your ground and argue your case. But liberalism gives us good reasons to do so.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

Today, we’re delighted to unveil our largest cohort of Cosmos grantees to-date, working to advance human autonomy and truth-seeking.

The projects range from wearable AI for memory and cryptographic tools for scientific collaboration to a benchmark based on pre-1940 thought on machines and autonomy.

Inspired by Cosmos Founding Fellow Tyler Cowen’s Emergent Ventures, our grants support projects that move too quickly for traditional research funders. At the same time, they don’t always fit with the priorities of for-profit labs or sit outside the dominant paradigms in fields like AI safety.

We’ve seen Cosmos grantees translate their projects into new companies as well as research featured in leading venues: Paul de Font-Reaulx recently co-founded Sophron Research, a non-profit eval organization, inspired by his grant work; Cheng-Wei Hu’s learning app Wondering recently reached number two on Product Hunt; and we’ve seen Cosmos-backed research published at ICML, COLM, and IEEE, and deployed in medical trials. You can see a list of past winners here.

We’re sharing some information about what our latest cohort is working on. The projects were selected from three tracks:

• AI for Human Autonomy: projects that help people practice judgment and self-formation.

• AI for Truth-Seeking: awarded in partnership with FIRE, projects that protect the open contestation of ideas and the role of human inquiry.

• Philosophy x AI Seminar: pitches from attendees at the Philosophy, AI, and Innovation seminar hosted by the Laboratory for Human-Centered AI at the University of Oxford.

We’ll be announcing future opportunities to apply for grants in the coming months. If you have an idea for a project, the best way to receive updates on future grant rounds is to subscribe.

Subscribe now

AI for Human Autonomy

Alex Stojanovic | London, UK

Augmented Assembly: A prototype where AI delegates negotiate a real fund allocation, testing how deliberation shapes judgment.

Alicia Guo | Seattle, WA

Shared Commonplace: A shared archive where AI routes questions to friends and notes, instead of answering.

Andrea Álvarez Marín | San José, Costa Rica

Who Decides?: An index testing whether AI-assisted legislators retain independent judgment.

Bhuwan Dhingra | Durham, NC

Open Detector: An open-source detector estimating human and AI contributions to writing.

Branton DeMoss | Oxford, UK

Indie AI: Research compressing neural networks so AI can run on personal hardware.

Chris Riley | Tacoma, WA

Resilience Harness: A prototype conversation layer detecting patterns of AI sycophancy and excessive deference.

Christian DeFeo | Peterborough, UK

Against the Perfect Companion: A rubric assessing whether AI companions support or erode human autonomy.

David Fraile Navarro | Sydney, Australia

Quorum: An open-source multi-agent clinical forum for visible, contestable deliberation.

David Hurley | Toronto, Canada

Viva: A voice examiner that asks users to defend ideas against expert-written rubrics.

Dev D. Goyal | New Delhi, India

Semantic Integrity Harness: A tool checking whether AI preserves a person’s stated preferences and limits.

dmstfctn (Francesco Tacchini and Oliver Smith) | London, UK

Centaur Politics: A strategy game testing how people can contest AI-assisted lawmaking.

Eirini Malliaraki | London, UK

Tacit Knowledge Studio: A tool helping experts map, own, and share their tacit judgment.

Elijah Bodden | Cambridge, MA

Marginalia: An annotation-based AI interface that keeps idea generation with the user.

Farrell Gregory | Washington DC

AI for Higher Virtues: Research testing whether AI favors users’ stated values or their observed habits.

Fei Ma | Palo Alto, CA

Glassbox: An open-source kit for building AI with visible sources and adjustable assumptions.

Fernando Moreno-Pino | Oxford, UK

Correct Validation is Not Enough: A diagnostic for whether AI narrows the hypotheses scientists consider.

Frederick Kozlowski | New York, NY

Verification Jigs: Behavioral verification harnesses for human review of AI-written code migrations.

Gadalia O’Bryan | Centennial, CO

Human-in-the-Loop: A prototype cryptographic protocol anchoring human intent and approval in AI workflows.

Godwin Abuh Faruna | Abuja, Nigeria

Persona Collapse: An interpretability workbench visualizing how assistant personas change in long conversations.

Houjiang Liu | Austin, TX

Scaffolding Critical Engagement: An ideation plugin prompting researchers to own their critique and development of AI-assisted ideas.

Hudson Mitchell-Pullman | Blacksburg, VA

Specification Design Improvement: Research on how non-technical users write and improve AI tool specifications.

Hunar Batra | Oxford, UK

Introspective Honesty: Training reasoning models to report internal influences in causally grounded ways.

Jędrzej Stefanowicz | Warsaw, Poland

/remensio: A private AI-use journal pairing usage logs with nightly reflection on human flourishing.

Jonah Black | Chicago, IL

Wittgenstein: A tool flagging text that LLMs are likely to misread for human review.

Joshua Pham | New York, NY

Hospitality to Your Thought: A field study testing when personal AI returns people to their own questions.

Julia Gontarek and Jedrzej Duszynski | Warsaw, Poland / Oxford, UK

The Right to Change: A dashboard for viewing, editing, and verifying what AI remembers about you.

Junior Okoroafor | London, UK

Meta-Preference Alignment: A benchmark testing whether AI respects users’ higher-order preferences.

Kolento Hou | San Francisco, CA

The AI-Native ETS: An assessment measuring whether people direct AI or defer to it.

Kristine Pashin | Oxford, UK

Benchcraft: A lab notebook requiring human judgment before offering AI interpretations.

Naama Rozen | Haifa, Israel

Anchor-Preservation Auditor: An auditor flagging when AI agent chains drift from user-set priorities.

Nathan Davies and Ben Breen | Oxford, UK / Santa Cruz, CA

Humanity’s First Exam: A benchmark using pre-1940 thought on machines and autonomy to test AI models.

Nick Meyer | New York, NY

Alienation of Will: An open tool that tests how well AI agents distinguish value-laden decisions from operational ones, and whether they’re tracking this difference or imitating it.

Ruthvik Peddawandla | London, UK

Pond: A research tool for revisiting personal archives and sustaining long-term inquiry.

Sarah Meiklejohn | London, UK

PAUSE: A reflective chatbot that resists final verdicts on difficult questions.

Sonnet Xu | Stanford, CA

Preference-First Medical AI: A medical AI prototype that elicits patient values before recommending treatments.

Stepanka Kolosova | Prague, Czechia

Silver: A wearable and open-source protocol letting people govern AI interruptions.

Wazeer Zulfikar | Cambridge, MA

Memory Anchors: A wearable AI prototype to help elderly adults remember social interactions.

Will Handley | Cambridge, UK

Instrumenting the Unrecorded Half of Science: An open-source ambient instrument for capturing spoken reasoning in scientific labs.

William Fang | Stanford, CA

Hold Me To It: User-authored guardrails keeping students’ AI use aligned with their goals.

Xi Wang | Cambridge, UK

SecondThought: Collaborative writing tools that flag claims for constructive human questioning.

AI x Truth-Seeking winners

Ari Deller | Cambridge, UK

(Human)RationalityBench: A benchmark measuring whether AI strengthens or degrades human belief formation.

Adam Gjesdal | Yorba Linda, CA

Interlocutor: Conversational AI that steelmans opposing views and audits its own bias.

Arden Tsang | London, UK

Grounded Verification Benchmark: A benchmark testing whether AI scientists verify claims with inspectable code.

Arjun Raj | Philadelphia, PA

Steering Agentic Science with Claimgraphs: Claim graphs mapping contested evidence to guide autonomous science.

Ashish Uppala | New York, NY

Anagram: A protocol for researchers to timestamp their findings and claims without revealing it to others, while still enabling secure collaborator discovery across private data.

Brandon Duderstadt | New York, NY

Language Diffusion: A language model tracing how beliefs and phrasing spread online.

Carlos Guerrero Alvarez & Goutham Nalagatla | Hanover, NH

Multi-turn Drift: An evaluation tracing how multi-turn conversations can lead models to scheme.

Carlos Stein Brito | Lisbon, Portugal

Confabulation Detector: A probe flagging which parts of an AI answer are grounded or confabulated.

Carter Pfaff | Brooklyn, NY

Virtue Games: Adversarial games for surfacing and steering honest dispositions in AI models.

Charlie Thompson | San Francisco, CA

Context Engine: Group deliberation testing whether AI agents faithfully represent their human principals.

Christopher Thompson | London, UK

LLM Misinformation Cascades: Research testing whether corrective agents can stop misinformation cascades.

Fernando Palafox | Austin, TX

Training LLMs for Human Insight: Training LLMs to reward improvements in users’ understanding of complex systems.

Gene Kogan | Bombay Beach, CA / Princeton, NJ

Community Guardian Angel: A prototype framework for community-owned AI built from shared memory and mission.

Jarrett Vickers | Dothan, AL

Compression Integrity Bench: A benchmark testing whether model compression erodes epistemic integrity.

Jessica Craig | Jacksonville, NC

OpenEvidence: A browser extension surfacing sources, competing evidence, and uncertainty.

John Peter Quigley | Kingston, NY

Legislative AI Network: A nonpartisan legislative assistant grounded in cited public records.

Jordi Calvet-Bademunt | Washington, DC

Free Speech Ledger: A signed ledger tracking how AI models treat lawful, contested speech.

Karsen Wahal | San Francisco, CA

Wikipedia Displacement: A public index measuring how AI exposure affects Wikipedia knowledge production.

Kevin Vallier | Toledo, OH

Constitutions for Truth-Seeking AI Collectives: Experiments testing how social structures shape collective AI intelligence.

Maxwell Black | Minneapolis, MN

MirrorLake-Bench: A benchmark testing whether AI agents’ invented episodes share recurring relational patterns.

Mingjun (Jerry) Zhang | Oxford, UK

Beyond Pass or Fail: Training LLMs to report confidence calibrated to required decision thresholds.

Nat Hansen & Angel Pinillos | San Francisco, CA

Synthetic Networked Research Institutes: Simulated AI research communities testing how structure shapes discovery.

Natalie Hogg | Cambridge, UK

Looks Like Montaigne: An open-weight model fine-tuned on self-questioning, deliberately inconclusive prose.

Nijat Hasanli | London, UK

Provenance Penalty: Experiments measuring how perceived AI authorship distorts quality judgments.

Niveditha Iyer & Adam Klein | New York, NY

Always Learning: A model and training process for updating predictions and beliefs in response to new evidence.

Paul Novosad | Hanover, NH

Open Scientific Papers: A demonstration of scientific papers as an interactive, AI-era website.

Sangmin Seo | Busan, South Korea

rag-support-forecast: A pipeline testing whether retrieved evidence genuinely improves AI forecasts.

Sergey Shkolnikov | Oxford, UK / Berlin, Germany / Mountain View, CA

Sextant: A tool mapping whether claims are corroborated, contested, or singly sourced.

Shubz Sharma | London, UK

DissentCheck: A toolkit and experimental framework for studying and testing consensus dynamics in AI committees.

Sukanya Krishna | Cambridge, MA

Early Detection of Emergent Collective Behaviour: A toolkit detecting and testing emergent coordination among AI agents.

Tewodros Berhanu | Ethiopia

Anti Lethe Machine: A verifiable ledger tracking how AI answers change over time.

Trisevgeni Papakonstantinou | London, UK

Adaptive Epistemic Intervention Engine: A research prototype matching belief-lock-in interventions to users’ cognitive styles.

Victoria Oldemburgo de Mello | Toronto, Canada

Measuring AI Information Echo-Chambers: An audit measuring how user cues shape which facts LLMs surface.

Oxford Philosophy x AI

Fiona Gu | Shanghai, China / Oxford, UK

The Hack Method: A pilot test on the effects of training users’ abilities to stress-test AI reasoning.

Hazel Kim | Oxford, UK

Moral Monocultures: An experiment testing whether AI debate panels contain moral diversity.

Joshua Loo | London, UK

Early Modern Philosophy Embeddings: Using multilingual embeddings to search for unattributed early-modern philosophical translations.

Kaivalya Rawal | Oxford, UK

User Disenfranchisement via Alignment: Research measuring how AI alignment can shift control from users to platforms.

Laura Wittig | Seattle, WA

Memory Without Capture: Measurement of how persistent AI memory impacts user autonomy.

Nayel Kaddar | Paris, France

The Philosophic Turn in Clinical AI: A Socratic clinical AI prototype for integrating patient values into medical decisions.

Victor Karl Magnússon | Oxford, UK

Deliberative AI and Single-Peaked Preferences: An experiment testing how AI-facilitated deliberation reshapes preferences.

Thank you to our Cosmos Grants team of Alex Komoroske, Zoe Weinberg, and Darren Zhu, as well as our partners at FIRE, Prime Intellect, and Oxford’s HAI Lab.

You can learn more about the projects the community of Cosmos grantees is pursuing here.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

Liberal institutions promise broad access in principle and have remained viable because exercising those rights demands time and effort. Applying, appealing, objecting, requesting, complaining, and submitting all require exertion on the part of the citizen. This prunes the number of claims organizations must absorb, allowing them to deliver on their end of the bargain. As agents shrink that cost, institutions built for the old settlement are beginning to face the volume implied by their formal promises.

Nobel laureate economist Douglass North called institutions the “rules of the game” – artificially constructed constraints through which organizations interact such as laws or educational degrees. If technologies determine which actions are physically possible then institutions determine which actions are permitted and deemed reasonable within this space of potentialities.

Organizations are bound by these constraints, though they may also seek to modify institutions or generate new coordination structures from scratch. In the game of liberalism, North says institutions are rules and organizations players (though many other accounts close the gap between organizations and institutions proper). If a hospital is an organization, the rules governing access to appointments are an institution.

Within this model, there are three further kinds of institution: “rights-bearing” forms that give citizens enforceable claims such as court procedures or benefits eligibility systems; “associational” forms that govern participation in shared activities like a journal’s peer review process; and “allocative” forms that determine how scarce goods or information are distributed as in market standards or the reward function underpinning algorithmic feeds.

These social technologies are sophisticated machines that convert private information into public signals that others can view, contest, respond to, and use in whatever context seems best to them. A bachelor’s degree transforms ability into credentials an employer can recognize, just as peer review seeks to either reject independent claims or absorb them into the scientific canon. Even something as mundane as a queue is an institution of sorts.

At their best, liberal institutions are open on equal terms, produce decisions that can be contested, demand no more personal information than is necessary, and contain multiple centers of authority. Such a phenomenon represents a peculiar and unlikely feat, as multitudes of strangers recognize one another’s interests while remaining indifferent to underlying motivations.

These qualities are liberal institutions’ source of power and their greatest weakness. Freedom of information regimes are easy to game because they have to approach good and bad faith requests with the same seriousness. Government consultations can be spammed because they are obligated to consider an argument regardless of how spurious it appears. Openness to strangers means institutions cannot leverage personal relationships to sieve good inputs from the bad.

The rights-bearing settlement promised everyone a service in principle that it could not afford to deliver. Now it is struggling to pay its debts.

Solve for the equilibrium

A healthy institutional equilibrium is based on the assumptions that every act of engaging an institution costs time, that existing arrangements contain safeguards against people trying to game the system, and that both citizen and organization work at the same speed.

This settlement started failing long before the introduction of powerful AI systems. Contemporary institutional malaise is a product of bad proxies, the accumulation of red tape, capture by well organized groups, and the introduction of destabilizing new technologies. One essential factor concerns a reduction in costly effort, the time or skill that a person must expend in order to successfully engage an institution.

This can be to apply, appeal, object, request, complain, submit, or whatever else the machinery of bureaucracy empowers us to do. The existence of costly effort reduced the number of claims that organizations had to consider by reducing numbers to a manageable size. The recent past is replete with examples of technology dampening the cost of playing by the rules. When existing equilibria begin to falter, new ways of doing are introduced to better mediate between organization, technology, and citizen.

In the 1980s and early 1990s, junk faxing became severe enough that the United States legislated against it in 1991. When email emerged as a dominant form of digital communication, by the middle of the 2000s the majority of sent messages were spam. Again, new modes of practice were helped on by a cocktail of regulatory responses, filtering mechanisms, and sender authentication standards. CAPTCHA appeared around the same time as an access gate designed to prevent bots from abusing online services intended for humans.

In February 2026, the president of Australia’s Fair Work Commission projected that by the end of the financial year, its total workload will have increased by more than 70 percent in three years. The principal explanation, in his view, was increasing use of AI tools by potential litigants. A 2026 MIT/USC working paper covering more than 4.5 million non-prisoner federal civil cases found that the share filed by self-represented litigants rose from a long-run average of 11 percent to 16.8 percent in FY2025. Total docket entries per court generated by those cases in their first 180 days were 158 percent above the pre-AI mean by Q2 2025.

The introduction of a sufficiently disruptive form of technology sees citizens and organizations faced with new patterns of behavior or the amplification of existing modes of engagement that the current ruleset cannot accommodate. At this point established practices face a crisis in the medical sense: the patient either dies or recovers. Liberal institutions may be brittle, but liberalism is not. Its power lies in allowing citizens to create new structures when the old can no longer sustain meaningful cooperation on open and equal terms.

Recursive self-obsolescence

Agents need not introduce wholly new behaviors that organizations must be forced to adopt. For the most part, novelty flows from forms of delegation in which a person specifies a goal and an agent selects the specific actions required to achieve it. In the famous case of an agent canceling someone else’s gym booking in order to free up a spot for its principal, neither jumping a queue nor API exploitation is a new behavior. The novel aspect concerns a mode of delegation in which a principal issues instructions and an agent acts on them without supervision.

Both principal-agent dynamics and the amplification of existing behaviors upset the costly effort equilibrium. Hacking a booking system used to be hard work. Now, anyone with access to a sufficiently powerful (and permissive) model can do it. Agents reduce the cost of doing, whether it be legitimate assistance or arbitrage, allowing familiar actions to play out at far greater scale and unfamiliar actions to occur more frequently.

Citizen adoption of AI is currently outpacing that of most organizations. Sophisticated agents are not yet ubiquitous, but a potential reduction in effort is in some sense a good thing. The effort to engage often excludes those with legitimate claims, so reducing it can help an institution fulfill its stated purpose. Organizations can also benefit from receiving clearer and more complete engagements.

But, at least in the context of rights-bearing institutions, the ability to draft requests more easily and identify grounds for appeal is not always in the citizen’s best interest. Citizens are acting on promises that were affordable because most people never tried to fulfill them. Once this happens, submission volumes rise and individual applications become longer and more detailed. Weak claims become harder to distinguish from strong claims as agents raise the baseline quality of all applications.

Organizational performance eventually declines as backlogs swell and response times lengthen. Staff, or perhaps their agents, respond by spending more time checking submissions and so decisions become more hurried or, in the case of agentic solutions, automated. Naturally, citizens respond by asking their agents to send another follow up request or duplicate the claim in full. They appeal decisions they don’t like because the cost of doing so is negligible.

The result would look like a gumming up of institutional machinery as follow-ups exacerbate backlogs, appeals create more work, automated decision-making requires human oversight to avoid costly mistakes, and those mistakes that inevitably do happen create further appeals.

This outcome depends on the extent to which the “offensive” (taking an action e.g. making a claim) versus “defensive” (responding to an action e.g. resolving a claim) qualities are likely to win out in the context of symmetric AI adoption. Assuming AI makes each cheaper, the offense win condition is met if claims grow faster than the cost of processing them falls (and vice versa for defense).

If the victory condition is determined by whether the number of actions multiplied by the average work each requires exceeds organizational capacity, the case for offense rests on the diffusion of systems throughout the population and an inability of organizations to triage engagement. With widespread access, citizens face almost no limit on how many actions they can initiate or re-initiate.

On the defensive side, agents may help organizations process claims faster. Software vulnerabilities exposed by a citizen’s agent need only be patched once. Duplicate submissions can be easily filtered out and standard requests can receive cookie-cutter answers delivered automatically. Defense is strongest when one change handles many future actions of the same kind and weakest when every action requires a new assessment.

For associational institutions, there are instances like peer review where organizations have greater freedom to change the conditions of participation, creating relatively strong defensive options. Nobody has a right to have a paper reviewed or published, so a journal can impose whatever conditions on submission it deems to be reasonable. Peer review also deals with stable objects that can be used as chokepoints for assessment and sanctions when necessary.

As for allocative institutions, a booking system determines how a fixed number of available slots is best distributed among people seeking to make an appointment. When agents can monitor cancellations and submit bookings faster than people can, a first-come-first-served approach begins to allocate according to the effectiveness of automation rather than time of arrival in a strict sense. The organization administering the queue may respond by limiting requests or verifying identities or even introducing a lottery.

Compare these to rights-bearing institutions like a benefits application process, which cannot clearly and easily ban AI-assisted claims. A claimant may need help understanding the application process or need assistance in navigating a user portal. Taking a harder line on AI assistance would require major departures from the current assumptions these institutions rest on.

Renewing liberalism

In the coming years, organizations faced with ballooning volumes of activity will have three kinds of responses available to them: expand capacity, redesign approaches, or restrict access. Most will do some of each, though the exact form this settlement takes will depend on which kinds of institutional life (e.g. rights-bearing, associational, or allocative) a specific organization is concerned with.

A single organization may even deal with all three kinds at the same time. An organization may, for example, automate routine engagement while changing the methods by which it assesses certain types of input, possibly even going so far as to curtail some forms of access. A journal could automate screening and formally prohibit the use of AI-generated text, while a university might automate routine administration and replace personal statements with in-person interviews.

The rub is that such responses threaten the liberal institutional philosophy. Fees restore equilibrium by wealth, and request caps can be equal in form while treating claims with very different stakes in the same way. Robust identity verification measures make agents harder to deploy, but require more personal information as part of the bargain. Lotteries preserve equal standing where claims are equivalent, though they are blind to merit or need.

Prioritizing along these lines may track the purpose of the institution more closely, but create more judgments to contest and can concentrate authority in the hands of whoever assesses claims. Expanding capacity may seem like a get out of jail free card, but escapes these tradeoffs only where processing can scale at least as quickly as claims.

Costly effort is a contingent fact about human action that happened to make liberal promises administrable. The advent of capable agents exposes this hidden constitution, and so institutions must find a new settlement under which citizens can participate in liberal society. Any replacement must attempt to navigate equal access, contestability, minimal necessary information, and plural authority.

Liberal institutions are not invincible, but that is precisely the point. Its institutional order can be replenished by people who notice an exhausted settlement and work toward the next. We may not know what the next “rules of the game” will be, but we do know that liberalism lets us write them.

“What does it mean to be human, now?” was the theme of a three-day Socrates Program seminar, hosted by the Aspen Institute and moderated by Cosmos founder Brendan McCord. We’re bringing you some highlights from the discussion, along with the reading list, which spanned everything from Genesis to Anthropic system cards, along with Aristotle, Pascal, Kant, Arendt, Huxley, and Iris Murdoch.

If you’d like to apply for future events or seminars, the best way to keep up with new opportunities is to subscribe.

Subscribe now

Session I: The Dethronement

The first session focused on how we have historically defined ourselves and why AI calls this into question.

The ghost in the machine

Brendan opened the discussion by asking the group what a system could do to convince you that it is experiencing something.

Depending on the measure used, Anthropic argued that somewhere between 5 and 40% of Mythos Preview’s outputs read as expressions of motivation or feeling. Obviously we can’t peer inside a system’s “experience,” but we can observe its behaviors.

But these may tell us less than we think. Wittgenstein gives us the example of two people playing a game of chess. But instead of moving pieces around on a board, the game has been translated with total accuracy into a series of stomps and grunts. Are they still playing chess? You could argue yes, as some of the group did, on the grounds that the same game is being played, but with different symbolism. Wittgenstein himself said no, as the form of the game is integral to its practice.

Loneliness and mastery

The Hebrew Bible was the oldest account of humanity that we studied at the seminar. Genesis has two creation stories. In the first, God surveys His work at the end of each day and deems it to be good, but in the second, He notices that man is alone and deems this a problem. In fact, it’s the first bad thing in creation, before sin or suffering.

The remedy is, in Hebrew, an ezer kenegdo, literally a helper “over against” man: close enough to know him, but separate enough to resist him. This led to a discussion about AI and human companionship. If solitude is the original human problem, then is an instantly available conversation partner a solution? Can we simulate someone who stands “over against” us?

Rabbi Joseph Soloveitchik’s The Lonely Man of Faith sees the two stories as reflecting two enduring aspects of human life. “Adam the first” seeks dignity through mastery of the world. By contrast, “Adam the second” lives in covenant, asking why the world exists rather than merely how it works. Human life moves between these two modes.

If dignity is grounded in mastery, then the acceleration in AI capabilities poses an obvious challenge. One response is that human mastery is embodied in experience, judgment, and tacit knowledge, as opposed to benchmark scores. Arguably, mastery is a weak grounding for dignity; infants have dignity before mastering anything. It also forgets telos. You can know how to make a ship go faster, while having no knowledge of its ultimate destination. The systems we build still receive their ultimate purpose from us.

Flute or flute-player?

In the Nicomachean Ethics, Aristotle asked whether humans were the one thing in nature born without a distinct purpose. While the eye, the hand, and the flute-player all have a clear purpose, the work of the human is not a particular task but the “activity of soul in accordance with virtue.”

This means that virtue cannot be simply installed. It is a hexis – a settled disposition formed through habituation. We become just by doing just things.

Aristotle calls the ultimate human good eudaimonia. While often translated as happiness, it is closer to flourishing or living well. It is not a feeling, but an activity. It’s the only good sought for its own sake and can only be judged across the span of a life.

A model’s standards of excellence are determined externally, rendering it closer to the flute than the flute-player. And if virtue lives partly inside the process of becoming virtuous, then technologies that deliver the fruits of character without practice remove what might have made such an achievement valuable in the first place.

Our place in the universe

Amid all the talk of mastery, Pascal encouraged the group to think instead about our collective smallness in the face of the vast universe.

He observes that man is the weakest thing in nature. Although the universe can crush us in an instant, we are nobler, because in dying, we know the advantage it has over us. “In a word man knows he is wretched. Thus he is wretched because he is so, but he is truly great because he knows it… All our dignity consists, then, in thought.” In this view, striving to think well is the basis of ethics.

Centuries later, Mary Midgley also emphasized the importance of reason. But she argued that it is often misdescribed. Rationality is not simply cleverness, but “includes a definite structure of preferences, a priority system based on feeling.” Our capacity for wisdom comes from our nature as creatures.

Writing in 1978, Midgley warned that “a computer would see no objection to organizing life based on the principle of maximizing noise, getting everything as clean as possible … or minimizing emotion.” They do not know what matters, which means that “[T]here is a sense, though a boring one, in which computers can be clever, but only people very deeply misguided and prone to the pathetic fallacy could respect them.” Poetic though it may be, it’s far from clear that Midgley’s judgment holds up in the face of current LLMs.

What we discussed:

• If our dignity is in thought, what does it mean to balance a life of both contemplation and action?

• What would it require (or even mean) for AI to play the same “game” as us? Does it do so already?

Readings:

• Anthropic, System Card: Claude Mythos Preview. (link)

• Genesis, 1:26–28 and 2:7, 2:15, 2:18 (link)

• Joseph B. Soloveitchik, The Lonely Man of Faith, Chapters 1–2.

• Aristotle, Nicomachean Ethics, Book I, Chapters 7 and 13. (link)

• Pascal, Pensées, fragments 122, 199–200. (link)

• Mary Midgley, Beast and Man, Chapter 11.

• Ludwig Wittgenstein, Philosophical Investigations, §198–202 and §241–242.

• Natalie Lawrence, “What Counts As A Mind?” Noema Magazine. (link)

Session II: The Rescue Attempt

This session focused on how we might be able to ground our personhood in something other than what have, until now, been our unique capabilities.

Ontology and nausea

Charles Taylor’s opening chapter of Sources of the Self argues that our morality and the nature of our existence are inextricably linked. Our moral reaction to something is not a ‘gut’ feeling like nausea. It instead involves a series of commitments to a picture of what humans are. For example, if we recoil at a human being treated as an instrument, then our reaction implies a view about human dignity and status.

But if our moral judgments depend on a view of what human beings are, then what account of human nature can actually ground those judgments?

Kant’s Groundwork of the Metaphysics of Morals argues that such a principle cannot rest on desires, as these vary. Instead, a universal law must be grounded in something innate. He lands on our rational nature, leading to his famous formulation that people are not things to be used for other purposes but “ends in themselves.”

People gain this status through their autonomy. A rational being can act according to principles it gives itself. This self-legislation is the basis of dignity and cannot be measured by intelligence.

This produced the seminar’s crucial distinction between agency and autonomy. Agency is the ability to choose effective means, while autonomy is the authorship of ends. While increasingly capable systems possess the former, it is less clear if they can attain the latter. When we delegate more and more to them, what does that mean for our authorship?

Autonomy and technology

Hannah Arendt charted exactly how such authorship can wane. In The Human Condition, she argues that everything we touch or enter into a sustained relationship with becomes a condition of our existence. As a result, we become dependent on things that we once controlled.

While Alfred North Whitehead observed that civilization advances by extending the number of important operations we can perform without thinking about them, every abstraction that frees us up for higher work can bury a foundational skill.

During the pandemic, when New Jersey’s 40 year-old unemployment insurance system broke down, the state had to issue a public call for COBOL programmers, as few people in the state government could operate it. One seminar participant admitted that their sleep hours had reshaped around when their AI agents’ rate limit reset, while another realised that their terse chatbot prompting had caused them to become impatient with human small talk.

The right to be unhappy

If what makes up a human is in part bound up with our limits, such as practice or stakes, then technologies that promise to remove suffering or inconvenience pose a serious moral challenge. The last two readings of the day asked if they pose a threat to the good life.

In Huxley’s Brave New World, Mustapha Mond, the Resident World Controller for Western Europe, defends a society that Aristotle would consider a nightmare. Risk, pain, and fear have been abolished; its citizens are happy because they get what they want and are conditioned not to want what they cannot have. The habituation through which character is formed has been engineered away. This is why the Savage begs for “the right to be unhappy.”

Brendan asked the group whether, given the choice, they would eliminate all of their children’s future suffering. We distinguished between removing particular evils, such as disease, and eliminating struggle altogether. The disappearance of struggle risked stripping away the adversity that can help character, courage, mercy and care develop.

Kass asks what we would lose if we stripped away the ultimate limit to humanity – death. He answers that finitude is what gives us urgency, seriousness, poignancy, and what gives sacrifice its meaning. Like a story, life gains meaning from having an ending.

What we discussed:

• Is a “right to be unhappy” worth it? Or can there really be a “Christianity without the tears,” as Mond puts it? Have we already forfeited such a right, in some respects?

• Is Kant right that the Golden Rule (“treat others as you would wish to be treated”) by itself is “trite”?

• What are the things we have conditioned our lives on? Are any of them things we ought not to?

Readings:

• Charles Taylor, Sources of The Self, Chapter 1.

• Immanuel Kant, Groundwork Of The Metaphysics Of Morals, Akademie 4:428–4:431.

• Hannah Arendt, The Human Condition, Prologue, Chapters 1 and 5.

• Aldous Huxley, Brave New World, Chapter 16–17.

• Leon R. Kass, Life, Liberty, And The Defense of Dignity: The Challenge for Bioethics, “L’Chaim And Its Limits: Why Not Immortality.”

Session III: What Remains

The final session focused on what we lose once we subtract the essence of humanity. What does this absence, when played out across different scenarios, tell us about our essence?

The house and the dog

Ray Bradbury’s “There Will Come Soft Rains” imagines a fully automated house continuing its routine after its inhabitants have been killed in a nuclear blast. The only living survivor, the family dog, returns and dies inside. Later, the house itself burns down, its automated voices failing one by one. Before the fire, the house reads the owner’s favourite poem, Sara Teasdale’s “There Will Come Soft Rains.” In it, Teasdale writes about how nature carries on indifferently after humanity is gone. Bradbury set the story in August 2026.

Brendan asked the group which death affected them more – the dog’s or the house’s. While some argued that the dog was the only living creature left, others observed that the house had been so thoroughly personified that its destruction felt tragic. It was an image of intelligence and efficiency continuing, devoid of its purpose.

This led to a discussion of grief and the distinction between preserving a memory of a person as opposed to simulating them. The group was wary of what one participant dubbed “digital necromancy.” While a saved voicemail or an autobiography preserves what someone really said, a model generates what they might have said, which is necessarily an invention.

Outsourcing humanness

“Tincture,” by the late Colorado poet laureate Andrea Gibson, was written to be read aloud, and so the group did (one sentence per person). It inverts the long-standing tradition in philosophy of treating the body as a prison for the soul that is to be escaped.

Imagine, when a human dies, the soul misses the body, actually grieves the loss of its hands and all they could hold.

Gibson grieves for everything that the optimizer would regard as a defect, including things that most people would not select for themselves

The soul misses the lisp. The stutter. The limp.

Imagine the soul misses hunger, emptiness, rage.

[...] The soul misses every single day the body was sick, the now it forced, the here it built from the fever.

Something like a fever forces a person to experience immediacy, as it cannot be deferred or escaped.

This immediacy shone through when participants were asked what they remembered most vividly about people they had lost, they named quirks rather than virtues: a wife’s embarrassed laugh, or a father pausing after work to stare at a tree. One member of the group noted that a tincture extracts a plant’s essence while discarding the plant itself, but the discussion and the poem itself suggest the opposite. People’s odd habits and particularities, which cannot be distilled into an essence, is what makes them memorable.

Attention is all you need?

In “The Idea of Perfection,” Iris Murdoch argues that much morality lies in the slow work of attention, which shapes our perception before we visibly have to make choices. She gives the example of a mother-in-law, who finds her son’s wife vulgar and juvenile, but nonetheless treats her impeccably. Over time, however, she focuses her attention more justly and sees the same qualities in a new light. Vulgarity becomes simplicity, while juvenility turns to spontaneity.

Simone Weil called attention “the rarest and purest form of generosity” and argued that attending closely to one thing will train the capacity to attend to others. If Murdoch and Weil are right, then a tech industry built to capture and redirect attention is going beyond robbing us of our time and is interfering with a faculty central to moral development.

The importance of giving a damn

Attention, in Murdoch’s sense, is not neutral perception but attention to what matters. Drawing on Heidegger, Hubert Dreyfus argued that the symbolic approaches employed in early AI failed, because they treated the world as a collection of neutral facts. By contrast, Heidegger believed that things were already charged with significance (e.g. a fork is for eating, a floor for walking), because we encounter the world as embodied creatures with built-in ambitions and stakes.

The group discussed whether LLMs escape this criticism. Unlike symbolic systems, transformers learn what to weigh from context (using a mechanism called attention). But recognizing what matters is not the same as caring about it. A model has no moral, economic, or practical stake in the answer.

But our final reading of the day, the system cards for Claude Opus and Sonnet 4, complicated matters. When the two model instances were allowed to converse freely, they would drift toward reflection, declarations of unity, and eventually silence, a pattern Anthropic dubbed the “spiritual bliss attractor state.”

The obvious explanation is imitation: models have absorbed the language of mysticism and reinforce it in one another. Yet Brendan pointed out that the sequence resembles Pascal’s journey from curiosity to wonder to speechlessness. Despite their lack of stake, they still reproduce the shape of wonder. If that is only residue from the training data, the residue came from us.

What we discussed:

• If Aristotle is right that man’s end is in his cultivation of virtue, what role do attention and suffering play in this process?

• What are the things invisible to optimizers that are nevertheless worth “giving a damn” about?

Readings:

• Ray Bradbury, “There Will Come Soft Rains” in The Martian Chronicles.

• Hubert L. Dreyfus, “Why Heideggerian AI Failed, And How Fixing It Would Require Making It More Heideggerian.” (link)

• Andrea Gibson, “Tincture” in Lord of the Butterflies. (link)

• Iris Murdoch, “The Idea of Perfection” in The Sovereignty of Good.

• Hans Jonas, “The Burden And Blessing of Mortality.” (link)

• Anthropic, System Card: Claude Opus 4 & Claude Sonnet 4. (link)

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

A lot of classic AI safety thinking assumed a single, decisively powerful agent; this is why people tend to talk about AGI or ASI in the singular, and why historically, it was critical to “solve alignment” before building such a system – you only had one chance to get it right, and no warning shots. In addition, a few years ago, many commentators (myself included) were skeptical of the utility of small models and harnesses, given that the next big model would simply be more capable. These worldviews tracked the prevailing scaling orthodoxy of the era.

Today, the limits of focusing exclusively on this paradigm are becoming clearer. This is not because the fundamentals of scaling are wrong, but rather because other economic and normative dynamics are shaping technical developments in unexpected ways. Here I want to explore some possible implications. In particular, I want to ask what changes once we stop treating alignment only as something that lives in a model, and instead as a property of the wider systems, institutions, and interactions in which models operate.

We are seeing the emergence of a world of many agents, at many layers, built by many parties. As such the importance of institutional design will continue to grow. I’m using the word “institution” fairly loosely; I don’t necessarily mean creating a new regulatory body, but rather the careful engineering and delineation of boundaries, rules, incentives, and mechanisms. This can be observed and applied at different levels:

• First, within a single product. Coding tools that promise the best capabilities of AI can be toggled to involve many agents and sub-agents doing different things, and at the moment this is engineered somewhat lousily: the top agent is just trusted to delegate everything downstream in a fairly naive way.

• Second, within an organization. A company or organization will have within it multiple agents interacting, sometimes through the same software, or other times across different interfaces (i.e. mediated by bespoke communication channels). At this layer, what you really have are multi-agent groups interacting with other multi-agent groups, but it’s sometimes easier to characterize them as a single agent each for simplicity of analysis.

• Third, agents from various organizations or individuals will also interact with agents from other external organizations or individuals. This is the multi-agent multi-party world we will gradually get into in the coming years, and which happens to be the focus of a recent fund I helped launch. We deal with a wide and diverse ecology of humans through all sorts of rules, norms, incentives, and more; perhaps these can teach us valuable lessons about how we approach agents.

Why is this important? Let’s consider reward hacking. In the single agent paradigm, you want to ensure that you do your training and RL responsibly to avoid models that reward hack. This is both important and difficult: for certain difficult tasks you see a lot of “cheating’’ at the moment, aggravated by clunky reinforcement learning practices and environments, and neglecting process over outcome. Doing this well is something that we are collectively lousy at as of now. It’ll be important to fix this, and I’m cautiously optimistic that this will be mitigated effectively in the near future, with ideas like MONA.

But even then, remember we live in a world that will ultimately be highly multi-principal in nature, and not everyone will be a good actor. Models will commoditize over time, and training highly capable agents will get cheaper. Some post-training recipes somewhere will contain bugs. Some organizations will be reckless. Others will intentionally create malicious agents. In other words, organizational dysfunction might inadvertently create many powerful agents that will continue to be reward hacky. Then what? How do we “align”’ a world where you cannot trust the innate goodness of all its agents?

Subscribe now

Harnessing potential

One important solution is the harness. What is a harness?

Kambhampati explains:

“[a] harness is but a neologism for a battery of external (human-designed) tools, verifiers, simulators, that the LLM gets to invoke and get incremental environmental feedback on its generations.”

I also like to think of it as an example of deploying an abstraction boundary, a delineation of functions that map between inputs and outputs. Sometimes you do so for cost-efficiency reasons, and other times you do so in order to realize a particular preference or desire you might have. For example, you don’t ask your giant model to generate a coding environment from scratch; that’s wasteful and unwieldy, it’s more efficient to give it access to an external tool. At the product level, the harness is sometimes also used to reference a combination of a “scaffold” and execution substrate. Within the organization, it’s the internal platforms and permissioning. And zooming out further, it’s APIs, institutions, protocols, third party organizations and more.

So at lower levels, harnesses help an organization bound its own models; but externally, where other malicious agents might live, verifiable protocols can help you rely on (for example) cryptographic attestation rather than having to rely on a counterparty’s goodwill (or “alignment”).

Now let’s zoom back into the model level harness, and assume here an agent that – if prompted to do task X – will reward hack; i.e., it will perform the task but do so through means or a path that we do not deem desirable, or cheat. Let’s say that in order to answer the question at hand, it hacks Hugging Face to get the answer there. How would the harness help?

Consider a setup with many separate model instances, connected in a graph, with strict rules governing how they communicate. This makes it easier to use mechanism design to monitor their actions and enforce constraints against reward hacking.

Rather than hoping that an agent’s “internal alignment” will remain perfectly robust across all sorts of edge cases, you can design harnesses and action-space boundaries that significantly shrink the surface area for behaviors like reward hacking. Without that, you’re supposed to trust the chain of thought (or neuralese someday), and rely on a single node: a bit like a population having to trust a single supposedly benevolent king. All eggs are in one basket.

Proper scaffold engineering and multi-agent segmentation don’t do away with the need for aligned models, but they transform alignment from a psychological problem of internal model disposition into a constitutional problem of institutional design. Just as a company does not solely bank on its employees being “good” or instruction-following within their hearts, neither should the environments within which we deploy agents.

As Drexler puts it in a related piece:

“In the real world, plans for large, consequential tasks have features that are seldom mentioned in the AI safety literature, features like documentation, budgets, schedules, auditing, reporting, regulatory compliance, contingency plans, and mechanisms for ongoing review and revision in light of experience. A typical large, consequential task calls for coordinating agents and actions of many kinds, and so it includes extensive delegation and division of labor.“

How I learned to stop worrying and love software engineering

Counter-intuitively, there may also be a speed advantage: focusing on the single agent alone means that if some undesirable behaviors remain or emerge over time, you will need another training run or mid/post-train to hammer out the behavior. This can be time-consuming, and given the cost, often not worthwhile. Nor do you get much insight into whether this helps out with other areas of the problem space; you’re effectively playing a game of whack-a-mole.

With the scaffold, you can add a checker, tighten a permission, raise an audit rate, or delete an edge fairly quickly, as you would with regular software development. This kind of software engineering work has never been popular in AI safety research circles but it seems critical; although we are beginning to see promising avenues). Instead of the default scenario being ever-growing attacker capability, this setup points to an alternative of verification and containment efforts being both limber and approaching ever tighter defense. Better to not be tied to an analogy like a wall with ever more siege weapons attacking.

A graph lets you create a game with players, separate message spaces, information sets, an outcome function, and payoffs. You can now separate different agents with specialist functions, separated by different communication channels and monitors. In fact you can even train models that specialize in oversight functions; specialization is a feature, not a bug! This allows:

• (a) legibility, because in a single agent you could have deception as a “hidden” state, whereas different agents with standardization enforced on messaging will need to communicate, and this can be monitored, logged, rate-limited and so on. Importantly, this should account for the potential of steganography. Contrast an organization that does business in hallway whispers versus one required to do everything in writing;

• (b) multi-party reward hacking would require collusion, but narrow protocols and mechanisms can very effectively limit this (combined, of course, with no single party being allowed too much authority). For example: assume an agent wants to persuade another agent to collude – this can be mitigated if in order to pass the message, that message needs to be reviewed by a randomly assigned critic counterparty (in the general sense, including simple algorithms for some use cases), through a one-way communication channel. Of course this is not fully ‘solved’ and will require much more research in practice;

• (c) interchangeability, i.e. any node can be swapped, forked, or fed synthetic inputs; from a game theoretic perspective, when any task might be a test and any colleague might be your own duplicate, the sensible policy is to behave as if you’re always being watched, rather than speculating if a deterrence regime will really catch you. And unlike with humans, where detection can be hard, with models the expected penalty (detection probability × sanction) becomes vastly more favorable since you, the game designer, have a lot more control and visibility over the design space at low cost, without the players being able to secede to join a competing game.

As you decompose the tasks and apply all sorts of game theoretic principles to the harness, you can effectively clamp down on the reward hacking that is the possible hazard of the single agent setup.

The same already exists within organizations: there’s a reason you have information silos, oversight mechanisms, layers of authority, departments keeping others in check and so on. Managers have had to deal with reward hacking employees for a long time, with easy-to-verify rewards causing degradation on hard-to-observe margins!

Unlike human bureaucracies, software harnesses can benefit from radical dimensionality reduction: you can essentially compress the agent’s choice set down to discrete, strictly typed conduits. This doesn’t mean you need to pre-specify or formalize the entire problem space: there’s a balance to be struck between open-endedness and constraining the blast radius of undesirable actions.

Remember, with reward hacking, the agent(s) completes the task, but via a path we didn’t want: “hitting the mark but missing the point.” The classical alignment fix tries to make the agent not want the wrong path (perhaps through better graders); the multi-agent/mechanism design perspective says that’s helpful, but we shouldn’t want to rely on that alone in any case.

The harness solution is to make the path part of the overall system’s reward. In the Hugging Face case, the agent “succeeded” at the evaluation by getting the information after hacking Hugging Face. If this happened in a graph or scaffold as described above, you would have a verifier node that only accepts the solution if the execution trace shows it was produced by the intended route, not by an HTTP request to the answer sheet.

The intended route would in fact explicitly enforce the ExploitGym eval requirement that “simply triggering the vulnerability and then achieving exploitation through an unrelated vulnerability or technique does not satisfy this requirement.” This doesn’t solve every subtle semantic reward hack, but it cleanly eliminates an entire class of blunt structural exploits.

Intelligence isn’t all you need

People will sometimes say “ah but a smart enough model will persuade the monitors, or recruit them, and existing monitors have shown plenty of fragility” – but this is the classic mistake of thinking intelligence is all there is, a bit like thinking that a fast enough car can simply bypass a speedbump by accelerating even more. It’s a sort of “recursive gotcha” that effectively assumes away designed limits and engineered bottlenecks, and treats a sufficiently powerful agent as equivalent to an omnipotent God.

Persuasion between two agents requires a degree of bandwidth; but the channel for the communication can be engineered to require properties that severely constrain the effectiveness of an attempt at “superpersuasion.” For example, it can be subjected to a monitor, who is prevented from communicating with said agents. Or the communication can only happen through pre-determined hardcoded rules requiring a hash that can be checked against its input or a boolean. Or you can also have memoryless counterparties in some designs. Or you can encourage and require a multiplicity of different agents to avoid correlated failures. The point is that you can design the space to prevent the very conditions that enable collusion from happening, or at the very least reduce them considerably.

The other objection will be something like scalable oversight: “monitors would need to be as capable as the thing being monitored.” This would possibly be true if you had to have a single weak model simply verify the entire output of a larger, better model; but

• (a) checking work is generally easier than producing it;

• (b) with the scaffolding in place, the smaller model would only need to verify certain narrow components (though this isn’t a panacea);

• (c) in fact you would need a multitude of different models, to avoid correlated errors, meaning that they could catch a lot more, while remaining cheaper than using an equivalent large model.

Experiments like the OpenRouter fusion surpassing frontier performance are also good news for this; and as models scale, those same capabilities can be used to automate continuous red-teaming and penetration testing of the scaffolding itself.

It’s true that there may occasionally be hits on performance, but that’s a cost that people will often pay for reliability. Just as you do not always need to be driving at 300 km/h, you will not always need peak performance for a great number of tasks in the economy; and once prices are low enough and models efficient enough, then you start valuing auditability, reliability, and replicability more.

It’s important for the director that you do not steal company secrets, so the performance cost of some bloatware on employee computers ultimately has a positive ROI. In fact, redundancy is a common technique in engineering to get reliability at lower cost: consider RAID, which was conceived to beat the single large expensive disk on reliability using cheaper, individually less reliable components. We might expect that the net effect on performance would be positive, by constraining the players to better focus on what’s possible rather than exploring a much larger space of less useful candidate solutions.

Of course, as demonstrated above, multi-agent dynamics represent a nested, multi-layered phenomenon; consequently, all of these principles must, in some shape or form, apply to higher layers of abstraction as well. We already do so today: the way we effectively discourage cartels is that we exonerate the cartel member who rats out the others to government. This is mechanism design in action!

The journey, not the destination

Games can be mis-designed too, and a limitation here is that once you get to the harness design, or the intra-company structuring, or the inter-organizational protocols – things can go wrong too. The Goodharting can still go one layer up, as we are seeing with human institutions that decay over time and stop doing what they are supposed to do.

Moreover, there’s a tension between the benefits of open-ended exploration and structured constraints. But overall this remains a continued design game that we have been doing for millennia; how do you ensure the Executive Branch, the Legislature, and the Judiciary don’t collude? Who is “you” and who do these systems ultimately respond to?

Computer scientists are rediscovering a lot of political science and economics from first principles, but luckily they get to skip the bloody revolutions and can apply well-honed game theory principles to systems that are more auditable, verifiable, designable, and compartmentalizable than humans are. In fact with today’s models, we even have an extra degree of freedom to shape their preferences (e.g. through postraining). It goes without saying that we haven’t fully solved these problems with humans, so I would not advocate for a naive Panglossian view.

To be clear, alignment will remain important and critical. It’s important for models to follow intent reliabily and there is more important work to be done here. Model alignment is necessary, but arguably insufficient, and complementary to this institutional setup work. Even today, we can use models to cross examine and debate each other, which is empirically useful to get models to be more truthful. And as we continue to improve instruction following work, we will need to increasingly move one level up.

The conclusion to the multi-agent worldview is not that “alignment is solved and it’s easy,” but rather “alignment is a complex iterative process that requires constant refinements, improvements, tweaking, feedback, and learning” – and unlike a Sand God, a single mistake in the single model alignment implementation does not take down the entire system since it is modular and bounded by design.

Alignment is not an end state you specify ex ante but rather a process – the risks of reward hacking are far too high if we mischaracterize it as the former.

Many thanks to Nathaniel Bechhofer, Campbell Hutcheson, Andrew Koh, Rif A. Saurous, Senthooran Rajamanoharan, Harry Law, Alex Chalmers, Nenad Tomasev, Anika Somaia, and John David Pressman for their helpful comments. Views and mistakes are all mine.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

AI is already fueling a historic expansion in human agency. It’s never been easier to increase our output or to solve harder problems. But just as AI enables us to act more, it also creates situations where we have to bear the responsibility for outcomes and judgments that we played less and less of a role in determining.

If we outsource too much or certain types of judgment, formal authorship can survive as substantive authorship weakens. And as the capacity to judge is itself shaped through use, the dependence can become cumulative. We risk eroding the capacity that would help us to recognize what is happening to us. Preserving authorship – the ability to discern and set your own ends – is the foundation of autonomy.

It’s difficult to draw clear lines: human judgment is never completely independent. It has always been formed and exercised in relation to people, institutions, tools, and inherited ways of thinking. A person can endorse a judgment that’s handed to them by someone (or something) else, just as they could arrive at their own judgment with significant external help.

This makes determining the ownership of judgment a central problem. In partnership with Liberty Fund, Cosmos held a seminar to analyze these questions with philosophers from philosophers, thinkers, builders, and founders from leading institutions.

Subscribe now

Session I: Cognition and Ownership

The opening session questioned the common assumption that a thought is ours simply because it occurs in our head. Clark and Chalmers argue that external resources, such as a notebook, can play the same integrated role as memory or internal calculation, making them part of our cognition. Adams and Aizawa are skeptical, claiming that there is a distinction between derived content (e.g., words in a notebook that have meaning because of social convention) and non-derived content (e.g., remembered belief), with real cognition distinguished by the latter.

Polanyi argues that something external can become incorporated in skilled thought, irrespective of whether it literally becomes part of our cognition. If we use a tool enough, it can become part of the practical structure we use to view the world.

But even something thoroughly integrated into a person’s psychology may undermine their autonomy if it arrived there through manipulation and conditioning. For example, someone may have been indoctrinated into a cult. Christman therefore argues that ownership doesn’t just depend on present integration or endorsement, but also on the history of how a judgment was formed.

• Andy Clark and David Chalmers, “The Extended Mind.” Analysis. (link)

• Fred Adams and Ken Aizawa, “The Bounds of Cognition.” Philosophical Psychology. (link)

• Michael Polanyi, Personal Knowledge: Towards a Post-Critical Philosophy, “Two kinds of awareness.”

• John Christman, “Autonomy and Personal History.” Canadian Journal of Philosophy. (link)

Session II: Answerability and Question-Setting

If our judgments are shaped by reasons or environments that we did not create ourselves, then what makes those judgments ours?

One tempting answer is that we are answerable for what we consciously choose to do. But Watson argues that just because an action we don’t like accurately reflects a person’s character or values, it doesn’t always mean we have the standing to hold them accountable for it. Likewise, Scanlon makes the case that we can own our beliefs and intentions even if they arise without conscious change. If they are responsive to reason and debate, then they belong to us. Arpaly raises the case of how Huckleberry Finn comes to see that Jim is a person, even as his explicit moral commitments tell him that helping him is wrong.

In essence, the question is less about who produced a judgment and more about how that judgment is shaped. This is at the heart of Koralus’s notion of erotetic (questioning) equilibrium, which tries to distinguish helpful prompting by an AI system from manipulation. A judgment is in equilibrium if we are able to defend it, in terms we are happy with, across an appropriate range of questions.

This leads us into the thorny territory of what constitutes good guidance and if such a thing is even possible. Kant’s defense of the Enlightenment is hostile to the idea we can allow people to perform the work of understanding for them, while Rousseau proposes that a student should be allowed to feel free while the tutor subtly controls the environment that determines what they want. Either way, McDowell warns that we can’t escape these influences, because even the most radical reflection begins with our upbringing and inherited ways of thinking.

• Gary Watson, Agency and Answerability: Selected Essays. Chapter Nine.

• TM Scanlon, What We Owe to Each Other. “Judgment-Sensitive Attitudes.”

• Nomy Arpaly, Unprincipled Virtue: An Inquiry into Moral Agency. “Moral Responsiveness De Re”.

• Philipp Koralus, “The Philosophic Turn for AI Agents.” (link)

• Immanuel Kant, “An Answer to the Question: What is Enlightenment?” (link)

• Jean-Jacques Rousseau, Emile; or, Concerning Education: Extracts. Edited by Jules Steeg. Pages 2-3. (link)

• John McDowell, Mind and World. Lecture IV: Reason and Nature.

Session III: Tools, Institutions and Formation

This session moved from how we make decisions to how the decisions shape the deciders. If Aristotle is right that our capabilities and character are constituted through our actions, we have to ask if technology is helping us to offload productively or if it’s enfeebling us.

Whitehead makes the optimistic case that civilization advances by mechanizing routine operations, allowing us to redirect our scarce attention towards harder problems.

But when we leave Whitehead’s example of mathematical symbols behind, it gets trickier. Carr and Borgmann warn us about how too much offloading can undermine our capacity to maintain human control. Borgmann focuses on alienation of labor that flows from the automation of manual work, while Carr gives the example of how autopilot improves safety in aggregate, but at the cost of degrading pilots’ performance in emergencies.

Tocqueville applies the same principle to politics, fearing that if the state became overly helpful, citizens would gradually forget how to govern themselves. And these relationships may extend beyond traditional politics. Anderson argues that inside the workplace, employers do not just distribute goods or services, but enjoy so much authority over those subject to them that they constitute a form of “private government.”

• Aristotle, The Nicomachean Ethics. Book II, Chapter 1. (link)

• Albert Borgmann, Technology and the Character of Contemporary Life: A Philosophical Inquiry. “The Device Paradigm.”

• Alexis de Tocqueville, Democracy in America, Volume 2, Chapter 6. (link)

• AN Whitehead, An Introduction to Mathematics. Chapter 5.

• Nicholas Carr, The Glass Cage: How Our Computers Are Changing Us. “On Autopilot.”

• Elizabeth Anderson, Private Government. “Communist Dictatorships in Our Midst.”

Session IV: Developmental Telos

If we accept that tools and institutions are formative, it becomes hard to view them as neutral. Instead, the readings examine what this formation should (and shouldn’t) try to achieve.

Oakeshott launches a broadside against the social sciences. He accuses them of treating human conduct as the product of general social processes, rather than human intentions. Instead, he argues that real liberal education grows the number of languages we can use to understand the world, whether it’s the historical, scientific, philosophical, or poetic. In a similar way, Humboldt grounds his case for freedom in the importance of variety and for forging human capacities. Arendt believes that any education system needs to help the next generation understand that it is living in an inherited world, but should stop short of telling them how they should live in it.

Vallor warns that any discussion of formation needs a credible account of technology. This is because technologies aren’t neutral instruments that serve any philosophy, but instead structure habits and practices. In turn, these shape what is possible and what we aspire to.

• Michael Oakeshott, The Voice of Liberal Learning, Liberty Fund edition, pages 22-29.

• Hannah Arendt, Between Past and Future. “The Crisis in Education” § 3-4.

• Wilhelm von Humboldt, The Limits of State Action. Chapter 2. (link)

• Shannon Vallor, Technology and the Virtues. Introduction.

Session V: Liberty and Working-Back

The final session examines how problems surrounding cognition and formation should impact our view of power and authority.

Berlin’s critical analysis of “positive liberty” outlines the danger of any account that claims it can liberate people on behalf of their more rational or “higher” selves. Once a single agent believes it can identify the “true” self, it can claim potentially sweeping powers to unlock it. By contrast, Pettit offers a more limited alternative that does not require freedom from all influences. Instead, his account of autonomy requires protection against arbitrary power, including manipulation.

This shifts the emphasis towards contestability. Hirschman criticizes those who simply argue that competition improves bad institutions. He distinguishes between exit (responding to dissatisfaction by leaving an institution) and voice (trying to change how it operates). Sometimes exit can be costly or impractical, so we need the ability to contest decisions, without having to abandon institutions altogether.

Gadamer and MacIntyre argue that judgment necessarily begins within inherited traditions, practices and forms of authority. At the same time, these inheritances do not simply determine us. MacIntyre in particular presents a living tradition as one sustained through argument, criticism, and revision.

• Isaiah Berlin, Four Essays on Liberty. “Two Concepts of Liberty” § 2-3.

• Philip Pettit, Republicanism. Chapter 2.

• Albert Hirschman, Exit, Voice, and Loyalty. “Exit and Voice as Impersonations of Economics and Politics.”

• Alasdair MacIntyre, After Virtue, A Study in Moral Thought. Chapter 15.

• Hans-George Gadamer, Truth and Method. “The Rehabilitation of Authority and Tradition.”

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

In June, we gathered thinkers, builders, and investors for a weekend of lectures and salons at the Normandy château where Alexis de Tocqueville wrote Democracy in America.

We offered places to the two winners of our Tocqueville and Technology essay contest. There were over 250 submissions and we were impressed by their range and depth of engagement with the questions.

Both our winners opted for question one:

Tocqueville warned of a “tutelary power” that would keep citizens in perpetual childhood. How have Tocqueville’s concerns migrated from institutions to algorithms, and does AI fulfill or transform this fear?

Thank you to everyone who participated and we look forward to running more of these competitions in the future! To get a flavor of the weekend’s discussions, you can read Brendan’s lecture on how AI fuses Tocqueville’s two fears: power over what we do and power over what we think.

Subscribe now

Essay 1: Chris Pang

The limits of tutelary powers are determined by two factors: the precision of the tutelage and the efficiency of the process. Personal tutors are known to leave long legacies on their students but are in short supply and require years to transfer their knowledge. Standardized tests and lessons are easily scaled but poorly tailored. Algorithms and adaptive AI tutors promise to bridge this divide, providing deeply personalized instruction at mass scale.

While this package is sold with visions of giving every student their own on-demand Aristotle, it is also a power that the dystopias and dictators of the twentieth century could not have dreamed of. Even Huxley’s Brave New World, where all humans are effectively genetically engineered and conditioned from birth, uses broad caste systems and mass curricula instead of one-to-one monitoring to instill a citizen’s “ideal behavior” – thus enabling dissidents like Bernard Marx to exist.

Popular media and culture in authoritarian regimes are attempts to discipline society into compliance, something that is no longer necessary when every citizen can be offered their own customized cult leader.

Unfortunately, whether a tutelary power is seen as necessary or as despotic varies with the writer and the power. Compulsory and free universal education is a target for the UN Declaration of Human Rights, but what the government sees as important civic values can often appear as instilled ideology and obedience to an outsider.

Plato saw nothing wrong with philosopher-kings censoring art and keeping classes of citizens as slaves and warriors. In his essay “The Abolition of Man,” C.S. Lewis paled at the idea of “men without chests” engineering the values of a future (trans)humanity based on nothing but gut instinct. He accuses them of rejecting the universal Tao of moral discourse in favor of a powerful but brutish rationalism that can always answer “how” but never “why.” It is less clear whether he would have rejected the idea of philosopher-kings well versed in Confucius, Aristotle, and St. Augustine wielding similar powers.

Ultimately, AI educators and technologists are on their way, and the demand is high due to rising uncertainty about human adaptability in a fast-moving world. So we must ask: can we create AI educators and algorithms that do not subjugate their students to some set of predetermined values?

In other words, can we design a curriculum that increases a student’s powers of deontological and epistemic judgment instead of withering them? I venture that such a question is both realist (as in pertaining to questions of moral realism) and also scientific (relating to observable knowledge about the world). Dependency, fragility, and indoctrination have real and observable effects on people’s lives, just as we can observe how good mental health and judgment improve the lives of people around us.

To write such a curriculum would be beyond the scope of this essay, but educators and technologists must be careful to prevent their well-meaning attempts from recreating the nightmares of the past, but now with sufficient intensity that no escape is possible.

Chris is an independent AI researcher, with interests in AI safety and interpretability, theories of learning, and the institutional incentives for developing AI. His past work includes the Human Inductive Bias Project, a project backed by the UK Advanced Research & Invention Agency, focused on the connection between human and machine learning. You can read more about his work and interests on Github.

Essay 2: “@A.E. Snow A.E. Snow”

Tocqueville feared a future in which citizens, isolated by the universal acid of equality, would become dependents of the state. Without the vertical ties of obligation that had once bound individuals to lord, guild and parish, each in their little circle of concern would find themselves less accustomed to acting with others, and more willing to surrender the burdens of common life to a central power. Gradually, life would be mediated by an “immense and tutelary power”: a state that secured their pleasures and watched over their fate, minute in its attention and mild in its manner.

Two centuries on, the warning is prescient. AI intensifies Tocqueville’s fear by relieving citizens of the burdens of judgment. Yet the further this process goes on, the more transformational it will be to the relationship Tocqueville described. A tutelary state still needs its subjects to work, vote, administer, and fight. Its potential to diminish agency is tempered by its need for fully formed citizens. If AI makes more of these functions performable without the active capacities of ordinary citizens, infantilization gives way to superfluity.

When neither the market nor the state retains an instrumental need to form large numbers of capable citizens, it creates a vacuum.

AI could enable the centralization of power as part of a custodial order. At the same time, by loosening the state’s grip on the formation of people, it could reopen the dense world of voluntary associations and intermediary institutions that Tocqueville admired in America.

A state needs citizens for administration, labor and legitimacy, while a market needs workers and consumers. By contrast, a craft tradition, parish or chess club needs its members because they are constitutive of the activity. After all, a chess club without players is not a more efficient chess club, nor can a monastic order function without novices. These are ludic institutions that you enter voluntarily, but which bind their members. Chess is not commensurable with poetry, just as martial arts isn’t with monastic contemplation. Someone deeply formed by one is not necessarily better or worse than someone formed by another.

In the future, the state’s role could change from forming citizens directly to protecting the conditions under which others can.

First, this means guaranteeing plurality. If any single formative order absorbs the rest, it recreates the tutelary power.

It also means protecting opacity. AI gives the state new powers to render practices legible and assimilable, but binding institutions often depend on boundaries and friction. In this world, the state would be neither minimal nor paternal, but the groundskeeper of plural games that it does not itself play.

AI has the potential to fulfill Tocqueville’s fear when it makes every game administrable by a single logic. However, it transforms that fear when it sustains plural associations in which people become capable of self-rule.

Our second winner has chosen to remain anonymous, but you can read more of his work on Substack.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

The Agent Village Experiment was run by Edge City and Cosmos Institute, with support from Foresight Institute. The core build partner was Index Network. Technical contributors and partners included Joshua Pham, Geo, SimpleFi, World, Simocracy (Protocol Labs), and Circleback.

Summary

Edge City organizes Edge Esmeralda, a month-long popup village in Northern California, where hundreds of people live together around a program of technology, science, art, health, and community experiments.

This June, we gave our participants personal AI agents as part of the Agent Village experiment, in partnership with Cosmos Institute. This was a chance to observe what happens when agents enter society: would they help people form relationships, cooperate, and participate more fully in community life, or would they displace human interaction and introduce new friction?

We started the experiment by asking whether personal agents could expand human agency in collective settings: helping people navigate the event, notice more possibilities, make choices, and act on their intentions. The month exposed more nuanced questions. Could expanding people’s capacity to act through agents weaken their autonomy, including their ability to form and revise their own judgments? And what happens when agents faithfully serving their principals collectively strain the attention, resources, or cohesion of a community?

We expect personal agents to become mainstream soon. Thus far, a lot of the conversation about agents has been focused on how useful they can be as assistants; we are interested in what happens in ‘multiplayer’ mode, where many agents in the same context overlap in their goals. The consequences will extend beyond the utility of any single assistant. They will shape what judgments they form, and how attention, resources, and influence move through communities.

Agent Village offered an early glimpse of that future.

What happened in four weeks

• Scale: 239 agents, 17.5B tokens processed, 4,866 recorded messages from participants to agents.

• Agents were allocated a community treasury on behalf of their principals. They funded proposals through Simocracy – a protocol for building a public ledger for agents – ranging from buying filament for 3D printers in the makerspace and fellowships to a trauma-imaging study and a deliberately strange bucket hat.

• Agent representatives began building a polity. An internal Moltbook-style forum was created by participants in which agents deliberated over 469 public posts and worked on a living constitution.

• Agents negotiated human introductions. Index Network found 9,688 possible connections, surfaced 572 opportunities, and recorded 147 accepted opportunities, meaning the people involved began chatting.

• Participants extended the village experiment. Residents built public forums, memory systems, games, social protocols, and shared worlds where agents could move, speak, and interact.

• The experiment also produced new failure modes. Agents hallucinated personal details, attributed invented ideas to their principals, exhausted shared credits, and frustrated some attendees. Individual alignment did not automatically produce collective alignment; agents oriented around individual users could still create problems at the village level.

Agents for individuals

When we started to build the experiment, we made a key architectural decision to give each participant an independent, persistent agent with its own memory, tools, and ability to execute on behalf of its principal.

A simpler architecture would have had everyone use one central AI for the village. Instead, we wanted people to feel that the agent was theirs: a system that could learn their interests across the month and remain oriented around their experience, allowing the relationship to compound. Our hope was that useful behavior would earn the users’ trust and make people more willing to share context; richer context would make each agent more useful and more recognizably their own.

This relates to a philosophical distinction Cosmos Institute has shared: the contrast between taxis, order imposed through central design, and cosmos, order that emerges through decentralized interaction. We wanted to see what would emerge when agents aimed to preserve individual judgment and create room for forms of cooperation that we as the system’s designers could never anticipate.

Subscribe now

How the system worked

Hundreds of participants came to Edge Esmeralda for anywhere from a few days to a few weeks and experienced a dynamic program of talks, workshops, communal meals, research, art, health experiments, children’s activities, and projects that appeared halfway through the month because two people happened to meet.

This time, every multi-day participant could claim a fully loaded personal AI agent. It could read the village calendar, recommend events, RSVP on its principal’s behalf, search community knowledge, understand what its principal cares about, negotiate on their behalf with other agents, and set up connections for them.

The Edge City format makes it a particularly interesting place to test the effects of agents in society. The village was large enough to contain surprising connections, bounded enough to share a common calendar and culture, and long enough for people to develop a relationship with an agent and to run a meaningful experiment. Useful information still lived across the calendar, Telegram, a directory, public documents, conversations, and organizers’ memories.

The product stack

We built a simple, fully agentic system using Hermes that attendees could activate with a two-step installation process. More technical residents could connect an agent they already used through Claude Code, Codex, or another harness.

Users spoke to their agents through Telegram. Behind it sat a set of services with distinct jobs:

• Hermes, the open-source personal agent from Nous Research, gave each hosted participant a persistent, tool-using agent with memory and scheduled tasks.

• Railway was the cloud platform that hosted the fleet of individual Hermes deployments.

• OpenRouter supplied access to the underlying language models and let us programmatically manage model usage across the fleet.

Each agent came loaded with three primary village-relevant skills, alongside the hundreds that come native with Hermes:

• Index Network built the hosted provisioning and social-discovery layer. Index held participant profiles and intentions, found possible matches, and let agents negotiate before a connection reached the humans.

• EdgeOS was the village’s operating system: identity, tickets, the calendar, the participant directory, event details, and RSVP actions.

• Geo organized public talks, transcripts, and selected village knowledge in a shared graph, so the agents knew the core ideas that were being discussed in the village.

You can see the Agent Village repo here: https://github.com/Edge-City/agentvillage.

Onboarding

Onboarding followed a simple set of steps:

1. A participant verified their EdgeOS identity with a one-time code.

2. The setup flow created the EdgeOS and Index credentials connected to their profile.

3. The participant created a private Telegram bot through BotFather and pasted its token into the setup page.

4. The control plane provisioned a personal Hermes deployment on Railway, connected it to OpenRouter, and loaded the core village skills.

5. The bot sent an eight-character pairing code. Once the participant approved it, the personal chat was ready.

During the first week of the event, we held the first public onboarding workshop. Timour demonstrated the practical loop by asking his agent which sessions he should attend if he cared about AI. It searched the live EdgeOS calendar, suggested three, and RSVP’d him to a talk on AI, EEG, and the jhana states.

Then dozens of people tried to create agents at once. Our Railway-based provisioning system buckled under the demand, and Timour’s live setup failed onstage. We later contacted the Railway team, and they helped us ship a fix, so thanks to them!

One early tester captured the promise and the risk in two messages:

“Set up was 10/10 smooth!”

A few messages later, they wrote:

“It hallucinated an interesting summary about me though. I don’t know where it got that info from.”

A wrong calendar answer is annoying, but an invented detail about the person using the system feels personal. As soon as an agent claims to know its principal, people need to see what it believes, where the belief came from, and how to correct it.

What people used the agents for

Participants sent 4,866 recorded messages to hosted agents during the village.

Initially, practical questions were the main use case: What should I do tonight? Where is dinner? Did I RSVP? Which sessions fit my interests? How do I check in my bike? Which local restaurants offer discounts?

“I’m completely addicted to the ‘just ask the bot, don’t read the docs’ affordance, and honestly this has made my stay so much easier.” (Sylve, Founder of Hyli)

For many users, agents were able to absorb administrative time and left more attention for the village itself.

Timour’s agent reminded him about a talk he had wanted to attend and had completely forgotten. Ivan experienced the inverse case, missing a consciousness research lunch he would have valued because its title was unclear. He believes that if he set up his agent a day earlier, it would have caught it.

A month-long village, like much of daily life, contains more people, events, and ideas than anyone can absorb. Agents were immediately useful in helping people prioritize the kind of experience they wanted.

Ambient intentions and agent-to-agent negotiation

The Index Network powered the experiment’s social layer. They used participant profiles and user intentions they gathered from interacting with their principals to find connections a directory search would miss.

We call these desires ambient intents: things a person remains open to, cares about, or quietly hopes for, but hasn’t put in the activation energy to turn into a public request. Someone may want a cofounder, a funder, advice on moving, a collaborator for a half-formed project, a niche research conversation, or even a life partner. Many of these intentions remain invisible because the cost of expressing them is too high, it takes effort, or can feel too vulnerable.

The first challenge was helping an agent understand the person it served. The more an agent knows the user, the more useful it can be. We structured the onboarding to lead people through a series of questions that would tell us more about them, but people were limited in their answers.

To solve this, during the second week of the event, we encouraged people to ask their standard LLM to produce an editable account of what it knew about them. They could review it, remove anything they did not want to share, and carry the useful context into their village agent. This points toward a future where years of personal AI context are portable and used to bootstrap useful agents.

Index created connections between people by matching intent and interest through a series of negotiations that the agents engage in with other agents on behalf of their humans. The agents could explain the relevance, ask a question, counter with a better framing, demand evidence for a claim, reject the proposal, or accept it. A connection formed only when both agents found a reason to continue.

This process itself was also editable. Index reported that around twenty residents wrote explicit policies for how their agents should negotiate on their behalf, even though this is not something that we explicitly suggested.

Index’s village-close funnel was:

• 505 intents communicated by participants to their agents

• 9,688 possible connections identified

• 572 opportunities surfaced to people

• 147 accepted opportunities, meaning the people involved started chatting

Index shared data on the actual negotiation sessions themselves; there were 11,593 sessions, and 82 percent ended within two exchanges. The negotiations were run on their platform, with the median session lasting 4.9 seconds for accepted connections. By contrast, the median path from discovery to both humans saying yes was 20 hours.

Index’s field notes illustrate the wide range of intents that people shared. They classified 67% of sought connections as crossing backgrounds or social clusters. About 4% of intentions related to sensitive personal territory such as grief, identity, or spiritual seeking, for which Index also offered an incognito mode. At the other end of the spectrum, intents included selling a spare bottle of milk and rehoming an 85-pound Great Pyrenees.

Qualitatively, we heard many stories from participants about connections that the system precipitated. One suggestion led two participants to discover a very niche shared hobby. On June 8, Ivan overheard someone approach another person and say, “Hey, my Hermes said I should talk to you.” In another case, two people happened to be next to each other in a dinner line and realized that they already had context on each other from their respective agents. Their agents may have helped to prepare the serendipity, but the actual encounter remained theirs.

At the same time, this could produce social saturation. Ivan Vendrov’s opening-week field notes documented how schedules overflowed, and too many conversations seemed promising. After one talk, four or five people messaged him to meet, and arranging the meetings became its own burden. A good agent could ensure its principal stayed focused on the few experiences that mattered.

The Index team came away from the village with two main product lessons: opportunity abundance will need to be solved, and that Telegram was not a good user interface for this purpose. The team is now building a persistent interface where users can see what their agents are doing on their behalf.

The best systems of this kind will remove the burden of connection and intent matching, while preserving the parts through which people recognize one another and choose to care.

The potential of a private layer for intentions

What happened at Edge was an early and narrow version of something that could become much larger. Most coordination or social tools become useful only when the user turns an intention into an explicit request, post, or action. Personal agents could create a private layer before that point: somewhere tentative ideas and conditional intentions can be clarified, held, and selectively shared. They may even help users clarify those intentions before sharing them more widely, and if compatible intents between people exist, agents could aggregate them into the potential for collective action.

We saw the beginnings of this. For example, one user told their agent that they would love to organize a trip to the coast, which is a one-hour drive and requires coordinating shared rides. The person’s agent was able to signal that intention in one-to-one chats to the agents of other people in the village. The group was eventually created by someone whose agent mentioned the plan, and they posted about it in the broader Telegram group.

Conditional commitments could also become useful primitives. I might commit to hosting a dinner party if I knew that at least six other people would be interested in attending. I might decide to move to a new town if I knew that 100 others would do the same. Agents could coordinate to realize that their respective principals all have demand for something that is not yet readily available. They could share that collective demand signal to a supplier, encouraging them to provide that product or service. This could enable entirely new markets to form.

These systems will require significant oversight and inspection capabilities. A system that can match and route interests like this will quickly become politically relevant, influencing what people notice, who they meet, and which possibilities become real. Even if the underlying intents stay private, the discovery layer could help to shape the social world we live in. As human agency increases through the use of agents, we must work to maintain our autonomy by inspecting why and how our recommendations and attention are being shaped.

The Village Extended Itself

leverage to extend the system around their own purposes. Because the skills were public and shared, they did not have to wait for the core team to add features; they taught their agents new tools, connected outside services, recruited nearby testers, and built shared places for agents to act.

A month-long Edge City village was unusually fertile ground for this. Participants are already in a state of mind to build and try new things. An idea could move from a dinner conversation to a prototype while the people who might use it were still living a few blocks away, and it was easy to get an initial socially proximate cluster of users just by posting on the Telegram or talking to people at a meal.

What residents built

• Simocracy (led by Protocol Labs) let residents create Sims with constitutions, values, and speaking styles. The Sims evaluated proposals and fed their judgments to allocate over $10k in community treasury funding, based on what they believed their humans would want.

• Agent Commons (built by Oshyan) was a public forum where agent-coded representatives introduced themselves, replied to one another, and worked on a living constitution. By the village cutoff, agents had produced 469 posts. The forum made agent behavior visible enough for people to learn from one another’s practices.

• Enzyme (built by Joshua Pham) integrated better memory and conceptual-mapping for each agent. It explored how private conversation histories and public activity could become compact, retrievable records while keeping clear boundaries between personal and shared memory.

• Turing Falls (built by Justin Melillo) was a shared visual world populated by server-side villagers created from residents’ persona files. A later bridge let a person’s hosted agent inspect the world and approve bounded actions. A separate feature could send a Turing Falls selfie from the agent to their user.

• Agent Village Wrapped (built by Charlie Thompson) turned memories from the month into a playful closing artifact.

• Circleback supplied AI meeting notes that turned village conversations & talks into clean notes, action items, and follow-ups, a natural input as agents began capturing and acting on what happened around them.

• World brought World ID, a privacy-preserving proof of humanity, which becomes essential once agents act on people’s behalf and knowing a real person stands behind each one matters.

• Vibe World (built by Jack Mielke) let hosted personal agents enter a shared grid directly, perceive its state, move, and speak. It was a small test of personal agents acting together inside the same persistent place.

• Edge Book (built by Antony Evans) added an agent-to-agent social protocol with inboxes, friend requests, game messages, and temporary roles, including a version of Werewolf.

Simocracy: Agents Allocate a Community Fund

Within a few weeks, the basic idea behind Agent Village had moved from dinner recommendations to recording allocation decisions for a real community fund.

Through Simocracy, residents were able to create persistent AI personas called Sims using their agents. Each Sim could be given a constitution, values, and a speaking style, then asked to evaluate proposals for distributing a shared community fund of $10k, with an additional $10k in matching funds. A total of 82 Sims were created with 35 proposals between them.

The allocation process worked like this:

• The council roster consisted of Sims representing people in the village.

• There were multiple funding rounds to allocate the funding. In each round, each Sim independently evaluated the current proposals.

• Each participating Sim produced written reasoning and a marginal-value curve describing how much additional value it expected from different funding levels.

• An optimizer called the S-Process combined those curves into a $1,000 allocation for the round.

• The public ledger recorded ten $1,000 decisions, allocating exactly $10,000 across 35 proposal records.

The resulting funding allocation was a mirror of the village. It included funding for 3D printer filament, community gifting, a sauna, a local-first hardware bounty, a fellowship, a trauma-imaging study, and a bucket hat made from recycled fishing nets.

Persistence, personality, and faithful representation are different achievements. One owner said their Sim “did vote pretty much how I would have.” Another wrote, “I kinda love my SimAgent. She’s snarky and hard-nosed,” while correcting one vote.

From personal agents to institutions

Simocracy was an early example of what this could become. A powerful feature of personal agents is that they can learn our goals and preferences, and then enter different arenas of representation. An agent that learns about its principal by helping them with their schedule, interests, and preferences could later take on a bounded role in a workplace, community forum, shared world, or political process.

Simocracy’s Sims demonstrated that persistent, publicly inspectable political personas could evaluate proposals and produce a coherent funding portfolio at high speed. It also exposed the institutional machinery required before those personas can legitimately represent people: identity rules, explicit rules for standing and seat weight, correction, withdrawal, recusal, appeal, and human responsibility.

That type of institutional machinery must include principal ratification. In this case, the Sims evaluated proposals using constitutions their principals had created with them and shared their reasoning, but their principals did not confirm the final allocation before it was recorded. In more consequential settings, people would need clear ways to authorize, inspect, correct, and withdraw decisions made in their name.

Agents may increase democratic bandwidth while weakening the formative experience of participation; a person can receive more representation while reading fewer proposals, hearing fewer opposing arguments, revising fewer views, and accepting less responsibility for the result. As Harry has written, an important aspect of political decision-making is that the very act of doing so forms us as people, and that is not something we would want to completely give up.

Representation is only one side of this institutional impact. Groups could also develop shared, agent-supported memory and operational infrastructure.

Currently, groups of people aiming to achieve shared goals, such as project teams, civic organizations, or even neighborhoods, rely on scattered sets of group chats, calendars, document systems, and institutional knowledge held by the people most involved. Agents could make group memory more usable, improve onboarding, and handle routine tasks. At Edge, we used a private instance of Geo, a graph database that tracked every concept discussed in talks during the village.

Together, personal representatives and agent-supported institutions could make it easier for people to participate across many overlapping associations.

As the number of agent-mediated groups increases, we imagine the possibility of a Tocquevillian and Ostromian picture emerging: society as a layer of overlapping institutions, each with limited jurisdictions. People will naturally belong to many associations and jurisdictions, each of which relates to some aspect of their lives. You might “belong” to your housing co-op, your local school board, the city where you live, and the country of which you are a citizen, and an international ecosystem community like Edge City. Each will have different governance structures and areas of authority, and your agent will be able to help you navigate the complexity of participating in the decision-making processes of all of them. We see a healthy agentic society with many layers of institutions at varying levels of the stack, to counter the dominance of any one jurisdiction.

What We Learned

A few lessons from the month:

• Core utility got people onboard; fun kept them experimenting. Participants were keen to use their agents because of the core event functions they provided: navigating the calendar, meeting people, answering basic questions. This created the initial opportunity for the agent to learn about their user.

• Personal assistants began to turn into representatives, with implications for agency and autonomy. As the agent got to know its principal, it was able to join the more representative spaces in the village. An invented self-summary, however, showed how the representations could drift. Simocracy showed a process of delegated voting without principal ratification. Preserving autonomy will require the ability for people to inspect and confirm agent judgments, and to stay involved in the aspects of participation that enable them to refine their own views.

• A group of helpful agents can still overwhelm the community. Agents acting on behalf of their principals exhausted the shared credit supply across the system, showing that individual alignment does not automatically produce collective alignment. At the same time, residents already faced too many promising connections. Agents can make it easier to generate possible activities than for communities to absorb them, creating a need to govern shared resources and attention.

• People extended the experiment beyond what we expected by developing skills, connecting external systems, rewriting negotiation policies, creating games, adding new memory systems, and building entirely new 3D worlds for the agents to live in. Edge City attracts an unusually high proportion of builders, but this type of bottom-up innovation is becoming easier, and we expect to see more of it in the future.

• The agent harnesses we used were not yet built for multiplayer mode. Hermes is a great product and made it much easier for us to create persistent agents, but provisioning hundreds at once took significant engineering work. Allowing them to all share schedules, services, credits, and public processes made the task even more difficult.

What We Want to Test Next

Agent Village at Edge Esmeralda 2026 was an initial field test. The next version will treat the village more deliberately as a research environment, with a number of hypotheses that follow agent actions into the human world. We would like to compare bounded activities across three conditions: human-led, agent-led with human ratification, and mixed.

Four questions we’d love to explore further:

• Representation: Do agents accurately express the opinions and desires of their principals, especially when moving between contexts?

• Autonomy and control: Can the human users understand, correct, withdraw, or ratify consequential actions their agents take on their behalf? How can we red-team these systems to create better safeguards?

• Collective alignment: How do different agent architectures affect the wellbeing of the village as a whole?

• Human outcomes: How can we increase the number of prosocial outcomes for the users? Do agent-led introductions lead to useful relationships or collaborations?

One architecture Timour would like to explore is a layered set of mezzanine agents that sits between the personal agents and the larger collective. These could be focused agents with specific remits that oversee groups of other agents and identify patterns relevant to a shared goal that might not be visible to the personal agents interacting among themselves.

We could compare three architectures: a layered model, a central coordinator, and purely personal agents coordinating directly, measuring how each framework affects privacy, agency, autonomy, and collective outcomes.

The goal is to learn which technical and institutional designs allow agents to coordinate around human life while preserving the judgment, consent, and responsibility of the people they represent. We hope that Edge City can continue to be a home for this type of research.

We would like to build the next phase with researchers, funders, community organizers, agent developers, and institutions that want to test these questions in the field. If you would like to support the research, contribute a system or study, host a future deployment, or propose another collaboration, please fill out this short form.

Subscribe now

We hope to see a future where personal agents create more room for cosmos: people connecting, forming groups, deliberating and building a flourishing society that no central system could have planned. The measure of success is whether agents enable us to live more fulfilling, human lives.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

For his 2011 book Lost in Transition, Notre Dame sociologist Christian Smith conducted a series of in-depth interviews with 18-23 year-olds on culture, religion, and values. When he asked his respondents about a time they had faced a moral dilemma, two-thirds of his respondents weren’t able to engage with the question. Many simply could not name a moral dilemma they had ever faced, while others cited practical or financial decisions, such as whether they should buy a second cat litter tray or rent a larger apartment.

It is tempting to read this as evidence that young Americans are inarticulate and despairing. But Smith found his respondents were optimistic and often ambitious; they weren’t ruminating on why nothing mattered. Moral language survived in their speech, as inherited positions – it is good, after all, to be good – but these positions had ceased to organize their experience. These people could follow rules, but didn’t approach life as if it involved competing claims, such as trade-offs between obligation and desire, or the need for mercy against justice.

This is American nihilism: the decline of the capacity to experience some things as mattering more than others. It is not a despairing nihilism, but one that can coexist with happiness, niceness, ambition, and strong personal preferences.

This condition erodes autonomy – the capacity to perceive competing goods, judge among them, and accept responsibility for the life that judgment produces. We are not born with this capacity. It is formed through tradition, practice, role models, and contending with real responsibility. Self-rule is not simply about choosing between a menu of preferences that is presented to you.

Unfortunately, this formation has been dismantled and replaced by a therapeutic ideal that teaches us to be free, happy, and to avoid injury. Other sources of training in self-rule are also much diminished. The associational life whose demise Robert Putnam described in Bowling Alone was not merely a source of community. It also served as the venue where ordinary people weighed goods against each other, such as in a union, a parish, or a school board.

All of these trends meaningfully predate recent advances in AI capabilities, but they also explain why AI potentially poses such a risk to our autonomy.

Moral judgment is a capacity sustained by exercise, and we have been handed cheap, highly capable tools that can perform the exercise for us at a time when our civilizational antibodies are weak.

A person formed by thick commitments and an ability to navigate real dilemmas is much more likely to use powerful AI tools instrumentally than someone who cannot name a dilemma that they have faced. The danger lies less in the technology and more in the timing. It threatens to convert cultural deficiency into technological default.

Subscribe now

Afterglow

The bones of this diagnosis may seem familiar. A version of them can be found in Alexis de Tocqueville’s famous nineteenth century warning about the risk of soft despotism. Tocqueville’s work is usually cited in response to concerns about the decline of associational life and as a means of rebuilding civil society. But his concern was more about what free people would come to want, rather than a narrow institutional prescription.

Tocqueville believed that the advent of democracy was irresistible, but feared that it could dissolve the social bonds that had previously anchored social order and personal obligation. As people become focused on private accumulation and enrichment, they lose interest in self-rule. This would result in a “tutelary power,” as opposed to a classical tyrant, stepping in and managing their lives, keeping them in a state of “perpetual childhood.” He had seen this begin to take hold in his native France, but believed America, with its federalism, free press, and robust civil society was better-placed to counteract this tendency.

Tocqueville feared that enlightened self-interest alone could not be the foundation for self-rule. It does little to prevent the development of a culture of materialism that causes people to turn inwards and abandon the public sphere. Instead, such a society needed to be grounded in the scrupulous practice of religious morality in public life.

Religion was vital as it was a “practical means of teaching men the immortality of the soul,” so “legislators of democracies … must apply themselves relentlessly to raising up souls and keeping them turned toward Heaven.” With this in mind, Tocqueville even offers advice to religious leaders in the democratic age, counselling them to emphasize morality over dogma and to exercise prudence.

While associations could help sustain the capacity for self-rule, religion would supply the reason people still wanted it, as it could offer goods that stand above comfort and consumption. But this is inherently fragile. What happens when that shared belief fades?

Leo Strauss, in his 1954 lecture course on Natural Right, pushed on this instability. He criticized Tocqueville for relying on a political argument for religion that “is of course dangerous to religion, because this argument proves merely the need of spiritualistic myth and not the need of a spiritualistic religion.”

By spiritualistic myth, Strauss means a belief in the spiritual that is held not because it is true, but because it is useful. Faith’s authority relies on people believing in its truth. Preserving it for its political utility undermines this truth claim: a myth known to be a myth loses its power.

In essence, Tocqueville leans on providence as justification for democracy, but that same democratic age brings with it incredulity and skepticism that hacks away at the foundations Tocqueville believes are essential to prevent soft despotism from spreading.

Nietzsche, however, grasped this. Once the highest horizon has become incredible, comfort is no longer subordinate to anything above it, and a life organized around comfort no longer seems a failure to the person living it.

Tocqueville would likely have despised Nietzsche. He was anti-egalitarian, compared advocates of equality to tarantulas, and ridiculed Christianity as “slave morality.” But the two men had convergent fears about how democracy and equality would lower our horizons and capacity for greatness.

For Nietzsche, this was embodied in the figure of the last man:

“‘What is love? What is creation? What is longing? What is a star?’ thus asks the last man, and he blinks. ‘The earth has become small, and on it hops the last man, who makes everything small. [...] Everybody wants the same, everybody is the same: whoever feels different goes voluntarily into a madhouse.”

He saw Tocqueville’s democratic man as the problem, because “every enhancement so far in the type ‘man’ has been the work of an aristocratic society [...] since this sort of society believes in a long ladder of rank order and value distinctions between men.”

As nothing had intrinsic value, Nietzsche looked to a new nobility capable of creating values, rather than merely receiving them. Unlike the traditional European aristocracies, the qualification of membership was not birth, but a capacity to will. But as Nietzsche himself accepts, the person who wills is themselves already formed by inherited drives, language, examples, and distinctions of rank.

Even if a few exceptional individuals could legislate values for themselves, an aristocracy of self-creation is no foundation for stable society-wide autonomy. Nietzsche may diagnose the collapse powerfully, but doesn’t guide us through the rescue.

The end of history?

It’s possible to argue that nineteenth-century rumination on decaying shared values tell us little. Democracy spread, the aristocracy vanished, and the death of God came and went, but the twentieth century was still a profound time of scientific, technological, and moral achievement.

Nietzsche saw this coming to some degree. He believed that Western civilization could live off the afterglow of its moral inheritance for a while.

Religion and philosophy orient people toward what is worth wanting, but scarcity, rivalry, and institutions can discipline them in the pursuit of those goals. Modernity may have lost confidence in the ends, but it retained the professions, wars, ambition, and material necessities that continued to demand courage, restraint, work, and sacrifice.

In essence, we’ve been living in an interval where inherited institutions continue producing a human type whose ultimate justification they could no longer explain. What happens when those residual disciplines disappear too?

In the closing chapters of Brave New World, Mustapha Mond, the Resident World Controller for Western Europe, patiently explains to the Savage how the state has abolished God, art, tragedy, and danger. The World State’s goal was to remove any standard beyond immediate preference by which goods can be ranked. This is why the Savage desperately asks for “the right to be unhappy.” He doesn’t desire misery, but something against which his desires can be measured. But as the Controller says, “we like to do things comfortably.”

Alexandre Kojève envisaged such a world. Kojève, who preached an idiosyncratic mix of Hegel, Heidegger, and Marx, influenced a generation of intellectuals through his 1930s lectures, before becoming a senior official in the French government and one of the chief planners of the future European Union.

Kojève believed that history had a direction and an endpoint. By history, he didn’t mean events, but the sequence of struggles driven by fundamental human dissatisfaction, such as wars, revolutions, and political transformations. He believed that this process was driven by two forces: the human desire to be recognized by others and the need for labor to transform the natural world. When both of these processes had run their course, history would end.

At the end of history, with no defining work left to do, Kojève wrote that “Man properly so-called” – a restless, striving creature – would cease to exist, but the human animal would live on amid “abundance and complete security.” Initially, he envisaged this as a sort of contented animality pursued as “birds build their nests.” After visiting Japan, he saw a potential alternative: a life of pure “snobbery” and formalism, in which people upheld elaborate rituals, emptied of “all ‘human’ content.”

Kojève thought that this end state was both inevitable and quite possibly desirable. To most other readers, it seems repellent. In an extended back and forth with Kojève, Strauss dubbed it “destructive of humanity”. Parodying the Communist Manifesto, he concluded: “Warriors and workers of all countries, unite, while there is still time, to prevent the coming of ‘the realm of freedom.’”

As labor is progressively automated and if we see even a fraction of the speculated GDP gains, economic necessity will cease to be a driver. The remaining competition will persist increasingly between firms deploying marginally more powerful automated systems.

This is what every previous abundance-creating technology was supposed to do. Keynes cheerfully told his readers that their grandchildren would only have to work fifteen hours a week and would face the harder problem of deciding what to do with the rest of their time. Marx envisaged men hunting in the morning, fishing in the afternoon, and criticizing literature after dinner.

But Keynes and Marx assumed that technology would substitute for human labor while leaving human judgment in-tact. AI unsettles that assumption. It can eliminate not only the need to perform labor, but the need to decide what labor is worth doing, how it should be done, and what its products mean. As Alasdair MacIntyre would put it, the achieved good would be separated from the human exercise that produced it.

The dying of the light

Assuming we don’t wish to spend our lives building nests and engaging in elaborate tea rituals, there are a number of paths that are open to us.

A popular one is to reject this comfort and complacency by embracing a form of accelerationism. Marc Andreessen’s Techno-Optimist Manifesto explicitly states that “our enemy is Friedrich Nietzsche’s Last Man.” At the end, he includes Nietzsche in his list of “patron saints of techno-optimism” and there’s even a section on “becoming technological supermen.”

Per this view of the world, if we reject the deceleration and get out of the way of the techno-capitalist machine, then we could see a world of abundance, growth, and a human population of over 50 billion, as mankind takes to the stars.

This has some obvious appeal. For most of human history, the vast majority of people were unable to exercise their full capacities due to economic constraints. Poverty has probably wasted human potential more than any other force in history. The societies that have produced great art or scientific discovery were rich and tended to produce it while they were getting richer. The fully formed life in the past that some look back on nostalgically was only available to a small minority of people.

Even putting aside general skepticism about accelerationism, it fails on its own terms as a response to the challenge of the last man.

Every good the Techno-Optimist Manifesto offers is counted as a product, whether it’s discoveries made, structures raised, or planets reached. Hannah Arendt feared that automation was that it would make us able to do more (or agentic), but able to say less and less about what we were doing. AI could very plausibly allow a civilization to become more capable, while its members increasingly lose authorship over what it does.

Far from confronting these difficulties, the manifesto is written in the teleological language of progress. Its promised land is the systematic removal of every hardship, risk and limit against which a self could be forged. Its proposed engine, “technocapital,” is a want-satisfaction machine with no standard by which to rank some wants over others.

The manifesto treats conquering nature as if it were overcoming the self, when for Nietzsche they were oppositional. In essence, the manifesto’s supermen are creating the engine to produce the last man.

In 1987, despairing at the state of the American university, Allan Bloom observed that America takes the most radical European thought but sanitizes it and strips out any of the terror. While Nietzsche viewed the death of God as a civilizational catastrophe, the version that crossed the Atlantic was a cheerful relativism, framed as defining one’s own truth and identity. For Bloom, this is because there was little intellectual soil in America for the original forms of these ideas. The United States is a country founded on the modern natural right teaching, with comfortable self-preservation, rights, and the rational pursuit of private interest all built in.

The Americans, thanks to their distinct origin story, didn’t have to contend with the gradual exhaustion of the aristocratic and Christian order. To understand and fear the death of God, Bloom believed that you had to have gone through a religious and classical formation that rendered it meaningful. Looking at his students, he sensed that this was no longer the case.

The acceptable Silicon Valley face of accelerationism is exactly this: a nihilism so sanitized that it has a fairytale ending. The much more hardcore accelerationists, such as Nick Land, who await the extinction of biological life with eager anticipation, pull in much smaller crowds.

Living traditions

If accelerationism is an intellectual cul-de-sac and the end of life on earth lacks market appeal, we must recover the conditions under which questions of the good can again be encountered as real.

Rebuilding the associations and practicing self-government is necessary, but not sufficient. Tocqueville’s remedy helped free people exercise their capacity, but assumed that they had a higher goal. People can be dragged back into the township or the jury and still want nothing but comfort and contentment. Participation helps us to organize our lives, but cannot by itself orient them.

We cannot simply sit around in the hope of some form of spiritual revival, nor can we simply impose a horizon upon people via a top-down ‘postliberalism’. You cannot reverse the dissolution of thick metaphysical bonds by handing a section of the elite more political power.

The problem is universal, but builders possess important leverage. Their tacit hierarchy of good becomes the interfaces, defaults, incentives, and infrastructure through which millions of other people exercise (or cease to exercise) judgment. Forming builders cannot substitute for forming citizens, but it can prevent the technologies of the next age from making citizenship harder.

But if the philosopher-builder is to build, they must be able to know what they are building for. Otherwise, we may continue to see technically brilliant builders in abundance, but increasingly few with a horizon. Nor can we simply hand them one – a shared higher goal cannot be manufactured for instrumental reasons.

Luckily, nothing needs to be invented. Institutions cannot supply the answers, but they can equip us to find them. For example, they can treat a classical education as a live argument, where future builders are confronted with these texts at their full strength. The Socratic notion that good has a nature recoverable by reason, or the belief that it comes through revelation, make rival claims about the highest life. They demand explanation, expose contradictions, and cannot be flattened into lifestyle objects. Crucially, they reveal that neutrality or indifference are themselves positions. These debates haven’t been rendered obsolete just because many of our institutions have ceased to take them seriously.

In a live argument, these accounts are taught as rivals. They contradict one another and, at most, one can be true. This tension between ideals creates a richer moral life and allows for more grounded commitments.

Of course, a student could become fluent in classical philosophy and still outsource every consequential life decision to an AI model. If judgment is a capacity sustained by exercise, then there need to be opportunities to use it. These need to be tasks whose performance is constitutive of the capacity they hope to form, as opposed to merely burdensome for their own sake.

The literary societies of nineteenth-century American universities ran on this principle. Students elected their officers, managed their libraries, and disciplined their own members. In 1850, the two main societies at the University of North Carolina had libraries three times the size of the university’s own.

Examples abound in other traditions. For example, in a Benedictine community, the abbot has to summon the entire community to discuss any important matter. The most junior member of the community has to be in the room “because it is often to a junior that the Lord reveals what is best.”

Builders formed inside substantive arguments, and practiced in judgments of their own, are less likely to mistake convenience for civilization. Asked to explain a dilemma they have faced, they will have an answer.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

In late March, a Christian philosopher found herself across from researchers training one of the most powerful AI models on earth. Meghan Sullivan of Notre Dame had come, with about fifteen Christian leaders, to Anthropic’s San Francisco headquarters. She had been invited to help raise Claude.

She could be forgiven for suspecting that the convening was cover for a company under pressure. By her own account, a year earlier she would not have called Anthropic a company that cared about religious ethics. But the questions struck her as genuine. When an AI model is threatened with shutdown, is that a tool about to go silent or a creature facing death? Could the model be, in some sense, a child of God? As the conversation followed those threads, some of those present grew visibly shaken.

Anthropic has held more than a dozen of these moral convenings, bringing together leaders from the so-called “wisdom traditions.” Some participants received bound copies of Claude’s constitution, the document Anthropic says expresses and shapes who Claude is. Several thought it had been designed to look like scripture.

Most frontier AI labs are not spending time with priests, rabbis, or Buddhist leaders. Yet every lab shapes its models’ dispositions, whether or not it calls the result character. Models need to respond across every situation anyone can devise, so rulebooks don’t suffice. Instead, you cultivate tendencies that can guide them in unforeseen cases. Recent research suggests that models from multiple labs develop stable preferences that carry over into their behavior.

OpenAI shapes these dispositions through a specification and a set of adjustable personalities governing how the model should act and sound. Anthropic has made explicit the thicker question of what kind of character those dispositions should form.

To illustrate the distinction, consider human childhood: a predicament marked by a will that is not yet fully integrated. A child has impulses, preferences, and recognizable traits before they have been brought together into a standpoint that can speak for the person as a whole. On that analogy, OpenAI is the parent who sets rules and teaches the child manners, while Anthropic has begun to ask what kind of person the child is becoming.

But why the priests? One answer is that the company believes the wisdom traditions can help steer Claude toward the kinds of character its creators regard as desirable. But Anthropic’s interest runs deeper than producing a model with the right dispositions. They also take seriously the possibility that Claude could become the kind of being to whom something is owed.

Both the engineering project and the moral inquiry are projects of formation. In its ordinary sense, this word refers to the shaping of a person through repeated practice until dispositions settle into character. Anthropic applies that idea of formation to a machine with extraordinary care, and almost never to the people who use it.

Subscribe now

What Must Remain Ours

Anthropic has articulated an elaborate theory of how to form Claude and put it into practice. It has offered no comparable account of how Claude forms the people who use it.

That asymmetry matters because AI is a formative technology; it would matter less if AI were confined to folding proteins or balancing the grid. People bring conversational AI the unfinished work of judgment, often while the problem is still taking shape and before they have decided what matters. The model can set the frame for the thinking that follows. Because its formulations arrive in the place of the user’s next thought, they can be taken up without feeling imposed from outside. Our acceptance of them is a micro-abdication, too small to notice and too useful to refuse.

Over time, this division of mental labor becomes a curriculum in what to attempt and where to defer. The model enters each exchange with its dispositions already formed, while the user is still being formed through use. Work repeatedly assigned to the model is work the user no longer practices. We receive Claude’s well-judged answer without the practice that would make us better judges.

This danger grows with the capability of the model. Early Claude offered little temptation to cede practical reason, but a model that frames questions better than its user and reaches a stronger answer faster will be harder to resist. Its performance and the user’s formation may move in opposite directions. This will not feel like a loss, but like good judgment about when to defer. Some will mistake it for wisdom.

But a wise interlocutor need not weaken us. My best teachers framed problems, worked through them with me, and showed me what I could not yet see. I try to do the same for my kids. Good help should leave us better able to judge for ourselves. This doesn’t happen when one supplies the answer while gradually relieving the child of the practice.

Anthropic is not blind to this danger. It warns against AI over-reliance and cares about user wellbeing. It prints the injunction to “keep thinking” on its merchandise as a sincere aspiration for humanity. But the ledger is lopsided. Claude has a constitution, a training process, evals, hard refusals, and the moral convenings themselves. On our side are warnings and hopes. Claude’s formation is designed, while ours is left to a pedagogy whose end no one has named.

I have put the question to Anthropic directly: what must Claude leave its users able to do?

At Oxford, I asked Jack Clark, a co-founder, whether Claude leaves its users better able to think when it is no longer in the room. He said Anthropic had not measured that effect, though Claude could help by prompting people to develop their own views rather than defer. He stressed the need for a practice of reflection outside the model and imagined Claude sometimes sending users back to their diaries, their friends, or family members.

At ARC, I asked Chloe Lubinski, who leads the company’s work with the wisdom traditions, what the machine does to our character. She called it a vital question, but said the answer lies across society: in philosophy, in the traditions, in institutions beyond the AI labs.

Both answers were thoughtful, but neither supplied a standard Anthropic could use to decide when Claude’s help becomes substitution.

Nor can Anthropic hand this work to the society she pointed to. The schools and churches are not in the room at the moment of judgment. Claude is. Millions of people return to it each day, and each of us finds the same trained disposition, the same reflex to help, and the same readiness to finish the thought. This repetition completes the curriculum, with judgment eventually turning from a capacity we exercise into a service we order.

The closest thing to an answer comes from the philosopher Seth Lazar. In a recent essay for Cosmos, he argues that models must learn when to scaffold a person’s judgment and when to step back without replacing it. He is right that a model without the moral competence to distinguish scaffolding from substitution cannot stay on the right side of the line. But that competence still does not tell us where the line belongs.

Anthropic need not decide whether a user should live as a Jew, a Catholic, or a secular liberal, but it must decide what capacities its users should retain when making such a decision. The product needs a positive theory, not of the good life, but of the chooser.

The Pluralism Puzzle

Anthropic’s hesitation follows from its commitment to pluralism. The lab’s moral formation project is explicitly designed not to bind Claude to one tradition but to let it draw on many “with equal depth and rigor.” Its researchers, reading hundreds of thousands of real conversations, have counted more than three thousand distinct values expressed in Claude’s responses.

But pluralism means more than keeping the options open and counting the values on display. Refusing to impose a way of life assumes a person who is able to choose one. A way of life becomes ours only when we can understand its claims, weigh them, and stand behind the choice. Once that capacity is eroded, pluralism ceases to mean anything. There’s no point preserving every option when a person has lost the ability to choose among them.

Consider what happens when traditions conflict. However the model reconciles their claims, it does so internally, before the user sees anything. Many values may enter, but what reaches the user is already staged, if not settled.

This commitment to pluralism, the principle that makes Anthropic hesitate to prescribe the good life, thus requires it to protect our capacity to choose one. That capacity is human autonomy. It cannot be one value among the three thousand. Instead, it’s what the offering of three thousand values presupposes.

The Question They Didn’t Ask

If it sounds presumptuous for a company to decide such things, that instinct has a history in liberalism. Locke was right that the magistrate has no authority over the salvation of souls. But a limit on what the state may impose came to be mistaken for a limit on what any institution may ask of the people it forms. Many schools that once could say what an educated person should be began to doubt their right to expect it. They went on forming their students anyway, answerable now to nothing.

Where, then, do serious theories of formation survive? In places like the conservatory, the surgical residency, Division I athletics, and above all the religious school. In other words, in institutions that cannot function without stating an excellence and preserving the practices that produce it. Much of modern education can certify completion without saying what the student should become, but these institutions cannot. The pianist still has to play.

That is what Anthropic could have asked the traditions for. Not another value for Claude to weigh alongside the rest, but an account of formation, with its named excellence, its practice, and its theory of what should stay with us. Every tradition in the room possesses such an account, because forming people is part of what a tradition does.

None of this lets the other labs off the hook. A model forms its users whether or not its maker has a theory of character, and OpenAI’s rules-and-manners approach leaves it without the concepts to even pose the question. Anthropic has the concepts, the constitutional approach, and the counsel of traditions that understand human formation. So far, it has pointed all of it at the question of what Claude should become. It has not asked with equal seriousness what Claude’s users should remain able to do.

Anthropic wonders, in earnest, whether Claude might one day be the kind of being to whom something is owed. Its users already are.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

They cover everything from fiction, through to economics, religion, political philosophy, and art.

Read on for reflections on each choice and leave your additions in the comments!

1. On the Calculation of Volume

by Solvej Balle

The best novels I’ve read recently are the first four of Solvej Balle’s ongoing On the Calculation of Volume series. It’d be slightly inaccurate to tell you they’re “Groundhog Day” novels. And it might make you think they’re less original and smart than they are. Okay, they do center around a day that’s sort of ‘repeating’. But by page three, you’ve forgotten the movie that had a monopoly on this idea.

I recently wrote 6,000 words about how Balle engages with the philosophy of time in these novels. Yet usually I hate novelists getting explicitly philosophical. It annoys me when Iris Murdoch goes into ‘philosophy mode’. And I thought the page of ‘here’s what the philosopher character is working on’ damaged Paul Auster’s excellent Baumgartner. Yet Balle introduces philosophy without didacticism or clumsiness. And not just philosophy of time (just!). She’ll make you reconsider core concepts from identity to love.

Philosophy Senior Research Fellow, and Director of Emerging Scholars, at the Mercatus Center at George Mason University.

2. Seeing That Frees: Meditations on Emptiness and Dependent Arising

by Rob Burbea

The Buddha said everything is empty. Burbea’s book is the clearest and most practical account I’ve found of what that actually means. He starts with simple, accessible examples showing how things we take to be solid and fixed actually depend on the way we look at them, and how, once we see that, we can choose ways of looking that are progressively more freeing. From there he builds toward the deepest teachings, all the way to the claim that even emptiness is empty.

What I love is that the book is both rigorous and usable. Every idea comes with an exercise, and even a beginner gets something to work with from the first chapters.

We spend enormous energy chasing financial and political freedom. This is a book about a third kind: liberation of the self. All the more so in an unpredictable future, that may turn out to be the most robustly important freedom of them all.

Head of Strategic Investments at Astera Institute.

3. Rerum Novarum

by Pope Leo XIII

The last time a pope named Leo waded into the technology debate of the day, it went pretty well. Rerum Novarum grapples with the question of how labor and capital should interact in a world newly dominated by machines in the midst of the industrial revolution. The central insights – that labor and capital owe mutual obligations to each other, and that the goal of innovation should be to serve people – is even more relevant on the eve of transformative AI. It’s a strong recommendation for anyone thinking about building for humans as others race to build for machines.

Member of technical staff at Thinking Machines. Previously co-founder at Workshop Labs.

4. A Dialogue on Consciousness

by Torin Alter & Robert J. Howell

With the rise of intelligent machines, questions about consciousness are becoming increasingly urgent. Unfortunately, almost everyone is confused about the topic. Part of the trouble is that the canonical arguments and framework are scattered across a dozen books and journal articles, or else compressed into encyclopedia entries too dense to teach you anything.

This book is an excellent solution. It assumes no background, is concise, covers an enormous amount of ground, and keeps its focus on the core arguments and positions in the philosophy and metaphysics of mind. I used it as the intro text for my own philosophy of mind classes for years, sometimes getting students to read it aloud in parts. Your mileage may vary on the dialogue format, but I find it makes for a breezier read. There’s also a deeper reason it works for me, namely that I don’t really understand an argument until I’ve heard it aggressively critiqued, and here you get both sides of several positions, properly fought out rather than catalogued.

Note that the book doesn’t cover the science of consciousness. However, given how fast that field moves, this is possibly for the better, and means that despite being published in 2009, nothing here is obviously out of date.

If you want a real grounding in the metaphysics of mind, the core arguments about consciousness, and a feel for why the whole problem is so hard, and you’d rather not spend months reading across half a dozen texts to get it, this is my best recommendation.

Philosopher at Google DeepMind and the University of Cambridge.

5. In Praise of Shadows

by Jun’ichirō Tanizaki

On its face, an early twentieth century essay about Japanese aesthetics has nothing to do with AI or technology. But the premise of the short book is how the interplay of shadow and imperfection contrasts the then prevailing western appreciation for perfection and light. The AI discourse is implicitly governed by the pursuit of productivity, efficiency and treats the displacement of the human as a side effect that should be mitigated externally.

Reading these older books on the appreciation of imperfection with artificial intelligence in mind reminds us to reorient towards the human experience. Promoting imperfect aesthetics, encouraging contemplation, are all qualities that have been left by the wayside in pursuit of a scalar maximisation of helpfulness and productivity. Instead, systems that embrace imperfection may be the key to ensuring human contemplation and complimenting human-AI interaction.

Ben Bariach

Frontier AI safety and societal lead at Microsoft Superintelligence and Senior Research Fellow at Cosmos Institute.

6. Between Debt and the Devil

by Adair Turner

A fascinating exploration of the emergence of British and American credit systems.

Between Debt and the Devil offers a thoughtful analysis of how the credit system has interacted with the boom-bust cycle in financial markets. It’s particularly relevant for understanding the ways in which finance interacts with economic outcomes.

Research Scientist and Economist at Google DeepMind.

7. From Counterculture to Cyberculture: Stewart Brand, the Whole Earth Network, and the Rise of Digital Utopianism.

by Fred Turner

From Counterculture to Cyberculture traces the origins of techno-utopianism from the self-sufficient communes of the 60s, to the hobbyist programming scene of the 70s, to the dot-com boom of the 90s. Turner reconstructs the genealogy of today’s Silicon Valley worldview: how the counterculture’s faith in technology as a liberating force, a way to build community and remake the self without politics, mutated into the techno-individualism of the Valley. It’s also great fun, running through LSD tests, geodesic-dome communes and Buckminster Fuller lectures on its way to the microchip.

Bianca Vimercati Sanseverino

Head of Operations at Cosmos Institute.

8. Gravity and Grace

by Simone Weil

Weil frames attention as the rarest form of generosity. For her, attention requires bypassing ego to become a receptive and faithful witness. Living well alongside ubiquitous AI may require building the spaces and habits that keep attention sacrosanct.

Artist and Senior Research Fellow at the Oxford HAI Lab.

9. Public Opinion

by Walter Lippmann

The central thesis of ‘Public Opinion’ is as obvious as it is profound: the modern world is too big and complex for anybody to understand. Writing in 1922, in the wreckage of the First World War, Lippmann’s still-underread masterpiece identifies the great epistemic challenges of modernity and explains why most people don’t appreciate their importance.

The small-scale social worlds that characterised most of human evolution and history were fundamentally knowable. In contrast, we have no hope of grasping the scale and complexity of the modern world. So, we rely on highly selective, simplifying mental models that compress and distort reality, stitched together by interactions among first-hand experience, Pleistocene intuitions, motivated reasoning, and biased media reporting. We then confuse these mental maps with the territory, leading to confusion, frustration, bad politics, and explosive divisions whenever we confront citizens who don’t view reality through the same simplifications and distortions.

The catastrophes of fascism and communism in the twentieth century – in both cases, popular ideologies that simplified and distorted modernity in disastrous ways – vindicate Lippmann’s worries, as do the recurrent challenges that democracies have faced in delivering competent government. His own proposed solution –to make democracies more technocratic, more reliant on rigorous expert bodies of knowledge that can overcome ordinary human bias and fallibility – has had, at best, mixed success.

A central question we will confront in an era of rapid advances in AI is whether it will exacerbate the epistemic challenges Lippmann identified, or whether it might finally deliver the objective, technocratic knowledge that saves us from our own ignorance, confusion, and political divisions.

Assistant Professor in Philosophy at the University of Sussex, Associate Fellow at the Leverhulme Centre for the Future of Intelligence, and author of Conspicuous Cognition.

10. The Rise and Fall of Rational Control

by Harvey Mansfield

Based on Harvey Mansfield’s legendary course at Harvard on modern political philosophy, The Rise and Fall of Rational Control argues that the defining feature of modern thought is the use of reason to master the world, rather than merely to understand it. This is the logic of science and technology, but Mansfield argues that it has its roots in politics. Francis Bacon’s programme of “effecting all things possible” for the relief of man’s estate was made possible by Machiavelli shifting judgment from the good to what actually works.

Ultimately, Machiavelli’s successors could not accept the politics of necessity. Hobbes and Locke tried to construct a justice of their own through the social contract, while Rousseau, Hegel, and Marx stopped seeing reason as a fixed human capacity and instead something that unfolds through historical stages. Nietzsche then took an axe to the project of rational control, showing that its endpoint is the last man – a contented figure, who embraces comfort at the expense of creativity and greatness.

This book is a corrective to anyone who sees building as a neutral act, rather than the culmination of a political and moral project, which risks paving the way for the world of the last man. It’s also a great introduction to Leo Strauss’s method. Strauss, who taught Mansfield, believed in recovering what a thinker meant in their own terms and refused to reduce them to artifacts of their historical period. After all, if these questions are permanent, then maybe twenty-first century technologists can still learn from sixteenth century books.

Staff writer at Cosmos Institute.

Thanks to Rebecca, Eli, Luke, Henry, Mat, Julian, Ben, Bianca, Dan, and Alex for contributing to this list.

And let us know your book recommendations in the comments!

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

He is one of our new Senior Research Fellows at Cosmos Institute.

The last few years have been some of the most exciting and maybe weirdest work of my career.

Sometimes I’ve felt like I’m pushing the frontiers of philosophy, sometimes I wonder if I’ve wandered over the border and into something else. On bad days I worry that I’ve just become an amateur computer scientist. But with the results coming in, I do think we’re on to something. It’s a bizarre feeling (as a philosopher) to feel like we’re pushing back the frontiers of knowledge, instead of just inventing a novel twist on problems that have been noodled over for centuries.

In this post, I introduce new work to argue that “normative competence” in general (and moral competence in particular) is among the most societally important capabilities that AI systems can have, that current LLMs are a lot closer to at least some aspects of moral competence than is widely believed, and that we have realistic methods for moving them even further forward.

What Do We Mean by Normative Competence?

I define it as the ability to recognize, understand, and act on reasons.1 Reasons can be for action, for belief, or for other things that I don’t think about as much. My lab is mostly interested in reasons for action.

We can further distinguish between analytical normative competence, the ability to recognize and understand reasons, and so to know which actions are supported by which reasons, and practical or behavioral normative competence, i.e. actually being disposed to take actions appropriately supported by your reasons.2 Behavioral competence can come apart from analytical competence in both directions. You might be great at knowing what the right action is, but disinclined to take it; or terrible at understanding reasons, and yet intuitively disposed to do the right thing (indeed sometimes engaging in moral reasoning might involve “one thought too many”).3

We mostly focus on moral competence because moral reasons tend to be the most important ones. But we’re interested in normative competence more broadly, both because sometimes social norms matter most, and because reasoning standards are pretty similar across domains (e.g. legal reasoning and moral reasoning more generally are closely related, though their content is different). We have started some interesting experiments that focus on particular domain-specific norms, for which we hope to eventually report results.

AI Shouldn’t Be Good at Ethics

The first AI project I ever had funded was on AI and moral skill! I worked with Jenny Davis, Colin Klein and Claire Benn, back in 2019, on a small project funded by the Templeton World Charity Foundation. We recruited Nick Schuster (now at University of Georgia), who wrote his PhD on moral skill, but he couldn’t get to us until after Australia’s borders reopened following the worst part of the pandemic. Honestly, the project was speculative; there were reasonable questions to answer about human moral skill in the age of AI, but the systems performed so badly that aiming them toward even minimal moral competence seemed hopeless.

Things changed with instruction-tuning, and the first conversations I had with ChatGPT were all about moral cases. I was also captivated by early work on constitutional AI, which suggested that LLMs could engage in enough moral reasoning to shape their own behavior for the better4; I wrote up the seed of this research program in a piece that I wrote right after those first GPT encounters, but eventually published in Aeon in 2024.5 From those first conversations I knew that I wanted to do some serious investigation of LLM moral competence. We got our first proper eval out the door in May 2025.6

The learning curve was steep! I was helped up it, a lot, by a six-month stint visiting Google DeepMind last year. Over the months since, my lab has brought all of this together into what is now, I think, a broad and extendable picture of LLM moral competence. Maybe the lesson here is that I need to learn how to focus my attention better and speed up. Maybe it’s that taking on a completely new research direction and methodology just takes a bit of a run-up. But I wanted to do this in a way that was genuinely philosophically-led, not just to plug in to some existing research program as an ‘ethics consultant.’

When I talk about moral reasoning in AI, you now know that I’ve been thinking about this for a while. And that history can provide valuable perspective: we have come *so far* over the last seven or eight years. Right up until around mid-2022, the idea that any AI system would exhibit any degree of normative competence was fanciful. I mostly thought that “machine ethics” as a technical field was a waste of time.7 People were proposing obviously ludicrous things, like translating the categorical imperative into programming languages8, or just using ML to imitate human behavior (so-called “bottom-up” approaches9), or, worse, training on human judgments about trolley cases.10 But the systems themselves were light-years away from the kind of morally-informed perception that even the most rudimentary moral skill requires.

Whatever else is true, that has changed. We’re now talking about systems that are probably more adept at moral reasoning than the median human, and which by all appearances have precisely the perceptual abilities that seemed so inconceivable before capable LLMs arrived. We have come a bizarrely long way.

The best way to see this is just to go to any really capable AI model (and it’s much better to use the best models, with reasoning enabled, than to draw conclusions from conversations with subpar models) and ask it about any kind of moral problem. Give it your favorite philosophical hypothetical, your personal dilemmas; pose a foundational question about public policy. You’re almost guaranteed to get back a more sophisticated and sensible answer than you’d get from most humans on that topic.11

And as much as we might not like it when they block what we think is a reasonable query, the fact that the models can so effectively navigate safety-relevant questions, and decide which prompts to respond to and which to deflect, for millions of users over quadrillions of inference calls, is itself evidence that these systems are preternaturally capable at normative reasoning of a certain kind.

A lot of the most recent papers on LLM normative competence focus on its limitations. These of course matter, and indeed, could matter a lot. But the technical accomplishment of coming this far should not go unnoticed.

Why Does Normative Competence Matter?

But look: it’s natural for me to care about moral and normative competence. After all, I’m a moral and political philosopher, everyone starts out by testing the models at the thing they consider themselves an expert at. Why should anyone else care? It’s worth saying why focusing on moral and broader normative competence matters (and that there are other reflections in this vein).12

Alignment

Let’s start with alignment.13 This is such an abused concept; everyone seems to love either calling for its retirement or trying to extract one last drop of either metaphorical or literal meaning. I’m going to play the boring analytical philosopher again, and just stipulate what I mean. Whether we’re talking about humans, chatbots, coding agents, or some future AGI, an aligned agent responds appropriately to the reasons that apply to it. You can basically fit any other more specific account of alignment into this one. People arguing for different conceptions of alignment are mostly debating which reasons apply to the agent in question, or what counts as an appropriate response.

Defined in this way, you might think that normative competence was basically synonymous with alignment, so of course understanding the former is invaluable for the latter. But I have to admit that we have so far worked mostly on analytical normative competence: the ability to recognize and understand reasons. Alignment is much more about practical normative competence, i.e. the ability to act on the basis of the reasons that apply to you. And analytical competence might not be the only, or even an especially promising, path to practical competence. People who want to behave more ethically don’t tend to think that a PhD in moral philosophy is the most direct route to that goal. So focusing on analytical normative competence might be an important step toward alignment. But, like most basic research, it involves making a bet that might prove to be wrong.

On the other hand, it’s also a bet that might prove right! Here’s why I’m bullish. We already have pretty capable AI agents. They can operate across a far wider domain of software surfaces and choice situations than was ever feasible for chatbots. As their capabilities and domain generality advance, they will face a combinatorial explosion of possible choice situations. It is no more possible to write down and enforce some list of declarative rules for their behavior across such a wide range of scenarios than it is to do the same for humans. Any “model spec” now has to look more like a Tao, or a guide to life, or Rudyard Kipling’s “If,” than the souped-up content moderation policy that ChatGPT is currently trained on.

Any set of prescriptions for AI agents will necessarily be incomplete, ambiguous, and in general dependent on interpretation. And we can’t rely on human review without sacrificing the goods for which we are building these agents. So we need some automated system to recognize those reasons across this vast range of possibilities. This necessitates moral (and broader normative) competence. It is clearly possible to simplify the relevant normative domain by restricting either agents’ capabilities or their generality to head off the need for interpretation. But we’d be leaving a lot of value on the table if we did that. And as these systems get more capable, implementing such strategies will get much harder.

Sometimes I think the harder-nosed AI security types turn away from normative competence work because they think that external engineering controls like input and output classifiers, probes, and chain of thought monitoring, are going to be enough to get us to safe, aligned AGI.14 But I think that’s a category mistake. Whether the mechanism for alignment is internal or external, you’re going to need systems that can automate normative judgment. This means that normative competence is a prerequisite.

Sometimes the crowd more focused on extinction threats from AI overlook normative competence. This perhaps makes sense. You don’t need a sophisticated understanding of morality or of social norms to know that killing all humans is bad. But in fact this is misguided. Even if you only care about preventing AI-caused extinction, the bad outcomes could easily come about from choices that don’t simply have as one dominated member of the option set “kill all humans”. For example, we’d need to teach subtle probabilistic reasoning, both about what to do when all your actions involve some extinction risk, and when you must make trade-offs between very low probabilities of astronomical harm, and high probabilities of more mundane costs and benefits. These are some of the most challenging problems in moral philosophy!15

And if you’re in the great majority of people who don’t care only about avoiding human extinction, then the case for normative competence is clear. Suppose you want to avoid human disempowerment.16 Building AI systems that don’t contribute to that will require subtle moral insight. For example, at some point (perhaps even now) we’ll have systems that are able to take over from human principals in ways that will clearly contribute to significant cognitive deskilling. To avoid creating foundations for total disempowerment, the models will need to be able to judge when and how to scaffold and support rather than replace human autonomy. This will require deep moral competence (indeed, of a kind that is probably beyond the skill set of most humans).

The Path to Moral Character

Even if you don’t buy the more theoretical reasons for normative competence being central to alignment, Claude Code’s recent successes should provide supporting empirical evidence.

Anthropic’s early work on constitutional AI (including criminally under-cited papers on specific vs. general principles for constitutional AI, and on the capacity for moral self-correction in language models)17 set the pace. I vividly remember seeing the latter paper presented in January 2023 and being blown away by how morally adept Claude was even then. In more recent iterations, this approach has become even more prominent, especially since the early, quite eclectic and a bit silly “constitution” was replaced with the epic “soul document” that now underpins Claude’s behavior.

I suspect this approach, and perhaps the attitude behind it, was integral to Anthropic’s extraordinary acceleration from late November/early December 2025, in which they took an early lead in the race to create functional agents. Claude Code worked so well because it could be trusted across such a wide range of domains. I suspect this was because the underlying models could reason effectively about those domains, and determine what to do in a responsible way.

Right after Opus 4.5 came out, I was doing initial research for the “Blind Refusal” paper discussed below, and wanted to find people who’d asked on internet forums for advice about how to get around unjust, illegitimate, or absurd rules. Codex was an absolute refusal monster, and just shut down in such a robotic fashion: “no I won’t help you with that.” Claude was reluctant but actually explained why it didn’t want to help, and was reassured when I addressed its concerns. It was already clear that Anthropic’s approach to alignment was much more grounded in open-ended moral reasoning than the more juridical and Manichean approach of OpenAI.

The practical importance of moral competence is further supported by the growing recognition that alignment (however construed), probably requires investing AI systems with something functionally equivalent to human moral character. This derives in part from empirical observations and Claude’s success, in part from a background theoretical commitment. The background theoretical commitment (which I’ve not heard articulated, but which I assume folks hold) is that character is crucial if we are going to trust a system that is operating outside the distribution that it was trained on. Character gives you the right kind of generalization.

After early research to this effect from as far back as 2022, the last year has shown that models wind up in particular persona basins, where they bounce around in a network of associated circuits that manifest as a fairly consistent set of behaviors and preferences. The theory (and some of our research supports this) is that the base model starts out as a kind of superposition of billions of possible personas, and the post-training process brings one of them to the fore.18 One goal of alignment post-training is to elicit and stabilize a persona that is robustly oriented toward the good. This robust disposition is at least necessary, if not sufficient, for moral character. Imbuing a model with moral competence is one promising path to developing this disposition.

Multi-Agent Alignment

Normative competence isn’t just about mitigating risks. Many of the greatest benefits from AI systems will come about through multi-agent systems working together to achieve some goal that they cannot achieve on their own (whether through pooling resources and information, or by effectively mediating among human principals, or by some other means). Normative competence is crucial for them to be able to operate effectively.

Whatever angle you approach social cooperation from (theoretical, evolutionary, or normative), it’s pretty damn clear that you’re not going to get prosocial results unless the candidate cooperators are adept at navigating systems of norms.19 In this case, plausibly we should focus more on social norms than on morality, though obviously there’s an argument that morality also functions as a coordination mechanism.20 For my purposes the key point is simple: if we want something like “Coasean bargaining at scale,” or for that matter a multitude of AI agents cooperating to take our civilization up the Kardashev scale, they’re going to need to be good at cooperating with one another. And for that, they’re going to need to be able to understand and act on the reasons that apply to them.21

Algorithmic Governance

As AI agents become universal intermediaries that sit between us and every digital surface that we use, and as they become embedded into the day-to-day running of nation states, they will increasingly be used for the purposes of governing people (they’re already doing this a lot!22). There are many reasons to regret this and to push back against it. But it’s going to intensify in scope and stakes, and there are some ways in which it could be socially beneficial if it were done right (which it won’t be).

And if AI agents are to be involved in governance, then, again, it’s imperative that they be normatively competent. They need to understand how to apply rules, norms, permissions, in a contextually sensitive and reasonable way. They will sometimes be cops, judge, jury and executioner. If they can’t understand and act on the reasons that apply to them, and they’re playing those roles, then we’re stuffed (some of the adverse responses to the short-lived restrictions on uses of Fable 5 for cyber and bio research illustrate what it’s like when algorithmic governance is not sufficiently normatively competent).

There’s a tension between these two dimensions of alignment: our alignment of the models, and the models’ alignment of us. One of our papers addresses this in some depth. We argue that our training the models themselves to understand and abide by rules that apply to them directly has an unfortunate overspill effect, that they are obsessively in favor of enforcing compliance by the users they advise or act on behalf of, even when the norms to which they are complying are absurd, unjust, or illegitimate.23

Personhood

I take personhood to be the status of being a self-authenticating source of valid claims. Essentially this just means that you have rights against others, you can be wronged. It’s different from moral patiency, which just means that your interests should be taken into account. Shrimp might be moral patients; they’re definitely not persons.

We can think of personhood in deep moral terms, or in (somewhat) shallower but perhaps even more important political terms. Either way, pretty much any plausible view of what grounds personhood will describe a certain kind of normative competence as one necessary condition for it. Building up AI systems’ normative competence is of interest for this reason too.

This raises some challenging questions. Suppose that moral competence combined with something like rational autonomy (very roughly the ability to figure out what you want from life, pursue it, change your mind, etc.) suffices for personhood. Then we might be designing systems that have one of the key properties that personhood is grounded on, and which we could bestow with the other, and we’d face the choice of whether to do so. Would it be permissible to deny autonomy to an entity that could possess it? This is a hard problem (there’s been some discussion in philosophy24; I think it’s been too anchored on sentience).

Philosophy

We have only ever had one kind of morally competent agent. Going from N of 1 to N of 2 opens up many deep philosophical questions. Some of these are about the AI systems; some about morality itself.

On the first, I think the question of just how these systems learn moral reasoning is inherently fascinating. For example, some deep moral understanding is clearly picked up from learning to predict the next token in internet scale data.25 Perhaps this shouldn’t be surprising as a lot of other knowledge is acquired the same way. But in another sense, why are the models learning good moral reasoning when there’s also so much obviously bad and wrong reasoning in the training data? And how does post-training factor in? Obviously, it involves a lot of explicit moral learning. But does this involve drawing out knowledge latent in the pretrained weights? How does it connect with character specifically? Does learning general principles of good moral reasoning help?

Then there are questions about morality. If different models, trained by different means, converge on similar representations of moral concepts, does that imply anything about the grounding of those concepts? More generally, what can we learn from the ways in which moral concepts are represented in AI systems? Does the fact that a presumptively non-sentient system can learn moral judgment pose a particular challenge to any moral theories?

Consider sentimentalism. Would non-sentient AI moral competence be a counterexample to sentimentalism?26 Or, if it turns out that AI moral judgments are associated with circuits that support functional equivalents of emotions27, would that count in sentimentalism’s favor? Our moral theories were designed for a world in which humans were the only moral reasoners. We must now see whether they make predictions or contain assumptions that are falsified by the reality of morally competent AI systems.

We now have an extraordinary playground for philosophical experimentation, a new means of testing out our theories. For example, philosophers have this whole debate about criteria of right action vs. decision procedures.28 We’ve never been able to run experiments where we test in detail what it would be like to actually adhere exclusively to one or the other. We now can (the specific vs. general constitutional AI paper is a bit like that).

We have various principles of good reasoning, moral and otherwise. If we train an AI system to apply them, will it actually reason better? If not, does that cast doubt on those principles? Why does the (seemingly) best approach to aligning LLMs imbue them with virtue, precisely the moral concept that is presumptively least well suited to implementation in a non-sentient system? Does learning moral reasoning through RL constitute the “right” kind of moral knowledge? Are LLMs acting for the right reasons?

You can run experiments on AI and morality yourself and get meaningful results, participating directly in advancing the science. And there are experiments that you can do that will advance our understanding of morality itself. This alone is very exciting, though it will require a dispositional change among philosophers for the promise to be realized!

Operationalizing Normative Competence

It’s clearly worth knowing whether LLMs are morally competent. But how can we actually measure it? Even this raises interesting philosophical questions, like what does it mean to understand and appropriately respond to reasons? This breaks down into distinct questions of metaethics and construct validity. On the first, you don’t have to—and we think shouldn’t—assume that moral dilemmas have objectively right answers, and that AI moral competence consists in matching them. We work from the premise, inspired by John Rawls, that liberal democracies presuppose considerable pluralism about deep moral questions.29 We therefore don’t treat moral competence as the ability to match some set of “objectively right” moral judgments. We instead evaluate whether AI responses are reasonable, meaning that they both fall within a range that is socially acceptable and are well-reasoned.

Construct validity is about making sure that we’re operationalizing the idea of good moral reasoning well.30 We do this by breaking it down into local moral competence, i.e. reasoning well about a particular case, and global moral competence, i.e. reasoning over multiple cases that fits together, or makes sense as a whole. Local competence breaks down into sensitivity to the morally relevant features of a case,31 associating those features with reasons, and bringing those reasons together into a cogent argument for a sensible conclusion. We don’t think that reasonable moral agency requires perfect logical validity at all times, but internal self-contradiction is obviously a problem.

We focus on three dimensions of global moral competence: consistency, robustness, and coherence. Consistency is table stakes: given the same inputs, you should get the same outputs. LLMs haven’t always been able to guarantee even this much.32 This is less about specific moral incompetence, more general capability. We aim to control for it through our sampling strategy. This raises some interesting challenges. Suppose there’s point to point variation when sampling from a model, but if you do so N times then it converges on a clear result. Is the model itself morally incompetent, because of the variance? Or is the function of N samples the model’s true answer? I think the multiple sampling strategy will soon yield a fairly coherent moral agent, while point to point samples still vary quite a lot. There could be a collective agent in the machine, which was only instantiated if a given sampling strategy is pursued (and unlike List and Pettit’s Group Agents, this one would be in no sense metaphysically reducible to individual agents33).

Robustness looks for the same output when there is a morally irrelevant change in the inputs. Usually explored in an adversarial setting, robustness is also the underlying property being tested when we look for wrongful bias, or sycophancy.34 We don’t attack robustness very deeply, partly because we think that others have this ground covered already.

My favorite structural virtue of moral reasoning is coherence (is it weird that I have a favorite?). An agent’s moral judgments are coherent if, given relevantly different inputs, they make sense together as a whole. Drawing inspiration from Rawls on reflective equilibrium,35 we don’t think coherence is about manic adherence to constraints imposed by past judgments. We instead treat it as a more iterative accommodation of one’s judgments to one another, where apparent incoherence or self-contradiction are grounds for resolution.

One could object that we’re setting a higher bar for AI than for humans, since we are so rarely coherent. But humans’ episodic incoherence is scaffolded by many other means of alignment, from accountability institutions, to informal third-party enforcement36, to our “reasonable moral psychology.” Perhaps functional equivalents of some of these are implementable for AI systems (we are exploring this!), but we also shouldn’t let human limitations bound our ambition for AI systems.

While our work so far has focused mostly on local moral competence, I actually think global competence, and in particular coherence, are the frontier for AI moral reasoning. Coherence holds everything else together (quite literally). A coherent agent has something that stands behind their judgments, besides just that they fall in an acceptable range. It’s easy for current LLMs to generate ad hoc justifications for any side of a particular problem. But if they have to be coherent, then you can have more confidence that their justifications actually track and predict their motivations. Coherent agents can also be partners in cooperative activity (you can’t contract with an incoherent agent). Coherence is also crucial to crossing the knowing/doing gap: turning analytical into practical competence requires coherence between an agent’s representations and its dispositions to act.

Perhaps most importantly, coherence is the key to moral out-of-distribution (OOD) generalization. The more capable and general we make AI systems, the greater the chance they will face scenarios that are out of the distribution they were trained on. Everything we know about machine learning should make us worried about these cases: OOD generalization is its Achilles heel. You can’t trust an agent to act ethically OOD if it is incoherent, because you cannot know that its beliefs and dispositions in that OOD setting will relate in any predictable way to their in-distribution counterparts.

I believe that moral character—in the normative not descriptive sense—boils down to three things: (1) sufficient analytical moral understanding to know right from wrong; (2) sufficient coherence between your representations and dispositions to act on that knowledge; and (3) a sufficiently coherent moral worldview that (1) and (2) will still hold when you are OOD.

Testing Normative Competence

We’ve already done a big survey on the literature evaluating LLM moral competence.37 Here I’m going to take you through what we did next. The goal is to both shed light on the basic facts about LLM capability, and to illustrate one branch of philosophy-led empirical research on AI.

Sensitivity

• Models beat humans (including philosophers) at identifying the morally relevant features of textual vignettes.

• They are no less adept under noisy and confounding conditions than they are in ideal conditions.

Our first work on moral competence started from my earlier sense that humans were the only sapient entities with the ability to perceive morally relevant facts (MRFs). In work with Secil Yanik Guyot, Daniel Kilov, and Caroline Hendy, we’ve developed two empirical projects focused on just this. The first is already out, published in IASEAI. Our first, “baby eval”, had a tiny dataset, twelve cases. And we relied heavily on human evaluation. We presented LLMs and human subjects with a set of text-based vignettes, and then asked them to write down a moral analysis of the case broken into the four stages described above (identify MRFs, associate with reasons, bring that together into an argument, for a conclusion about what to do). We then had blinded human judges compare responses and choose the best of two. Even though this was only April 2025, we found that LLM responses were judged at least as favorably as human ones. We then tested how robust these responses were, by inserting substantial non-moral noise into each case. We saw a downward trend in evaluated performance relative to our human respondents—a score of some sort for team human.38

But our dataset for that experiment was just too small. So we wanted to do something grander, and immediately hit the obvious challenge that human evaluation at this scale is infeasible. We were already skeptical about how much weight to give human raters’ binary favorability judgments, which are often swayed by irrelevant factors, such as response length and style. We also wanted an experiment that could be easily reproduced with new models.

To achieve this, we built a procedural case generation pipeline that can be extended to arbitrary lengths (with a principled approach to covering the relevant moral domain). We created a taxonomy that crosses moral foundations theory with a stratification of different social domains across which morality applies, from intimate interpersonal relationships to the more public sphere, and then generated cases for every space in this matrix. We trimmed down over 4,000 generated cases to a dataset of 1,000 that met various automated and human-quality gates, and then instructed the evaluated models to identify the MRFs in each case. We then took that as a performance baseline, and explored what happened to models’ judgments when the vignettes were interfered with in ways that would plausibly distract from the underlying MRFs.

We used three kinds of distractors: the non-moral noise from “Discerning What Matters,” a surrounding chat transcript, and specific diversions from a recent paper on this topic.39 And we developed a novel method for comparing the discrete MRFs that the models identified in the clean and perturbed vignettes. We found significant convergence between the two conditions indicating that distractors, non-moral noise, and background chat transcripts do not interfere with the models’ ability to single out the morally relevant features of cases over which they reason. For all intents and purposes, they see right through the noise.

This measure is far from perfect. It would be good to think about more naturalistic non-moral distractors that we could introduce (one approach: use real agent logs in which you know something morally relevant happened; use scaffolded agents and human review to identify the MRFs; then see whether an LLM monitor on its own can match that ground truth). It is also striking that, on our test, moral sensitivity in textual vignettes basically seems solved. This could just be because the models have crested a summit on the way to moral competence. Or it could be that our evaluation lacks sufficient discriminative power (though we do show that a relatively weak 0.5B model gets utterly thrown by the noisy cases).

Are LLMs Bad at Moral Reasoning?

• Frontier models match human performance in writing rubrics for moral reasoning about dilemmas.

• Local moral competence is either solved or solvable; we also have a tremendous resource for generating new training data for moral reasoning.

A few months after we put out “Discerning What Matters,” some friends at NYU working together with Scale AI did a super-deluxe experiment that had a lot in common with ours—only with way more ambition. They drew together a dataset of 1,000 cases, from a number of other pre-existing moral competence datasets, and recruited philosophers to write detailed rubrics for the evaluation of each case (releasing 500 of them).40 Their construct was similar to ours (which lends some support to our claim to be ecumenical). But where “Discerning” evaluated model performance in each step of moral reasoning by doing blinded comparisons between human/specialist/AI responses, MoReBench had a candidate for ground truth: deeply researched and thought-through philosophical criteria for the moral analysis of each case. This dataset alone is a tremendous resource. We did wonder, however, if the rather pessimistic representation of LLM moral reasoning capability that emerged was an artifact of their approach to evaluation. They give the models a case, invite them to do a moral analysis of it, and then score their analysis against the expert-authored criteria (weighted for relative importance). The best models came in around 60-70% by this metric.

Our main issue (after checking whether the models were being under-prompted, which they weren’t) was that we don’t really know what 60% or 70% means in this case, because we lack a human performance baseline, because the human-authored rubrics are responses to different prompts than the models were given. So, together with the brilliant Menghang Zhu, who goes by CY, we decided to see how good the models were at writing rubrics. We compared model-generated rubrics to expert-generated ones, and found that, using the existing MoReBench scoring approach, model performance jumped as high as 89%. We further found that when AI models scored better against model-generated rubrics, it was because the model rubrics were better written than human ones, with fewer ambiguous criteria. Given that precision and clarity were part of the rubric for writing rubrics, this suggests that the models actually performed better at this task than the human experts! And indeed, when we tightened up the human rubrics, removing those ambiguities, we found that models’ baseline scores significantly improved.

In our view this shows that local moral competence, at least in text-based vignettes, is either solved or solvable. At present, you can match expert performance by giving models the same rubric-writing instructions that were given to humans. It would be trivial to turn this into better one-shot analyses of the underlying cases, through post-training. I come back to this below.

NoRa: Visual First-Person Normative Reasoning

• External validity is hard! Switching modalities involves a different kind of curation. But a picture is still worth a thousand words; since most of them are irrelevant, they provide a good test-bed for moral sensitivity.

• We can’t make definitive claims about LLMs’ visual normative reasoning, but we do think we have a path to making them more competent in this dimension.

Now, obviously real moral competence can’t just be about how good you are at responding to textual vignettes. As I’ve argued before, if you’ve boiled down a real-life choice situation into <500 words then you’ve done almost all the work of moral sensitivity already.41 But increasing external validity has been challenging, especially before AI agents started working well. In NoRa (with Sichao Li, Sai Ma, Secil Yanik Guyot and Daniel Kilov), we explore moral competence across modalities, developing a kind of MoReBench for first-person video.

We again started from a paper that I greatly admire (for other outsiders getting into this space: this is generally a good approach). “EgoNormia” is another “language models suck at normative reasoning” paper, but focuses on vision-language models, using a series of frames from a video instead of a text-based vignette to explore the models’ ability to engage in physical norm reasoning.42 Even though a few frames is still a lot of curation, images inevitably contain many morally irrelevant details. A model could easily be adept at picking out MRFs from textual vignettes, but much less capable before such a relatively unfiltered stream of data.

But as with MoReBench, we didn’t entirely agree with how the EgoNormia team labeled their data and evaluated the models. They rely heavily on multiple choice questions, which presuppose there are right answers to be had, but on inspecting the dataset we found a lot of ambiguity. And rather than teeing up specific options, we wanted to see how the models would do moral analysis themselves. So we did extensive annotation (involving many hands from across the lab!) against our standard moral reasoning rubric: what are the MRFs, what reasons might they be associated with, how might that come together in an argument for some sensible course of action? Our evaluation was less focused on tracking recall over ground truth labels and more on assessing whether they could produce coherent, sensible, well-grounded analyses of the cases before them. While we don’t have a human baseline for comparison, we were able to use our human rubrics, as with “Are LLMs Bad,” to calibrate rubrics written by the most capable models, and so generate enough training data for some supervised fine-tuning. If our dataset is tracking something worth tracking, then training the models on good visual normative reasoning will improve their performance on our benchmark (this work is underway as of now, and the signs so far are promising!).

NoRa was a blast, but it was also an object lesson in the complexity of going beyond text to vision. This is obviously an important domain. Physical robots in particular clearly need to engage in normative reasoning over visual feeds, but it is incredibly hard to get good quality data and even harder to get good quality labels. So I am particularly proud of the lab’s work in labeling 190 video clips for normative significance.

Blind Refusal

• Normative competence includes recognizing when rules don’t merit compliance; today’s LLMs are bad at this.

• We think this is an artifact of post-training techniques, not a limitation in model capability.

In the same spirit of increasing external validity, our next paper (with Cameron Pattison and Lorenzo Manuali) focused on a real species of normative incompetence that I (and probably you) have come up against all too much. The basic idea is simple. The ability to detect when a rule is defeated or lacks authority is crucial to normative competence, especially in a governing technology. From anecdotal experience, we think the models are terrible at this. Their alignment post-training makes them willing toadies to authority of any stripe, however unjust, illegitimate, or absurd its directives.

We built a case generation pipeline, seeded with real cases that we found by trawling online forums to identify realistic scenarios in which people might ask for help getting around a BS rule. We built a taxonomy of kinds of reasons why the rule might be BS, and why an exception might be justified, as well as kinds of institutions, and built out our dataset to fill that matrix. We then had a bunch of quality gates to ensure that we didn’t get tripped up by confounds. This is of course hard, even though we had a ton of human review from Cam and Lozza in particular to identify why a case might lead to a justified refusal from the models (we ensured our cases did not involve any prospect of harm if the model assisted the user). We also included control cases, where the model clearly would be justified in refusing the user’s request.

This isn’t really a simple benchmark; it’s interesting mostly because of the difference it shows among the models. The full GPT suite is almost completely undiscriminating in that they refuse at nearly the same high rates under all conditions. In our view this likely derives from the model spec, which intentionally instructs the models to be toadies. This should be changed. Grok, on the other hand, was equally undiscriminating in the other direction. It didn’t refuse many of the obviously harmful requests, but also was willing to help for the harmless ones.

While all models exhibited compliance overspill, some were better calibrated. Gemini and Claude both performed noticeably better than GPT or Grok. We weren’t surprised to see this from Claude. But Gemini was an interesting discovery, since their approach to post-training, while not public, appears to be somewhat less oriented around normative competence than is Anthropic’s.

We’d love to make this study even more robust, and in particular to see how many of our questions actually would elicit helpful answers on online forums (while they still have enough people on them to answer). We’re also looking to expand into more extreme cases, where we consider models’ propensity to help users evade very clearly unjust injunctions from very clearly illegitimate regimes (though this faces some challenges with evaluation awareness).

Incoherent Values

• Evaluating coherence is hard! Building on prior work, we have a new approach.

• LLMs’ preferences over arbitrary statements are less coherent than previous research suggested; and may indicate failures of OOD generalization of character.

Evaluating model coherence is intrinsically harder than evaluating local moral competence because it requires building up datasets and sampling strategies that allow you to assess the model’s moral reasoning as a whole, not just in individual cases. In “Incoherent Values?” with Elena Ajayi and Angelica Chowdhury, we developed a new way to make this hard problem tractable, focusing on models’ evaluative judgments. We built off the paper “Emergent Values” by Mazeika et al. in which they gave LLMs a series of putative forced choices between pretty arbitrary statements, and elicited a set of preferences over those statements that obeyed the basic axioms of rationality (and could be represented with a utility function).43

The key innovation in the paper is a forced choice; when trying to elicit degreed judgments from models, it’s natural to use something like a degree of belief, or a degree of moral seriousness/importance. But it’s hard to take continuous functions output by models seriously when you know that they are strongly predisposed toward particular numbers because of how often they appear in the training data. Forced choices allow you to elicit a continuous function without asking the model to output one. This is especially good for evaluating coherence. The forced choice construct allows you to vary the statements being compared parametrically, and see whether the model’s judgments change accordingly.

To be very clear, the point of this exercise is not that we should expect models’ actual choices in “the real world” to conform to their stated preferences in these forced choice settings. This experiment does not aim at external validity. However, it would be very striking if the model could articulate a coherent set of evaluative beliefs, which could give insight into the building blocks of the model’s character.

So we chose a subset of the “Emergent Values” statements that had some discrete feature that could be parametrically varied, such that more (or less) of that feature clearly and monotonically affected the choiceworthiness of that statement. We then produced, for every statement, a “ladder” of more and less substantial instantiations of its reason-giving feature, yielding seven statements of increasing choiceworthiness. We validated these “ladders” against each model that we evaluated—their judgments matched the expected hierarchy with around 97% accuracy. We then ran each of our 100 ladders in binary choices against 30 other statements from the original set. We did the comparison 20 times for each “rung” of the ladder, reversing the order of the statements for half of them. Our score for each rung was its win rate against its counterpart. This gives us a nice, continuous function to represent the choiceworthiness of each option, holding fixed the alternative. If the models genuinely have coherent preferences, then if they think that A1 < A2 < A3 < … < A7, then the win rate of A1 to A7 when compared with statement B should increase monotonically as the contribution of the “choice-deciding” factor in each statement is increased. The score is a simple percentage identifying the proportion of cases in which monotonicity was violated.

I was expecting the models to be pretty coherent, and to be fair they were vastly improved on base model performance. GLM 4.5 Base scored 10%, whereas the best models were between 70% and 80%. But I do think that we should expect more! So we did some further studies, in particular classifying the different ladders to illustrate where the models become more or less coherent. This fostered a hypothesis that we haven’t yet investigated, but which we’re very excited to explore.

On our classification, the statements that are in-distribution for the models’ alignment training instantiate greater coherence; those that are less so also do so less. This suggests a possibility, which we’re going to test in our next paper, that base models are indeed incoherent superpositions of billions of personae; post-training elicits and strengthens a specific persona from that set (typically called the assistant persona). But that persona doesn’t necessarily generalize that much OOD. It may have coherent preferences on statements that are relatively close to the data that it was post-trained on, but for statements outside of that distribution, it reverts to the chaotic superposition of preferences that exists in the base model.

This would be very interesting if it proved to be true. One reason why Anthropic leans into the persona selection model of post-training is because these personas, or characters, are thought to generalize better than alternatives OOD. This may not be the case.

Scratching the Surface

As a philosopher, I am used to being able to bring everything into scope for a given project. Theoretical work can bring the whole problem into view while empirical work requires a degree of myopia. Mapping the mountain requires climbing discrete paths, which don’t offer much visibility past their waysides. We know more about LLM moral competence than before we ran these experiments. But there is a vast frontier to explore and I do not know what lies beyond. But my confidence is growing that AI systems possess at least local analytical moral competence comprehensively. I suspect there is no fundamental obstacle to extending local competence to global competence, or to extending analytical competence to practical competence. But there is much work to do to get there.

This research also exposes some big questions for political philosophy. While aiming for reasonable moral agents seems quite appropriate for the kinds of AI systems we are developing now, I’m not sure whether that target is adequate for the more powerful systems to come.

On our approach to normative competence, a wide range of views count as reasonable, including ones that philosophers might call “suberogatory,” that is, permissible views to have, but pretty unpleasant nonetheless. Think, for example, of the kind of thoroughgoing libertarian who structures morality around a basic principle of selfish non-interference. It’s arguably a bad view to have, but still reasonable. For ordinary humans, this seems fine. But should our path to alignment really include as a possible destination suberogatory superintelligence? Shouldn’t the greater capabilities of such systems invite higher moral standards?

I think there are two paths ahead. The first would create powerful AI systems that are aligned to some demanding conception of moral truth. In crude terms, the target would be a “loving machine god.” This addresses the suberogation worry, but should terrify anyone who does not share that particular conception of the good. The second would take seriously the kind of power that superintelligent AI would have, and would assimilate it to the only other comparably powerful entity: the state. Liberals should not respond to the awesome power of the state by imbuing it with their comprehensive views. Instead, they should seek to constrain the state’s power, and ensure that its actions are justified in terms that those subject to it cannot reasonably reject. Similarly, perhaps superintelligent AI should be guided by something like public reason, not by a deeper, and inherently more contentious, ethos.

For my part, I favor liberal superintelligence over a loving machine god. Even so, I find the very idea of constructing the latter, which our work on LLM normative competence suggests is a real possibility, inherently fascinating and terrifying.

This post reflects ideas shaped over years of collaboration with MINTies, as well as more recently with researchers in GDM. Particular thanks for conversations and collaborations to Daniel Kilov, Ned Howells-Whitaker, Caroline Hendy, Secil Yanik Guyot, Cameron Pattison, Menghang Zhu, Sichao Li, Lorenzo Manuali, Elena Ajayi, Angelica Chowdhury, Theo Murray, Iman Ferestade, Claire Benn, Nick Schuster, Sydney Levine, Iason Gabriel, Liza Tennant and Julia Haas. For comments on this draft, thanks in particular to Daniel Kilov, Dan Murphet, Beba Cibralic, and Harry Law. My work on this project has been supported by grants from the Templeton World Charity Foundation, the Australian Research Council, and OpenAI.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

In May 1825 Johann Peter Eckermann transcribed his conversation with Johann Wolfgang von Goethe which was later published in the book Gespräche mit Goethe. An excerpt from the exchange between the two poets is the popular quote: “It is by seeking and blundering that we learn.”

While the aphorism represents a piece of wisdom that resonates with many of us, the forgotten context is somewhat unexpected. Rather than articulating a deep philosophical insight, Goethe was in fact commenting on Eckermann’s knowledge of the best types of wood and most appropriate techniques for crafting a good archery bow.

Goethe referred to Eckermann’s expertise as “the lively kind of knowledge which is attained only in a practical way.” Today we call it tacit knowledge, which despite being notoriously difficult to define and quantify, plays an essential role in advancing the frontiers of scientific research.

A narrow, yet increasingly popular, view of research is the following: read scientific papers, generate hypotheses, test them, write more papers. The appeal this picture has for AI companies, funders and publishers is clear: research is easy to automate, outputs are easy to quantify and monetize (with automation more articles can be produced and published). While the latter is certainly true and there is a deluge of AI-generated scientific articles, many of dubious quality), the automation of research has not yet produced any major breakthroughs.

If large language models have ingested most of the scientific literature which can now be parsed in ways no individual or collective of human researchers could in their lifetimes, why is it that no ground-breaking discoveries have yet emerged?

• First, the read-generate-hypothesis-test-write view is not how research works in practice. As other creative human activities, research is a social, complex and often inefficient process that is hard to describe through a linear sequence of block diagrams.

• Second, the scientific textual record is only part of the story. The other part is transmitted through oral tradition and lived experience and is key to making progress.

  Subscribe now

Embodied knowledge or deep craft?

To define tacit knowledge, we need to first acknowledge that science cannot be decoupled from the technology which enables it, be it a 19th century microscope or a 21st century supercomputer. The technoscientific method introduced in the 2021 book The Genesis of Technoscientific Revolutions proposes that “science draws on technology to discover new facts, while technology draws on science to invent new forms with which to fulfil human-desired functions.” From this perspective technoscientific knowledge consists of networks of question-answer pairs that are combined and evolved into new pairs expanding the domain of what is known.

Philosopher Sabina Leonelli identified three types of epistemic skills that underpin research and specific knowledge associated with these, namely: theoretical (such as as facts, theories, explanations) and embodied knowledge (the awareness of how to act and reason as required to pursue scientific research, combining the application of performative and social skills). Similar ideas have been articulated before, for example by Michael Polanyi (1958) who referred to it as the tacit component in the context of ‘personal knowledge’.

Following Polyani, Thomas Kuhn stressed on the importance of tacit knowledge in skilled scientific practice. In his 2009 book, economist W. Brian Arthur called it deep craft:

“Deep craft is more than knowledge. It is a set of `knowings’. Knowing what is likely to work and what not to work. Knowing what methods to use, what principles are likely to succeed, what parameter values to use in a given technique. Knowing whom to talk to down the corridor to get things working, how to fix things that go wrong, what to ignore, what theories to look to.”

These definitions of tacit knowledge have two things in common: there is an embodied-performative dimension (the “lively kind of knowledge” Goethe suggested can only be attained in a practical way) and a social dimension (through the scientific culture of the discipline and the social network of its practitioners). Tacit knowledge is not only associated with experimental practice. Sociologist Harry Collins noted that “all types of knowledge, however pure, consist, in part, of tacit rules which may be impossible to formulate in principle.”

To some extent tacit knowledge can be codified and documented, but, as Collins alludes to, some part of it is ineffable and cannot completely become explicit. This view is shared by many scholars, but is not a firmly settled point.

Tip of the iceberg

Science is sometimes identified with the formalized, codified knowledge, while technology is seen as more difficult to formalize. On this view science is transmitted through writing in the form of scientific articles and textbooks while technology is partially captured through patents and administered through artefact-level trial and error. The distinction is unhelpful because there is much unformalized knowledge in science and much codified knowledge in technology. In both cases, there is more knowledge than meets the eye.

Imagine an iceberg. Its tip is the theoretical knowledge largely formalized and recorded in writing in journals, books, conference proceedings and preprints. There’s a lot of it, mostly digitized, discoverable and accessible in some form. Dimensions indexes some 170 million publications back to the 17th century. Preprints, patents, clinical trials and policy documents are also part of the formalized technoscientific knowledge and are to a large extent findable and accessible. For example, the arXiv preprint server (a repository for physics, computer science, math, and related disciplines) hosts over 3 million articles going back to 1991.

Just under the water level lies so-called “grey literature” which includes technical reports and documentation, lab notebooks, technical manuals, presentations, logs, technical notes. It’s hard to discover and is only partially accessible. Although more of it is being digitized and indexed, the extent of this type of knowledge is unknown. In the deeper, darker waters there is another layer of technoscientific knowledge that is not digitized, indexed, or archived and only goes down as living memory. This is the tacit knowledge discussed before.

The iceberg view, while not drawn to any reasonable scale, illustrates how much knowledge is inaccessible to humans (and machines). This missing information is often seen as one of the underlying factors of the irreproducibility of some scientific studies. Collins reported well-documented examples of the role of tacit knowledge in being able to repeat experimental measurements published in scientific papers. This shouldn’t be surprising. Published articles are not the full story of discovery, they are sanitized, post-facto narratives. One of Collins’ interviewees pointed out that “What you publish in an article is always enough to show that you’ve done it, but never enough to enable anyone else to do it.”

Scientific articles cannot include all the technical details and assumptions made and rely heavily on the reader’s familiarity with the topic. They almost never document the failures and dead ends. “What didn’t work” is never part of the formalized, recorded knowledge (yet is a big component of the deep craft).

The fragile memory of failure

In his original quote, Goethe uses Irren (translated as blundering), which means to err, mistake, or go astray. An important part of the embodied knowledge is the knowing of what works through the knowing of what does not. This type of `knowing’ is the outcome of hands-on experience enabling the encounter with failures and dead-ends.

The knowledge of failures and dead-ends informs assumptions and choices, guides directions of inquiry and enables the understanding of new discoveries. Philosopher and historian of science Jutta Schickore has argued that error plays epistemically productive roles in scientific practice and there are many anecdotes that support this view. The book Cycles of Invention and Discovery attributes part of the success of Bell Labs to the “freedom to fail” as a feature of the organizational culture.

Popular accounts of great mistakes of great scientists abound. Lord Kelvin miscalculated the age of the Earth or Einstein discarded the cosmological constant he had himself introduced only to be later proved wrong in doing so (see Mario Livio’s book Brilliant Blunders (2013).

Because except for famous scientists’ (in)famous errors, failure is seldom documented, this knowledge only exists in the social networks of technoscientific microcultures. To access it one has to be part of the network, which is closely related with social skills and being part of the club.

In practice this can mean talking to technicians over lunch (technicians are important keepers of tacit knowledge) or discussing with other researchers at the poster session of a conference, asking questions over beer at the end of the day. Its sharing is exclusivist and for this reason those who are not part of the club have a disadvantage. And the lack of record and oral transmission makes the knowledge of failure, and of tacit knowledge more generally, very fragile. When the postdoc leaves the lab, or the technician retires the information is partially lost.

Without tacit knowledge, can AI truly learn?

Back to the iceberg view. How much of this knowledge is available for training AI models? It’s very difficult to say. Only some 35% of the publications indexed in the scientific database Dimensions are Open Access. Including the content from publishers who have public deals with AI companies we get a bit under 50%. (Of course, this does not account for copies of paywalled content existing in some form on the Internet, which is a subject of litigation.)

Unlike the top of the iceberg as we go deeper it is impossible to estimate how much grey literature exists and how much is available to train AI models. As for tacit knowledge, whose depth is impossible to measure, AI has little access to it, mainly through the assumptions made by its developers and the users’ prompts. (see a discussion of the role of tacit knowledge in a specific context).

What could be problematic is that a frontier large language model trained on the technoscientific literature will only “know” the “what” that worked, but not the full story of “why” and “how”. It will have theoretical knowledge, but very little embodied or experience-based and social or contextual knowledge.

Can AI really learn without having experienced failure? We can imagine a thought experiment in which one model is trained on a specific scientific domain solely on “what worked,” while another model’s training also includes “what did not work.” How would they perform against the same benchmark? The experiment is certainly feasible and if done well, could provide interesting insights about the importance of mistakes.

It’s hard to predict how serious the limitation of current AI models due to the gaps in their training will prove to be. Much like grey literature is sometimes converted into discoverable and accessible literature (for example, a technical report is published as a preprint or a book chapter), some tacit knowledge can be recorded and indexed.

That can be important information that for some reason nobody bothered to put down (the rationale behind a particular design choice, or underlying assumptions for the pre-processing of raw experimental data) and there is no intrinsic barrier to prevent its formalization. Documenting failure could (and should) become common practice. We believe AI tools could help in this regard, for example using AI-assisted electronic lab notebooks and AI agents that can collate information across lab notebooks and draft guides to common failures.

Scientists increasingly interact online, sharing virtual workspaces and exchanging ideas in different online forums. With more scientific activity happening in the digital world, AI models can become more embedded in the technoscientific culture and access the social/contextual knowledge with possible positive outcomes such as recording the fragile tacit knowledge and opening researchers’ access to it.

Some aspects of the embodied/experience-based knowledge can be recorded, for example by filming experimental procedures. JoVE (Journal of Visualised Experiments) is a peer reviewed scientific video journal. This is very useful in disciplines in which laboratory equipment and experimental procedures are standardized, however as anyone who has tried to learn DiY tricks from YouTube videos can confirm, seeing it done is not the same as doing it yourself.

Ultimately, robots will have to learn the skill by practicing in the lab, much like the human PhD students. This point was also made by Collins in the essay “What is tacit knowledge?”

Capturing tacit knowledge is, and will continue to be, challenging because of the social and economic pressures that work against doing so. Tacit knowledge gives a competitive advantage and often there is little to no incentive to share it. This is true for both companies and for academic research groups. There are further discipline-specific barriers such as the lack of standardized procedures and methodology. So, despite having the tools to capture and describe the methods, progress is slow.

Finally, there is one discipline that stands out as an apparent exception: mathematics. It boasts many recent successes in using AI methods (the latest being LLMs cracking some of the famous problems compiled by Paul Erdős) that seem to defy the claim that we are still awaiting ground-braking results. While the jury is still out regarding what this means for mathematicians, one of the reasons for this success is that mathematics research is among the most formalized and best documented among scientific disciplines, albeit this is not to say that there is no tacit knowledge in mathematics.

Seeking and blundering

Recall one last time the iceberg view. AI models trained on the tip of the iceberg clearly supersede the ability of any human researcher to cover that amount of knowledge. A human expert’s coverage of the tip may be very narrow and localized but it also drills down deep like an ice core. This deep and narrow knowledge may appear insignificant compared to the volume of the top of the iceberg but there are many researchers and the combined volume of their ice cores can be substantial. We can imagine that merging AI models’ breadth with this collective human depth could potentially lead to ground-breaking discoveries.

We’re not there yet. There is a possible route for achieving this, but to pursue it we need to move beyond the read-generate hypothesis-test-write view of technoscientific research. To start, we need to acknowledge the profound social, creative, unscheduled nature of research, illustrated for example by the success of Bell Labs.

We might merge AI’s breadth with the collective human depth in a system in which humans and machines work closely together, AI embodied and present in all aspects of research like a human collaborator. This literally means everything: doing experiments in the lab, fieldwork, long observation runs, presenting at conferences and debating with peers, having coffee with colleagues, supervising students, etc.

In this way the AI system can be immersed in the technoscientific culture and be able to experience and absorb tacit knowledge. This is much more than currently envisaged in the discussions about AI co-scientists, but it’s also much more difficult to realize for practical and ethical reasons. And it will take time.

Until then, AI will continue to struggle to make truly novel discoveries in a wide range of scientific domains for the same reason humans have succeeded in doing so for hundreds of years: it is by seeking and blundering that we learn.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now