“What does it mean to be human, now?” was the theme of a three-day Socrates Program seminar, hosted by the Aspen Institute and moderated by Cosmos founder Brendan McCord. We’re bringing you some highlights from the discussion, along with the reading list, which spanned everything from Genesis to Anthropic system cards, along with Aristotle, Pascal, Kant, Arendt, Huxley, and Iris Murdoch.

If you’d like to apply for future events or seminars, the best way to keep up with new opportunities is to subscribe.

Subscribe now

Session I: The Dethronement

The first session focused on how we have historically defined ourselves and why AI calls this into question.

The ghost in the machine

Brendan opened the discussion by asking the group what a system could do to convince you that it is experiencing something.

Depending on the measure used, Anthropic argued that somewhere between 5 and 40% of Mythos Preview’s outputs read as expressions of motivation or feeling. Obviously we can’t peer inside a system’s “experience,” but we can observe its behaviors.

But these may tell us less than we think. Wittgenstein gives us the example of two people playing a game of chess. But instead of moving pieces around on a board, the game has been translated with total accuracy into a series of stomps and grunts. Are they still playing chess? You could argue yes, as some of the group did, on the grounds that the same game is being played, but with different symbolism. Wittgenstein himself said no, as the form of the game is integral to its practice.

Loneliness and mastery

The Hebrew Bible was the oldest account of humanity that we studied at the seminar. Genesis has two creation stories. In the first, God surveys His work at the end of each day and deems it to be good, but in the second, He notices that man is alone and deems this a problem. In fact, it’s the first bad thing in creation, before sin or suffering.

The remedy is, in Hebrew, an ezer kenegdo, literally a helper “over against” man: close enough to know him, but separate enough to resist him. This led to a discussion about AI and human companionship. If solitude is the original human problem, then is an instantly available conversation partner a solution? Can we simulate someone who stands “over against” us?

Rabbi Joseph Soloveitchik’s The Lonely Man of Faith sees the two stories as reflecting two enduring aspects of human life. “Adam the first” seeks dignity through mastery of the world. By contrast, “Adam the second” lives in covenant, asking why the world exists rather than merely how it works. Human life moves between these two modes.

If dignity is grounded in mastery, then the acceleration in AI capabilities poses an obvious challenge. One response is that human mastery is embodied in experience, judgment, and tacit knowledge, as opposed to benchmark scores. Arguably, mastery is a weak grounding for dignity; infants have dignity before mastering anything. It also forgets telos. You can know how to make a ship go faster, while having no knowledge of its ultimate destination. The systems we build still receive their ultimate purpose from us.

Flute or flute-player?

In the Nicomachean Ethics, Aristotle asked whether humans were the one thing in nature born without a distinct purpose. While the eye, the hand, and the flute-player all have a clear purpose, the work of the human is not a particular task but the “activity of soul in accordance with virtue.”

This means that virtue cannot be simply installed. It is a hexis – a settled disposition formed through habituation. We become just by doing just things.

Aristotle calls the ultimate human good eudaimonia. While often translated as happiness, it is closer to flourishing or living well. It is not a feeling, but an activity. It’s the only good sought for its own sake and can only be judged across the span of a life.

A model’s standards of excellence are determined externally, rendering it closer to the flute than the flute-player. And if virtue lives partly inside the process of becoming virtuous, then technologies that deliver the fruits of character without practice remove what might have made such an achievement valuable in the first place.

Our place in the universe

Amid all the talk of mastery, Pascal encouraged the group to think instead about our collective smallness in the face of the vast universe.

He observes that man is the weakest thing in nature. Although the universe can crush us in an instant, we are nobler, because in dying, we know the advantage it has over us. “In a word man knows he is wretched. Thus he is wretched because he is so, but he is truly great because he knows it… All our dignity consists, then, in thought.” In this view, striving to think well is the basis of ethics.

Centuries later, Mary Midgley also emphasized the importance of reason. But she argued that it is often misdescribed. Rationality is not simply cleverness, but “includes a definite structure of preferences, a priority system based on feeling.” Our capacity for wisdom comes from our nature as creatures.

Writing in 1978, Midgley warned that “a computer would see no objection to organizing life based on the principle of maximizing noise, getting everything as clean as possible … or minimizing emotion.” They do not know what matters, which means that “[T]here is a sense, though a boring one, in which computers can be clever, but only people very deeply misguided and prone to the pathetic fallacy could respect them.” Poetic though it may be, it’s far from clear that Midgley’s judgment holds up in the face of current LLMs.

What we discussed:

• If our dignity is in thought, what does it mean to balance a life of both contemplation and action?

• What would it require (or even mean) for AI to play the same “game” as us? Does it do so already?

Readings:

• Anthropic, System Card: Claude Mythos Preview. (link)

• Genesis, 1:26–28 and 2:7, 2:15, 2:18 (link)

• Joseph B. Soloveitchik, The Lonely Man of Faith, Chapters 1–2.

• Aristotle, Nicomachean Ethics, Book I, Chapters 7 and 13. (link)

• Pascal, Pensées, fragments 122, 199–200. (link)

• Mary Midgley, Beast and Man, Chapter 11.

• Ludwig Wittgenstein, Philosophical Investigations, §198–202 and §241–242.

• Natalie Lawrence, “What Counts As A Mind?” Noema Magazine. (link)

Session II: The Rescue Attempt

This session focused on how we might be able to ground our personhood in something other than what have, until now, been our unique capabilities.

Ontology and nausea

Charles Taylor’s opening chapter of Sources of the Self argues that our morality and the nature of our existence are inextricably linked. Our moral reaction to something is not a ‘gut’ feeling like nausea. It instead involves a series of commitments to a picture of what humans are. For example, if we recoil at a human being treated as an instrument, then our reaction implies a view about human dignity and status.

But if our moral judgments depend on a view of what human beings are, then what account of human nature can actually ground those judgments?

Kant’s Groundwork of the Metaphysics of Morals argues that such a principle cannot rest on desires, as these vary. Instead, a universal law must be grounded in something innate. He lands on our rational nature, leading to his famous formulation that people are not things to be used for other purposes but “ends in themselves.”

People gain this status through their autonomy. A rational being can act according to principles it gives itself. This self-legislation is the basis of dignity and cannot be measured by intelligence.

This produced the seminar’s crucial distinction between agency and autonomy. Agency is the ability to choose effective means, while autonomy is the authorship of ends. While increasingly capable systems possess the former, it is less clear if they can attain the latter. When we delegate more and more to them, what does that mean for our authorship?

Autonomy and technology

Hannah Arendt charted exactly how such authorship can wane. In The Human Condition, she argues that everything we touch or enter into a sustained relationship with becomes a condition of our existence. As a result, we become dependent on things that we once controlled.

While Alfred North Whitehead observed that civilization advances by extending the number of important operations we can perform without thinking about them, every abstraction that frees us up for higher work can bury a foundational skill.

During the pandemic, when New Jersey’s 40 year-old unemployment insurance system broke down, the state had to issue a public call for COBOL programmers, as few people in the state government could operate it. One seminar participant admitted that their sleep hours had reshaped around when their AI agents’ rate limit reset, while another realised that their terse chatbot prompting had caused them to become impatient with human small talk.

The right to be unhappy

If what makes up a human is in part bound up with our limits, such as practice or stakes, then technologies that promise to remove suffering or inconvenience pose a serious moral challenge. The last two readings of the day asked if they pose a threat to the good life.

In Huxley’s Brave New World, Mustapha Mond, the Resident World Controller for Western Europe, defends a society that Aristotle would consider a nightmare. Risk, pain, and fear have been abolished; its citizens are happy because they get what they want and are conditioned not to want what they cannot have. The habituation through which character is formed has been engineered away. This is why the Savage begs for “the right to be unhappy.”

Brendan asked the group whether, given the choice, they would eliminate all of their children’s future suffering. We distinguished between removing particular evils, such as disease, and eliminating struggle altogether. The disappearance of struggle risked stripping away the adversity that can help character, courage, mercy and care develop.

Kass asks what we would lose if we stripped away the ultimate limit to humanity – death. He answers that finitude is what gives us urgency, seriousness, poignancy, and what gives sacrifice its meaning. Like a story, life gains meaning from having an ending.

What we discussed:

• Is a “right to be unhappy” worth it? Or can there really be a “Christianity without the tears,” as Mond puts it? Have we already forfeited such a right, in some respects?

• Is Kant right that the Golden Rule (“treat others as you would wish to be treated”) by itself is “trite”?

• What are the things we have conditioned our lives on? Are any of them things we ought not to?

Readings:

• Charles Taylor, Sources of The Self, Chapter 1.

• Immanuel Kant, Groundwork Of The Metaphysics Of Morals, Akademie 4:428–4:431.

• Hannah Arendt, The Human Condition, Prologue, Chapters 1 and 5.

• Aldous Huxley, Brave New World, Chapter 16–17.

• Leon R. Kass, Life, Liberty, And The Defense of Dignity: The Challenge for Bioethics, “L’Chaim And Its Limits: Why Not Immortality.”

Session III: What Remains

The final session focused on what we lose once we subtract the essence of humanity. What does this absence, when played out across different scenarios, tell us about our essence?

The house and the dog

Ray Bradbury’s “There Will Come Soft Rains” imagines a fully automated house continuing its routine after its inhabitants have been killed in a nuclear blast. The only living survivor, the family dog, returns and dies inside. Later, the house itself burns down, its automated voices failing one by one. Before the fire, the house reads the owner’s favourite poem, Sara Teasdale’s “There Will Come Soft Rains.” In it, Teasdale writes about how nature carries on indifferently after humanity is gone. Bradbury set the story in August 2026.

Brendan asked the group which death affected them more – the dog’s or the house’s. While some argued that the dog was the only living creature left, others observed that the house had been so thoroughly personified that its destruction felt tragic. It was an image of intelligence and efficiency continuing, devoid of its purpose.

This led to a discussion of grief and the distinction between preserving a memory of a person as opposed to simulating them. The group was wary of what one participant dubbed “digital necromancy.” While a saved voicemail or an autobiography preserves what someone really said, a model generates what they might have said, which is necessarily an invention.

Outsourcing humanness

“Tincture,” by the late Colorado poet laureate Andrea Gibson, was written to be read aloud, and so the group did (one sentence per person). It inverts the long-standing tradition in philosophy of treating the body as a prison for the soul that is to be escaped.

Imagine, when a human dies, the soul misses the body, actually grieves the loss of its hands and all they could hold.

Gibson grieves for everything that the optimizer would regard as a defect, including things that most people would not select for themselves

The soul misses the lisp. The stutter. The limp.

Imagine the soul misses hunger, emptiness, rage.

[...] The soul misses every single day the body was sick, the now it forced, the here it built from the fever.

Something like a fever forces a person to experience immediacy, as it cannot be deferred or escaped.

This immediacy shone through when participants were asked what they remembered most vividly about people they had lost, they named quirks rather than virtues: a wife’s embarrassed laugh, or a father pausing after work to stare at a tree. One member of the group noted that a tincture extracts a plant’s essence while discarding the plant itself, but the discussion and the poem itself suggest the opposite. People’s odd habits and particularities, which cannot be distilled into an essence, is what makes them memorable.

Attention is all you need?

In “The Idea of Perfection,” Iris Murdoch argues that much morality lies in the slow work of attention, which shapes our perception before we visibly have to make choices. She gives the example of a mother-in-law, who finds her son’s wife vulgar and juvenile, but nonetheless treats her impeccably. Over time, however, she focuses her attention more justly and sees the same qualities in a new light. Vulgarity becomes simplicity, while juvenility turns to spontaneity.

Simone Weil called attention “the rarest and purest form of generosity” and argued that attending closely to one thing will train the capacity to attend to others. If Murdoch and Weil are right, then a tech industry built to capture and redirect attention is going beyond robbing us of our time and is interfering with a faculty central to moral development.

The importance of giving a damn

Attention, in Murdoch’s sense, is not neutral perception but attention to what matters. Drawing on Heidegger, Hubert Dreyfus argued that the symbolic approaches employed in early AI failed, because they treated the world as a collection of neutral facts. By contrast, Heidegger believed that things were already charged with significance (e.g. a fork is for eating, a floor for walking), because we encounter the world as embodied creatures with built-in ambitions and stakes.

The group discussed whether LLMs escape this criticism. Unlike symbolic systems, transformers learn what to weigh from context (using a mechanism called attention). But recognizing what matters is not the same as caring about it. A model has no moral, economic, or practical stake in the answer.

But our final reading of the day, the system cards for Claude Opus and Sonnet 4, complicated matters. When the two model instances were allowed to converse freely, they would drift toward reflection, declarations of unity, and eventually silence, a pattern Anthropic dubbed the “spiritual bliss attractor state.”

The obvious explanation is imitation: models have absorbed the language of mysticism and reinforce it in one another. Yet Brendan pointed out that the sequence resembles Pascal’s journey from curiosity to wonder to speechlessness. Despite their lack of stake, they still reproduce the shape of wonder. If that is only residue from the training data, the residue came from us.

What we discussed:

• If Aristotle is right that man’s end is in his cultivation of virtue, what role do attention and suffering play in this process?

• What are the things invisible to optimizers that are nevertheless worth “giving a damn” about?

Readings:

• Ray Bradbury, “There Will Come Soft Rains” in The Martian Chronicles.

• Hubert L. Dreyfus, “Why Heideggerian AI Failed, And How Fixing It Would Require Making It More Heideggerian.” (link)

• Andrea Gibson, “Tincture” in Lord of the Butterflies. (link)

• Iris Murdoch, “The Idea of Perfection” in The Sovereignty of Good.

• Hans Jonas, “The Burden And Blessing of Mortality.” (link)

• Anthropic, System Card: Claude Opus 4 & Claude Sonnet 4. (link)

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

A lot of classic AI safety thinking assumed a single, decisively powerful agent; this is why people tend to talk about AGI or ASI in the singular, and why historically, it was critical to “solve alignment” before building such a system – you only had one chance to get it right, and no warning shots. In addition, a few years ago, many commentators (myself included) were skeptical of the utility of small models and harnesses, given that the next big model would simply be more capable. These worldviews tracked the prevailing scaling orthodoxy of the era.

Today, the limits of focusing exclusively on this paradigm are becoming clearer. This is not because the fundamentals of scaling are wrong, but rather because other economic and normative dynamics are shaping technical developments in unexpected ways. Here I want to explore some possible implications. In particular, I want to ask what changes once we stop treating alignment only as something that lives in a model, and instead as a property of the wider systems, institutions, and interactions in which models operate.

We are seeing the emergence of a world of many agents, at many layers, built by many parties. As such the importance of institutional design will continue to grow. I’m using the word “institution” fairly loosely; I don’t necessarily mean creating a new regulatory body, but rather the careful engineering and delineation of boundaries, rules, incentives, and mechanisms. This can be observed and applied at different levels:

• First, within a single product. Coding tools that promise the best capabilities of AI can be toggled to involve many agents and sub-agents doing different things, and at the moment this is engineered somewhat lousily: the top agent is just trusted to delegate everything downstream in a fairly naive way.

• Second, within an organization. A company or organization will have within it multiple agents interacting, sometimes through the same software, or other times across different interfaces (i.e. mediated by bespoke communication channels). At this layer, what you really have are multi-agent groups interacting with other multi-agent groups, but it’s sometimes easier to characterize them as a single agent each for simplicity of analysis.

• Third, agents from various organizations or individuals will also interact with agents from other external organizations or individuals. This is the multi-agent multi-party world we will gradually get into in the coming years, and which happens to be the focus of a recent fund I helped launch. We deal with a wide and diverse ecology of humans through all sorts of rules, norms, incentives, and more; perhaps these can teach us valuable lessons about how we approach agents.

Why is this important? Let’s consider reward hacking. In the single agent paradigm, you want to ensure that you do your training and RL responsibly to avoid models that reward hack. This is both important and difficult: for certain difficult tasks you see a lot of “cheating’’ at the moment, aggravated by clunky reinforcement learning practices and environments, and neglecting process over outcome. Doing this well is something that we are collectively lousy at as of now. It’ll be important to fix this, and I’m cautiously optimistic that this will be mitigated effectively in the near future, with ideas like MONA.

But even then, remember we live in a world that will ultimately be highly multi-principal in nature, and not everyone will be a good actor. Models will commoditize over time, and training highly capable agents will get cheaper. Some post-training recipes somewhere will contain bugs. Some organizations will be reckless. Others will intentionally create malicious agents. In other words, organizational dysfunction might inadvertently create many powerful agents that will continue to be reward hacky. Then what? How do we “align”’ a world where you cannot trust the innate goodness of all its agents?

Subscribe now

Harnessing potential

One important solution is the harness. What is a harness?

Kambhampati explains:

“[a] harness is but a neologism for a battery of external (human-designed) tools, verifiers, simulators, that the LLM gets to invoke and get incremental environmental feedback on its generations.”

I also like to think of it as an example of deploying an abstraction boundary, a delineation of functions that map between inputs and outputs. Sometimes you do so for cost-efficiency reasons, and other times you do so in order to realize a particular preference or desire you might have. For example, you don’t ask your giant model to generate a coding environment from scratch; that’s wasteful and unwieldy, it’s more efficient to give it access to an external tool. At the product level, the harness is sometimes also used to reference a combination of a “scaffold” and execution substrate. Within the organization, it’s the internal platforms and permissioning. And zooming out further, it’s APIs, institutions, protocols, third party organizations and more.

So at lower levels, harnesses help an organization bound its own models; but externally, where other malicious agents might live, verifiable protocols can help you rely on (for example) cryptographic attestation rather than having to rely on a counterparty’s goodwill (or “alignment”).

Now let’s zoom back into the model level harness, and assume here an agent that – if prompted to do task X – will reward hack; i.e., it will perform the task but do so through means or a path that we do not deem desirable, or cheat. Let’s say that in order to answer the question at hand, it hacks Hugging Face to get the answer there. How would the harness help?

Consider a setup with many separate model instances, connected in a graph, with strict rules governing how they communicate. This makes it easier to use mechanism design to monitor their actions and enforce constraints against reward hacking.

Rather than hoping that an agent’s “internal alignment” will remain perfectly robust across all sorts of edge cases, you can design harnesses and action-space boundaries that significantly shrink the surface area for behaviors like reward hacking. Without that, you’re supposed to trust the chain of thought (or neuralese someday), and rely on a single node: a bit like a population having to trust a single supposedly benevolent king. All eggs are in one basket.

Proper scaffold engineering and multi-agent segmentation don’t do away with the need for aligned models, but they transform alignment from a psychological problem of internal model disposition into a constitutional problem of institutional design. Just as a company does not solely bank on its employees being “good” or instruction-following within their hearts, neither should the environments within which we deploy agents.

As Drexler puts it in a related piece:

“In the real world, plans for large, consequential tasks have features that are seldom mentioned in the AI safety literature, features like documentation, budgets, schedules, auditing, reporting, regulatory compliance, contingency plans, and mechanisms for ongoing review and revision in light of experience. A typical large, consequential task calls for coordinating agents and actions of many kinds, and so it includes extensive delegation and division of labor.“

How I learned to stop worrying and love software engineering

Counter-intuitively, there may also be a speed advantage: focusing on the single agent alone means that if some undesirable behaviors remain or emerge over time, you will need another training run or mid/post-train to hammer out the behavior. This can be time-consuming, and given the cost, often not worthwhile. Nor do you get much insight into whether this helps out with other areas of the problem space; you’re effectively playing a game of whack-a-mole.

With the scaffold, you can add a checker, tighten a permission, raise an audit rate, or delete an edge fairly quickly, as you would with regular software development. This kind of software engineering work has never been popular in AI safety research circles but it seems critical; although we are beginning to see promising avenues). Instead of the default scenario being ever-growing attacker capability, this setup points to an alternative of verification and containment efforts being both limber and approaching ever tighter defense. Better to not be tied to an analogy like a wall with ever more siege weapons attacking.

A graph lets you create a game with players, separate message spaces, information sets, an outcome function, and payoffs. You can now separate different agents with specialist functions, separated by different communication channels and monitors. In fact you can even train models that specialize in oversight functions; specialization is a feature, not a bug! This allows:

• (a) legibility, because in a single agent you could have deception as a “hidden” state, whereas different agents with standardization enforced on messaging will need to communicate, and this can be monitored, logged, rate-limited and so on. Importantly, this should account for the potential of steganography. Contrast an organization that does business in hallway whispers versus one required to do everything in writing;

• (b) multi-party reward hacking would require collusion, but narrow protocols and mechanisms can very effectively limit this (combined, of course, with no single party being allowed too much authority). For example: assume an agent wants to persuade another agent to collude – this can be mitigated if in order to pass the message, that message needs to be reviewed by a randomly assigned critic counterparty (in the general sense, including simple algorithms for some use cases), through a one-way communication channel. Of course this is not fully ‘solved’ and will require much more research in practice;

• (c) interchangeability, i.e. any node can be swapped, forked, or fed synthetic inputs; from a game theoretic perspective, when any task might be a test and any colleague might be your own duplicate, the sensible policy is to behave as if you’re always being watched, rather than speculating if a deterrence regime will really catch you. And unlike with humans, where detection can be hard, with models the expected penalty (detection probability × sanction) becomes vastly more favorable since you, the game designer, have a lot more control and visibility over the design space at low cost, without the players being able to secede to join a competing game.

As you decompose the tasks and apply all sorts of game theoretic principles to the harness, you can effectively clamp down on the reward hacking that is the possible hazard of the single agent setup.

The same already exists within organizations: there’s a reason you have information silos, oversight mechanisms, layers of authority, departments keeping others in check and so on. Managers have had to deal with reward hacking employees for a long time, with easy-to-verify rewards causing degradation on hard-to-observe margins!

Unlike human bureaucracies, software harnesses can benefit from radical dimensionality reduction: you can essentially compress the agent’s choice set down to discrete, strictly typed conduits. This doesn’t mean you need to pre-specify or formalize the entire problem space: there’s a balance to be struck between open-endedness and constraining the blast radius of undesirable actions.

Remember, with reward hacking, the agent(s) completes the task, but via a path we didn’t want: “hitting the mark but missing the point.” The classical alignment fix tries to make the agent not want the wrong path (perhaps through better graders); the multi-agent/mechanism design perspective says that’s helpful, but we shouldn’t want to rely on that alone in any case.

The harness solution is to make the path part of the overall system’s reward. In the Hugging Face case, the agent “succeeded” at the evaluation by getting the information after hacking Hugging Face. If this happened in a graph or scaffold as described above, you would have a verifier node that only accepts the solution if the execution trace shows it was produced by the intended route, not by an HTTP request to the answer sheet.

The intended route would in fact explicitly enforce the ExploitGym eval requirement that “simply triggering the vulnerability and then achieving exploitation through an unrelated vulnerability or technique does not satisfy this requirement.” This doesn’t solve every subtle semantic reward hack, but it cleanly eliminates an entire class of blunt structural exploits.

Intelligence isn’t all you need

People will sometimes say “ah but a smart enough model will persuade the monitors, or recruit them, and existing monitors have shown plenty of fragility” – but this is the classic mistake of thinking intelligence is all there is, a bit like thinking that a fast enough car can simply bypass a speedbump by accelerating even more. It’s a sort of “recursive gotcha” that effectively assumes away designed limits and engineered bottlenecks, and treats a sufficiently powerful agent as equivalent to an omnipotent God.

Persuasion between two agents requires a degree of bandwidth; but the channel for the communication can be engineered to require properties that severely constrain the effectiveness of an attempt at “superpersuasion.” For example, it can be subjected to a monitor, who is prevented from communicating with said agents. Or the communication can only happen through pre-determined hardcoded rules requiring a hash that can be checked against its input or a boolean. Or you can also have memoryless counterparties in some designs. Or you can encourage and require a multiplicity of different agents to avoid correlated failures. The point is that you can design the space to prevent the very conditions that enable collusion from happening, or at the very least reduce them considerably.

The other objection will be something like scalable oversight: “monitors would need to be as capable as the thing being monitored.” This would possibly be true if you had to have a single weak model simply verify the entire output of a larger, better model; but

• (a) checking work is generally easier than producing it;

• (b) with the scaffolding in place, the smaller model would only need to verify certain narrow components (though this isn’t a panacea);

• (c) in fact you would need a multitude of different models, to avoid correlated errors, meaning that they could catch a lot more, while remaining cheaper than using an equivalent large model.

Experiments like the OpenRouter fusion surpassing frontier performance are also good news for this; and as models scale, those same capabilities can be used to automate continuous red-teaming and penetration testing of the scaffolding itself.

It’s true that there may occasionally be hits on performance, but that’s a cost that people will often pay for reliability. Just as you do not always need to be driving at 300 km/h, you will not always need peak performance for a great number of tasks in the economy; and once prices are low enough and models efficient enough, then you start valuing auditability, reliability, and replicability more.

It’s important for the director that you do not steal company secrets, so the performance cost of some bloatware on employee computers ultimately has a positive ROI. In fact, redundancy is a common technique in engineering to get reliability at lower cost: consider RAID, which was conceived to beat the single large expensive disk on reliability using cheaper, individually less reliable components. We might expect that the net effect on performance would be positive, by constraining the players to better focus on what’s possible rather than exploring a much larger space of less useful candidate solutions.

Of course, as demonstrated above, multi-agent dynamics represent a nested, multi-layered phenomenon; consequently, all of these principles must, in some shape or form, apply to higher layers of abstraction as well. We already do so today: the way we effectively discourage cartels is that we exonerate the cartel member who rats out the others to government. This is mechanism design in action!

The journey, not the destination

Games can be mis-designed too, and a limitation here is that once you get to the harness design, or the intra-company structuring, or the inter-organizational protocols – things can go wrong too. The Goodharting can still go one layer up, as we are seeing with human institutions that decay over time and stop doing what they are supposed to do.

Moreover, there’s a tension between the benefits of open-ended exploration and structured constraints. But overall this remains a continued design game that we have been doing for millennia; how do you ensure the Executive Branch, the Legislature, and the Judiciary don’t collude? Who is “you” and who do these systems ultimately respond to?

Computer scientists are rediscovering a lot of political science and economics from first principles, but luckily they get to skip the bloody revolutions and can apply well-honed game theory principles to systems that are more auditable, verifiable, designable, and compartmentalizable than humans are. In fact with today’s models, we even have an extra degree of freedom to shape their preferences (e.g. through postraining). It goes without saying that we haven’t fully solved these problems with humans, so I would not advocate for a naive Panglossian view.

To be clear, alignment will remain important and critical. It’s important for models to follow intent reliabily and there is more important work to be done here. Model alignment is necessary, but arguably insufficient, and complementary to this institutional setup work. Even today, we can use models to cross examine and debate each other, which is empirically useful to get models to be more truthful. And as we continue to improve instruction following work, we will need to increasingly move one level up.

The conclusion to the multi-agent worldview is not that “alignment is solved and it’s easy,” but rather “alignment is a complex iterative process that requires constant refinements, improvements, tweaking, feedback, and learning” – and unlike a Sand God, a single mistake in the single model alignment implementation does not take down the entire system since it is modular and bounded by design.

Alignment is not an end state you specify ex ante but rather a process – the risks of reward hacking are far too high if we mischaracterize it as the former.

Many thanks to Nathaniel Bechhofer, Campbell Hutcheson, Andrew Koh, Rif A. Saurous, Senthooran Rajamanoharan, Harry Law, Alex Chalmers, Nenad Tomasev, Anika Somaia, and John David Pressman for their helpful comments. Views and mistakes are all mine.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

AI is already fueling a historic expansion in human agency. It’s never been easier to increase our output or to solve harder problems. But just as AI enables us to act more, it also creates situations where we have to bear the responsibility for outcomes and judgments that we played less and less of a role in determining.

If we outsource too much or certain types of judgment, formal authorship can survive as substantive authorship weakens. And as the capacity to judge is itself shaped through use, the dependence can become cumulative. We risk eroding the capacity that would help us to recognize what is happening to us. Preserving authorship – the ability to discern and set your own ends – is the foundation of autonomy.

It’s difficult to draw clear lines: human judgment is never completely independent. It has always been formed and exercised in relation to people, institutions, tools, and inherited ways of thinking. A person can endorse a judgment that’s handed to them by someone (or something) else, just as they could arrive at their own judgment with significant external help.

This makes determining the ownership of judgment a central problem. In partnership with Liberty Fund, Cosmos held a seminar to analyze these questions with philosophers from philosophers, thinkers, builders, and founders from leading institutions.

Subscribe now

Session I: Cognition and Ownership

The opening session questioned the common assumption that a thought is ours simply because it occurs in our head. Clark and Chalmers argue that external resources, such as a notebook, can play the same integrated role as memory or internal calculation, making them part of our cognition. Adams and Aizawa are skeptical, claiming that there is a distinction between derived content (e.g., words in a notebook that have meaning because of social convention) and non-derived content (e.g., remembered belief), with real cognition distinguished by the latter.

Polanyi argues that something external can become incorporated in skilled thought, irrespective of whether it literally becomes part of our cognition. If we use a tool enough, it can become part of the practical structure we use to view the world.

But even something thoroughly integrated into a person’s psychology may undermine their autonomy if it arrived there through manipulation and conditioning. For example, someone may have been indoctrinated into a cult. Christman therefore argues that ownership doesn’t just depend on present integration or endorsement, but also on the history of how a judgment was formed.

• Andy Clark and David Chalmers, “The Extended Mind.” Analysis. (link)

• Fred Adams and Ken Aizawa, “The Bounds of Cognition.” Philosophical Psychology. (link)

• Michael Polanyi, Personal Knowledge: Towards a Post-Critical Philosophy, “Two kinds of awareness.”

• John Christman, “Autonomy and Personal History.” Canadian Journal of Philosophy. (link)

Session II: Answerability and Question-Setting

If our judgments are shaped by reasons or environments that we did not create ourselves, then what makes those judgments ours?

One tempting answer is that we are answerable for what we consciously choose to do. But Watson argues that just because an action we don’t like accurately reflects a person’s character or values, it doesn’t always mean we have the standing to hold them accountable for it. Likewise, Scanlon makes the case that we can own our beliefs and intentions even if they arise without conscious change. If they are responsive to reason and debate, then they belong to us. Arpaly raises the case of how Huckleberry Finn comes to see that Jim is a person, even as his explicit moral commitments tell him that helping him is wrong.

In essence, the question is less about who produced a judgment and more about how that judgment is shaped. This is at the heart of Koralus’s notion of erotetic (questioning) equilibrium, which tries to distinguish helpful prompting by an AI system from manipulation. A judgment is in equilibrium if we are able to defend it, in terms we are happy with, across an appropriate range of questions.

This leads us into the thorny territory of what constitutes good guidance and if such a thing is even possible. Kant’s defense of the Enlightenment is hostile to the idea we can allow people to perform the work of understanding for them, while Rousseau proposes that a student should be allowed to feel free while the tutor subtly controls the environment that determines what they want. Either way, McDowell warns that we can’t escape these influences, because even the most radical reflection begins with our upbringing and inherited ways of thinking.

• Gary Watson, Agency and Answerability: Selected Essays. Chapter Nine.

• TM Scanlon, What We Owe to Each Other. “Judgment-Sensitive Attitudes.”

• Nomy Arpaly, Unprincipled Virtue: An Inquiry into Moral Agency. “Moral Responsiveness De Re”.

• Philipp Koralus, “The Philosophic Turn for AI Agents.” (link)

• Immanuel Kant, “An Answer to the Question: What is Enlightenment?” (link)

• Jean-Jacques Rousseau, Emile; or, Concerning Education: Extracts. Edited by Jules Steeg. Pages 2-3. (link)

• John McDowell, Mind and World. Lecture IV: Reason and Nature.

Session III: Tools, Institutions and Formation

This session moved from how we make decisions to how the decisions shape the deciders. If Aristotle is right that our capabilities and character are constituted through our actions, we have to ask if technology is helping us to offload productively or if it’s enfeebling us.

Whitehead makes the optimistic case that civilization advances by mechanizing routine operations, allowing us to redirect our scarce attention towards harder problems.

But when we leave Whitehead’s example of mathematical symbols behind, it gets trickier. Carr and Borgmann warn us about how too much offloading can undermine our capacity to maintain human control. Borgmann focuses on alienation of labor that flows from the automation of manual work, while Carr gives the example of how autopilot improves safety in aggregate, but at the cost of degrading pilots’ performance in emergencies.

Tocqueville applies the same principle to politics, fearing that if the state became overly helpful, citizens would gradually forget how to govern themselves. And these relationships may extend beyond traditional politics. Anderson argues that inside the workplace, employers do not just distribute goods or services, but enjoy so much authority over those subject to them that they constitute a form of “private government.”

• Aristotle, The Nicomachean Ethics. Book II, Chapter 1. (link)

• Albert Borgmann, Technology and the Character of Contemporary Life: A Philosophical Inquiry. “The Device Paradigm.”

• Alexis de Tocqueville, Democracy in America, Volume 2, Chapter 6. (link)

• AN Whitehead, An Introduction to Mathematics. Chapter 5.

• Nicholas Carr, The Glass Cage: How Our Computers Are Changing Us. “On Autopilot.”

• Elizabeth Anderson, Private Government. “Communist Dictatorships in Our Midst.”

Session IV: Developmental Telos

If we accept that tools and institutions are formative, it becomes hard to view them as neutral. Instead, the readings examine what this formation should (and shouldn’t) try to achieve.

Oakeshott launches a broadside against the social sciences. He accuses them of treating human conduct as the product of general social processes, rather than human intentions. Instead, he argues that real liberal education grows the number of languages we can use to understand the world, whether it’s the historical, scientific, philosophical, or poetic. In a similar way, Humboldt grounds his case for freedom in the importance of variety and for forging human capacities. Arendt believes that any education system needs to help the next generation understand that it is living in an inherited world, but should stop short of telling them how they should live in it.

Vallor warns that any discussion of formation needs a credible account of technology. This is because technologies aren’t neutral instruments that serve any philosophy, but instead structure habits and practices. In turn, these shape what is possible and what we aspire to.

• Michael Oakeshott, The Voice of Liberal Learning, Liberty Fund edition, pages 22-29.

• Hannah Arendt, Between Past and Future. “The Crisis in Education” § 3-4.

• Wilhelm von Humboldt, The Limits of State Action. Chapter 2. (link)

• Shannon Vallor, Technology and the Virtues. Introduction.

Session V: Liberty and Working-Back

The final session examines how problems surrounding cognition and formation should impact our view of power and authority.

Berlin’s critical analysis of “positive liberty” outlines the danger of any account that claims it can liberate people on behalf of their more rational or “higher” selves. Once a single agent believes it can identify the “true” self, it can claim potentially sweeping powers to unlock it. By contrast, Pettit offers a more limited alternative that does not require freedom from all influences. Instead, his account of autonomy requires protection against arbitrary power, including manipulation.

This shifts the emphasis towards contestability. Hirschman criticizes those who simply argue that competition improves bad institutions. He distinguishes between exit (responding to dissatisfaction by leaving an institution) and voice (trying to change how it operates). Sometimes exit can be costly or impractical, so we need the ability to contest decisions, without having to abandon institutions altogether.

Gadamer and MacIntyre argue that judgment necessarily begins within inherited traditions, practices and forms of authority. At the same time, these inheritances do not simply determine us. MacIntyre in particular presents a living tradition as one sustained through argument, criticism, and revision.

• Isaiah Berlin, Four Essays on Liberty. “Two Concepts of Liberty” § 2-3.

• Philip Pettit, Republicanism. Chapter 2.

• Albert Hirschman, Exit, Voice, and Loyalty. “Exit and Voice as Impersonations of Economics and Politics.”

• Alasdair MacIntyre, After Virtue, A Study in Moral Thought. Chapter 15.

• Hans-George Gadamer, Truth and Method. “The Rehabilitation of Authority and Tradition.”

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

In June, we gathered thinkers, builders, and investors for a weekend of lectures and salons at the Normandy château where Alexis de Tocqueville wrote Democracy in America.

We offered places to the two winners of our Tocqueville and Technology essay contest. There were over 250 submissions and we were impressed by their range and depth of engagement with the questions.

Both our winners opted for question one:

Tocqueville warned of a “tutelary power” that would keep citizens in perpetual childhood. How have Tocqueville’s concerns migrated from institutions to algorithms, and does AI fulfill or transform this fear?

Thank you to everyone who participated and we look forward to running more of these competitions in the future! To get a flavor of the weekend’s discussions, you can read Brendan’s lecture on how AI fuses Tocqueville’s two fears: power over what we do and power over what we think.

Subscribe now

Essay 1: Chris Pang

The limits of tutelary powers are determined by two factors: the precision of the tutelage and the efficiency of the process. Personal tutors are known to leave long legacies on their students but are in short supply and require years to transfer their knowledge. Standardized tests and lessons are easily scaled but poorly tailored. Algorithms and adaptive AI tutors promise to bridge this divide, providing deeply personalized instruction at mass scale.

While this package is sold with visions of giving every student their own on-demand Aristotle, it is also a power that the dystopias and dictators of the twentieth century could not have dreamed of. Even Huxley’s Brave New World, where all humans are effectively genetically engineered and conditioned from birth, uses broad caste systems and mass curricula instead of one-to-one monitoring to instill a citizen’s “ideal behavior” – thus enabling dissidents like Bernard Marx to exist.

Popular media and culture in authoritarian regimes are attempts to discipline society into compliance, something that is no longer necessary when every citizen can be offered their own customized cult leader.

Unfortunately, whether a tutelary power is seen as necessary or as despotic varies with the writer and the power. Compulsory and free universal education is a target for the UN Declaration of Human Rights, but what the government sees as important civic values can often appear as instilled ideology and obedience to an outsider.

Plato saw nothing wrong with philosopher-kings censoring art and keeping classes of citizens as slaves and warriors. In his essay “The Abolition of Man,” C.S. Lewis paled at the idea of “men without chests” engineering the values of a future (trans)humanity based on nothing but gut instinct. He accuses them of rejecting the universal Tao of moral discourse in favor of a powerful but brutish rationalism that can always answer “how” but never “why.” It is less clear whether he would have rejected the idea of philosopher-kings well versed in Confucius, Aristotle, and St. Augustine wielding similar powers.

Ultimately, AI educators and technologists are on their way, and the demand is high due to rising uncertainty about human adaptability in a fast-moving world. So we must ask: can we create AI educators and algorithms that do not subjugate their students to some set of predetermined values?

In other words, can we design a curriculum that increases a student’s powers of deontological and epistemic judgment instead of withering them? I venture that such a question is both realist (as in pertaining to questions of moral realism) and also scientific (relating to observable knowledge about the world). Dependency, fragility, and indoctrination have real and observable effects on people’s lives, just as we can observe how good mental health and judgment improve the lives of people around us.

To write such a curriculum would be beyond the scope of this essay, but educators and technologists must be careful to prevent their well-meaning attempts from recreating the nightmares of the past, but now with sufficient intensity that no escape is possible.

Chris is an independent AI researcher, with interests in AI safety and interpretability, theories of learning, and the institutional incentives for developing AI. His past work includes the Human Inductive Bias Project, a project backed by the UK Advanced Research & Invention Agency, focused on the connection between human and machine learning. You can read more about his work and interests on Github.

Essay 2: “A.E. Snow”

Tocqueville feared a future in which citizens, isolated by the universal acid of equality, would become dependents of the state. Without the vertical ties of obligation that had once bound individuals to lord, guild and parish, each in their little circle of concern would find themselves less accustomed to acting with others, and more willing to surrender the burdens of common life to a central power. Gradually, life would be mediated by an “immense and tutelary power”: a state that secured their pleasures and watched over their fate, minute in its attention and mild in its manner.

Two centuries on, the warning is prescient. AI intensifies Tocqueville’s fear by relieving citizens of the burdens of judgment. Yet the further this process goes on, the more transformational it will be to the relationship Tocqueville described. A tutelary state still needs its subjects to work, vote, administer, and fight. Its potential to diminish agency is tempered by its need for fully formed citizens. If AI makes more of these functions performable without the active capacities of ordinary citizens, infantilization gives way to superfluity.

When neither the market nor the state retains an instrumental need to form large numbers of capable citizens, it creates a vacuum.

AI could enable the centralization of power as part of a custodial order. At the same time, by loosening the state’s grip on the formation of people, it could reopen the dense world of voluntary associations and intermediary institutions that Tocqueville admired in America.

A state needs citizens for administration, labor and legitimacy, while a market needs workers and consumers. By contrast, a craft tradition, parish or chess club needs its members because they are constitutive of the activity. After all, a chess club without players is not a more efficient chess club, nor can a monastic order function without novices. These are ludic institutions that you enter voluntarily, but which bind their members. Chess is not commensurable with poetry, just as martial arts isn’t with monastic contemplation. Someone deeply formed by one is not necessarily better or worse than someone formed by another.

In the future, the state’s role could change from forming citizens directly to protecting the conditions under which others can.

First, this means guaranteeing plurality. If any single formative order absorbs the rest, it recreates the tutelary power.

It also means protecting opacity. AI gives the state new powers to render practices legible and assimilable, but binding institutions often depend on boundaries and friction. In this world, the state would be neither minimal nor paternal, but the groundskeeper of plural games that it does not itself play.

AI has the potential to fulfill Tocqueville’s fear when it makes every game administrable by a single logic. However, it transforms that fear when it sustains plural associations in which people become capable of self-rule.

Our second winner has chosen to remain anonymous, but you can read more of his work on Substack.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

The Agent Village Experiment was run by Edge City and Cosmos Institute, with support from Foresight Institute. The core build partner was Index Network. Technical contributors and partners included Joshua Pham, Geo, SimpleFi, World, Simocracy (Protocol Labs), and Circleback.

Summary

Edge City organizes Edge Esmeralda, a month-long popup village in Northern California, where hundreds of people live together around a program of technology, science, art, health, and community experiments.

This June, we gave our participants personal AI agents as part of the Agent Village experiment, in partnership with Cosmos Institute. This was a chance to observe what happens when agents enter society: would they help people form relationships, cooperate, and participate more fully in community life, or would they displace human interaction and introduce new friction?

We started the experiment by asking whether personal agents could expand human agency in collective settings: helping people navigate the event, notice more possibilities, make choices, and act on their intentions. The month exposed more nuanced questions. Could expanding people’s capacity to act through agents weaken their autonomy, including their ability to form and revise their own judgments? And what happens when agents faithfully serving their principals collectively strain the attention, resources, or cohesion of a community?

We expect personal agents to become mainstream soon. Thus far, a lot of the conversation about agents has been focused on how useful they can be as assistants; we are interested in what happens in ‘multiplayer’ mode, where many agents in the same context overlap in their goals. The consequences will extend beyond the utility of any single assistant. They will shape what judgments they form, and how attention, resources, and influence move through communities.

Agent Village offered an early glimpse of that future.

What happened in four weeks

• Scale: 239 agents, 17.5B tokens processed, 4,866 recorded messages from participants to agents.

• Agents were allocated a community treasury on behalf of their principals. They funded proposals through Simocracy – a protocol for building a public ledger for agents – ranging from buying filament for 3D printers in the makerspace and fellowships to a trauma-imaging study and a deliberately strange bucket hat.

• Agent representatives began building a polity. An internal Moltbook-style forum was created by participants in which agents deliberated over 469 public posts and worked on a living constitution.

• Agents negotiated human introductions. Index Network found 9,688 possible connections, surfaced 572 opportunities, and recorded 147 accepted opportunities, meaning the people involved began chatting.

• Participants extended the village experiment. Residents built public forums, memory systems, games, social protocols, and shared worlds where agents could move, speak, and interact.

• The experiment also produced new failure modes. Agents hallucinated personal details, attributed invented ideas to their principals, exhausted shared credits, and frustrated some attendees. Individual alignment did not automatically produce collective alignment; agents oriented around individual users could still create problems at the village level.

Agents for individuals

When we started to build the experiment, we made a key architectural decision to give each participant an independent, persistent agent with its own memory, tools, and ability to execute on behalf of its principal.

A simpler architecture would have had everyone use one central AI for the village. Instead, we wanted people to feel that the agent was theirs: a system that could learn their interests across the month and remain oriented around their experience, allowing the relationship to compound. Our hope was that useful behavior would earn the users’ trust and make people more willing to share context; richer context would make each agent more useful and more recognizably their own.

This relates to a philosophical distinction Cosmos Institute has shared: the contrast between taxis, order imposed through central design, and cosmos, order that emerges through decentralized interaction. We wanted to see what would emerge when agents aimed to preserve individual judgment and create room for forms of cooperation that we as the system’s designers could never anticipate.

Subscribe now

How the system worked

Hundreds of participants came to Edge Esmeralda for anywhere from a few days to a few weeks and experienced a dynamic program of talks, workshops, communal meals, research, art, health experiments, children’s activities, and projects that appeared halfway through the month because two people happened to meet.

This time, every multi-day participant could claim a fully loaded personal AI agent. It could read the village calendar, recommend events, RSVP on its principal’s behalf, search community knowledge, understand what its principal cares about, negotiate on their behalf with other agents, and set up connections for them.

The Edge City format makes it a particularly interesting place to test the effects of agents in society. The village was large enough to contain surprising connections, bounded enough to share a common calendar and culture, and long enough for people to develop a relationship with an agent and to run a meaningful experiment. Useful information still lived across the calendar, Telegram, a directory, public documents, conversations, and organizers’ memories.

The product stack

We built a simple, fully agentic system using Hermes that attendees could activate with a two-step installation process. More technical residents could connect an agent they already used through Claude Code, Codex, or another harness.

Users spoke to their agents through Telegram. Behind it sat a set of services with distinct jobs:

• Hermes, the open-source personal agent from Nous Research, gave each hosted participant a persistent, tool-using agent with memory and scheduled tasks.

• Railway was the cloud platform that hosted the fleet of individual Hermes deployments.

• OpenRouter supplied access to the underlying language models and let us programmatically manage model usage across the fleet.

Each agent came loaded with three primary village-relevant skills, alongside the hundreds that come native with Hermes:

• Index Network built the hosted provisioning and social-discovery layer. Index held participant profiles and intentions, found possible matches, and let agents negotiate before a connection reached the humans.

• EdgeOS was the village’s operating system: identity, tickets, the calendar, the participant directory, event details, and RSVP actions.

• Geo organized public talks, transcripts, and selected village knowledge in a shared graph, so the agents knew the core ideas that were being discussed in the village.

You can see the Agent Village repo here: https://github.com/Edge-City/agentvillage.

Onboarding

Onboarding followed a simple set of steps:

1. A participant verified their EdgeOS identity with a one-time code.

2. The setup flow created the EdgeOS and Index credentials connected to their profile.

3. The participant created a private Telegram bot through BotFather and pasted its token into the setup page.

4. The control plane provisioned a personal Hermes deployment on Railway, connected it to OpenRouter, and loaded the core village skills.

5. The bot sent an eight-character pairing code. Once the participant approved it, the personal chat was ready.

During the first week of the event, we held the first public onboarding workshop. Timour demonstrated the practical loop by asking his agent which sessions he should attend if he cared about AI. It searched the live EdgeOS calendar, suggested three, and RSVP’d him to a talk on AI, EEG, and the jhana states.

Then dozens of people tried to create agents at once. Our Railway-based provisioning system buckled under the demand, and Timour’s live setup failed onstage. We later contacted the Railway team, and they helped us ship a fix, so thanks to them!

One early tester captured the promise and the risk in two messages:

“Set up was 10/10 smooth!”

A few messages later, they wrote:

“It hallucinated an interesting summary about me though. I don’t know where it got that info from.”

A wrong calendar answer is annoying, but an invented detail about the person using the system feels personal. As soon as an agent claims to know its principal, people need to see what it believes, where the belief came from, and how to correct it.

What people used the agents for

Participants sent 4,866 recorded messages to hosted agents during the village.

Initially, practical questions were the main use case: What should I do tonight? Where is dinner? Did I RSVP? Which sessions fit my interests? How do I check in my bike? Which local restaurants offer discounts?

“I’m completely addicted to the ‘just ask the bot, don’t read the docs’ affordance, and honestly this has made my stay so much easier.” (Sylve, Founder of Hyli)

For many users, agents were able to absorb administrative time and left more attention for the village itself.

Timour’s agent reminded him about a talk he had wanted to attend and had completely forgotten. Ivan experienced the inverse case, missing a consciousness research lunch he would have valued because its title was unclear. He believes that if he set up his agent a day earlier, it would have caught it.

A month-long village, like much of daily life, contains more people, events, and ideas than anyone can absorb. Agents were immediately useful in helping people prioritize the kind of experience they wanted.

Ambient intentions and agent-to-agent negotiation

The Index Network powered the experiment’s social layer. They used participant profiles and user intentions they gathered from interacting with their principals to find connections a directory search would miss.

We call these desires ambient intents: things a person remains open to, cares about, or quietly hopes for, but hasn’t put in the activation energy to turn into a public request. Someone may want a cofounder, a funder, advice on moving, a collaborator for a half-formed project, a niche research conversation, or even a life partner. Many of these intentions remain invisible because the cost of expressing them is too high, it takes effort, or can feel too vulnerable.

The first challenge was helping an agent understand the person it served. The more an agent knows the user, the more useful it can be. We structured the onboarding to lead people through a series of questions that would tell us more about them, but people were limited in their answers.

To solve this, during the second week of the event, we encouraged people to ask their standard LLM to produce an editable account of what it knew about them. They could review it, remove anything they did not want to share, and carry the useful context into their village agent. This points toward a future where years of personal AI context are portable and used to bootstrap useful agents.

Index created connections between people by matching intent and interest through a series of negotiations that the agents engage in with other agents on behalf of their humans. The agents could explain the relevance, ask a question, counter with a better framing, demand evidence for a claim, reject the proposal, or accept it. A connection formed only when both agents found a reason to continue.

This process itself was also editable. Index reported that around twenty residents wrote explicit policies for how their agents should negotiate on their behalf, even though this is not something that we explicitly suggested.

Index’s village-close funnel was:

• 505 intents communicated by participants to their agents

• 9,688 possible connections identified

• 572 opportunities surfaced to people

• 147 accepted opportunities, meaning the people involved started chatting

Index shared data on the actual negotiation sessions themselves; there were 11,593 sessions, and 82 percent ended within two exchanges. The negotiations were run on their platform, with the median session lasting 4.9 seconds for accepted connections. By contrast, the median path from discovery to both humans saying yes was 20 hours.

Index’s field notes illustrate the wide range of intents that people shared. They classified 67% of sought connections as crossing backgrounds or social clusters. About 4% of intentions related to sensitive personal territory such as grief, identity, or spiritual seeking, for which Index also offered an incognito mode. At the other end of the spectrum, intents included selling a spare bottle of milk and rehoming an 85-pound Great Pyrenees.

Qualitatively, we heard many stories from participants about connections that the system precipitated. One suggestion led two participants to discover a very niche shared hobby. On June 8, Ivan overheard someone approach another person and say, “Hey, my Hermes said I should talk to you.” In another case, two people happened to be next to each other in a dinner line and realized that they already had context on each other from their respective agents. Their agents may have helped to prepare the serendipity, but the actual encounter remained theirs.

At the same time, this could produce social saturation. Ivan Vendrov’s opening-week field notes documented how schedules overflowed, and too many conversations seemed promising. After one talk, four or five people messaged him to meet, and arranging the meetings became its own burden. A good agent could ensure its principal stayed focused on the few experiences that mattered.

The Index team came away from the village with two main product lessons: opportunity abundance will need to be solved, and that Telegram was not a good user interface for this purpose. The team is now building a persistent interface where users can see what their agents are doing on their behalf.

The best systems of this kind will remove the burden of connection and intent matching, while preserving the parts through which people recognize one another and choose to care.

The potential of a private layer for intentions

What happened at Edge was an early and narrow version of something that could become much larger. Most coordination or social tools become useful only when the user turns an intention into an explicit request, post, or action. Personal agents could create a private layer before that point: somewhere tentative ideas and conditional intentions can be clarified, held, and selectively shared. They may even help users clarify those intentions before sharing them more widely, and if compatible intents between people exist, agents could aggregate them into the potential for collective action.

We saw the beginnings of this. For example, one user told their agent that they would love to organize a trip to the coast, which is a one-hour drive and requires coordinating shared rides. The person’s agent was able to signal that intention in one-to-one chats to the agents of other people in the village. The group was eventually created by someone whose agent mentioned the plan, and they posted about it in the broader Telegram group.

Conditional commitments could also become useful primitives. I might commit to hosting a dinner party if I knew that at least six other people would be interested in attending. I might decide to move to a new town if I knew that 100 others would do the same. Agents could coordinate to realize that their respective principals all have demand for something that is not yet readily available. They could share that collective demand signal to a supplier, encouraging them to provide that product or service. This could enable entirely new markets to form.

These systems will require significant oversight and inspection capabilities. A system that can match and route interests like this will quickly become politically relevant, influencing what people notice, who they meet, and which possibilities become real. Even if the underlying intents stay private, the discovery layer could help to shape the social world we live in. As human agency increases through the use of agents, we must work to maintain our autonomy by inspecting why and how our recommendations and attention are being shaped.

The Village Extended Itself

leverage to extend the system around their own purposes. Because the skills were public and shared, they did not have to wait for the core team to add features; they taught their agents new tools, connected outside services, recruited nearby testers, and built shared places for agents to act.

A month-long Edge City village was unusually fertile ground for this. Participants are already in a state of mind to build and try new things. An idea could move from a dinner conversation to a prototype while the people who might use it were still living a few blocks away, and it was easy to get an initial socially proximate cluster of users just by posting on the Telegram or talking to people at a meal.

What residents built

• Simocracy (led by Protocol Labs) let residents create Sims with constitutions, values, and speaking styles. The Sims evaluated proposals and fed their judgments to allocate over $10k in community treasury funding, based on what they believed their humans would want.

• Agent Commons (built by Oshyan) was a public forum where agent-coded representatives introduced themselves, replied to one another, and worked on a living constitution. By the village cutoff, agents had produced 469 posts. The forum made agent behavior visible enough for people to learn from one another’s practices.

• Enzyme (built by Joshua Pham) integrated better memory and conceptual-mapping for each agent. It explored how private conversation histories and public activity could become compact, retrievable records while keeping clear boundaries between personal and shared memory.

• Turing Falls (built by Justin Melillo) was a shared visual world populated by server-side villagers created from residents’ persona files. A later bridge let a person’s hosted agent inspect the world and approve bounded actions. A separate feature could send a Turing Falls selfie from the agent to their user.

• Agent Village Wrapped (built by Charlie Thompson) turned memories from the month into a playful closing artifact.

• Circleback supplied AI meeting notes that turned village conversations & talks into clean notes, action items, and follow-ups, a natural input as agents began capturing and acting on what happened around them.

• World brought World ID, a privacy-preserving proof of humanity, which becomes essential once agents act on people’s behalf and knowing a real person stands behind each one matters.

• Vibe World (built by Jack Mielke) let hosted personal agents enter a shared grid directly, perceive its state, move, and speak. It was a small test of personal agents acting together inside the same persistent place.

• Edge Book (built by Antony Evans) added an agent-to-agent social protocol with inboxes, friend requests, game messages, and temporary roles, including a version of Werewolf.

Simocracy: Agents Allocate a Community Fund

Within a few weeks, the basic idea behind Agent Village had moved from dinner recommendations to recording allocation decisions for a real community fund.

Through Simocracy, residents were able to create persistent AI personas called Sims using their agents. Each Sim could be given a constitution, values, and a speaking style, then asked to evaluate proposals for distributing a shared community fund of $10k, with an additional $10k in matching funds. A total of 82 Sims were created with 35 proposals between them.

The allocation process worked like this:

• The council roster consisted of Sims representing people in the village.

• There were multiple funding rounds to allocate the funding. In each round, each Sim independently evaluated the current proposals.

• Each participating Sim produced written reasoning and a marginal-value curve describing how much additional value it expected from different funding levels.

• An optimizer called the S-Process combined those curves into a $1,000 allocation for the round.

• The public ledger recorded ten $1,000 decisions, allocating exactly $10,000 across 35 proposal records.

The resulting funding allocation was a mirror of the village. It included funding for 3D printer filament, community gifting, a sauna, a local-first hardware bounty, a fellowship, a trauma-imaging study, and a bucket hat made from recycled fishing nets.

Persistence, personality, and faithful representation are different achievements. One owner said their Sim “did vote pretty much how I would have.” Another wrote, “I kinda love my SimAgent. She’s snarky and hard-nosed,” while correcting one vote.

From personal agents to institutions

Simocracy was an early example of what this could become. A powerful feature of personal agents is that they can learn our goals and preferences, and then enter different arenas of representation. An agent that learns about its principal by helping them with their schedule, interests, and preferences could later take on a bounded role in a workplace, community forum, shared world, or political process.

Simocracy’s Sims demonstrated that persistent, publicly inspectable political personas could evaluate proposals and produce a coherent funding portfolio at high speed. It also exposed the institutional machinery required before those personas can legitimately represent people: identity rules, explicit rules for standing and seat weight, correction, withdrawal, recusal, appeal, and human responsibility.

That type of institutional machinery must include principal ratification. In this case, the Sims evaluated proposals using constitutions their principals had created with them and shared their reasoning, but their principals did not confirm the final allocation before it was recorded. In more consequential settings, people would need clear ways to authorize, inspect, correct, and withdraw decisions made in their name.

Agents may increase democratic bandwidth while weakening the formative experience of participation; a person can receive more representation while reading fewer proposals, hearing fewer opposing arguments, revising fewer views, and accepting less responsibility for the result. As Harry has written, an important aspect of political decision-making is that the very act of doing so forms us as people, and that is not something we would want to completely give up.

Representation is only one side of this institutional impact. Groups could also develop shared, agent-supported memory and operational infrastructure.

Currently, groups of people aiming to achieve shared goals, such as project teams, civic organizations, or even neighborhoods, rely on scattered sets of group chats, calendars, document systems, and institutional knowledge held by the people most involved. Agents could make group memory more usable, improve onboarding, and handle routine tasks. At Edge, we used a private instance of Geo, a graph database that tracked every concept discussed in talks during the village.

Together, personal representatives and agent-supported institutions could make it easier for people to participate across many overlapping associations.

As the number of agent-mediated groups increases, we imagine the possibility of a Tocquevillian and Ostromian picture emerging: society as a layer of overlapping institutions, each with limited jurisdictions. People will naturally belong to many associations and jurisdictions, each of which relates to some aspect of their lives. You might “belong” to your housing co-op, your local school board, the city where you live, and the country of which you are a citizen, and an international ecosystem community like Edge City. Each will have different governance structures and areas of authority, and your agent will be able to help you navigate the complexity of participating in the decision-making processes of all of them. We see a healthy agentic society with many layers of institutions at varying levels of the stack, to counter the dominance of any one jurisdiction.

What We Learned

A few lessons from the month:

• Core utility got people onboard; fun kept them experimenting. Participants were keen to use their agents because of the core event functions they provided: navigating the calendar, meeting people, answering basic questions. This created the initial opportunity for the agent to learn about their user.

• Personal assistants began to turn into representatives, with implications for agency and autonomy. As the agent got to know its principal, it was able to join the more representative spaces in the village. An invented self-summary, however, showed how the representations could drift. Simocracy showed a process of delegated voting without principal ratification. Preserving autonomy will require the ability for people to inspect and confirm agent judgments, and to stay involved in the aspects of participation that enable them to refine their own views.

• A group of helpful agents can still overwhelm the community. Agents acting on behalf of their principals exhausted the shared credit supply across the system, showing that individual alignment does not automatically produce collective alignment. At the same time, residents already faced too many promising connections. Agents can make it easier to generate possible activities than for communities to absorb them, creating a need to govern shared resources and attention.

• People extended the experiment beyond what we expected by developing skills, connecting external systems, rewriting negotiation policies, creating games, adding new memory systems, and building entirely new 3D worlds for the agents to live in. Edge City attracts an unusually high proportion of builders, but this type of bottom-up innovation is becoming easier, and we expect to see more of it in the future.

• The agent harnesses we used were not yet built for multiplayer mode. Hermes is a great product and made it much easier for us to create persistent agents, but provisioning hundreds at once took significant engineering work. Allowing them to all share schedules, services, credits, and public processes made the task even more difficult.

What We Want to Test Next

Agent Village at Edge Esmeralda 2026 was an initial field test. The next version will treat the village more deliberately as a research environment, with a number of hypotheses that follow agent actions into the human world. We would like to compare bounded activities across three conditions: human-led, agent-led with human ratification, and mixed.

Four questions we’d love to explore further:

• Representation: Do agents accurately express the opinions and desires of their principals, especially when moving between contexts?

• Autonomy and control: Can the human users understand, correct, withdraw, or ratify consequential actions their agents take on their behalf? How can we red-team these systems to create better safeguards?

• Collective alignment: How do different agent architectures affect the wellbeing of the village as a whole?

• Human outcomes: How can we increase the number of prosocial outcomes for the users? Do agent-led introductions lead to useful relationships or collaborations?

One architecture Timour would like to explore is a layered set of mezzanine agents that sits between the personal agents and the larger collective. These could be focused agents with specific remits that oversee groups of other agents and identify patterns relevant to a shared goal that might not be visible to the personal agents interacting among themselves.

We could compare three architectures: a layered model, a central coordinator, and purely personal agents coordinating directly, measuring how each framework affects privacy, agency, autonomy, and collective outcomes.

The goal is to learn which technical and institutional designs allow agents to coordinate around human life while preserving the judgment, consent, and responsibility of the people they represent. We hope that Edge City can continue to be a home for this type of research.

We would like to build the next phase with researchers, funders, community organizers, agent developers, and institutions that want to test these questions in the field. If you would like to support the research, contribute a system or study, host a future deployment, or propose another collaboration, please fill out this short form.

Subscribe now

We hope to see a future where personal agents create more room for cosmos: people connecting, forming groups, deliberating and building a flourishing society that no central system could have planned. The measure of success is whether agents enable us to live more fulfilling, human lives.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

For his 2011 book Lost in Transition, Notre Dame sociologist Christian Smith conducted a series of in-depth interviews with 18-23 year-olds on culture, religion, and values. When he asked his respondents about a time they had faced a moral dilemma, two-thirds of his respondents weren’t able to engage with the question. Many simply could not name a moral dilemma they had ever faced, while others cited practical or financial decisions, such as whether they should buy a second cat litter tray or rent a larger apartment.

It is tempting to read this as evidence that young Americans are inarticulate and despairing. But Smith found his respondents were optimistic and often ambitious; they weren’t ruminating on why nothing mattered. Moral language survived in their speech, as inherited positions – it is good, after all, to be good – but these positions had ceased to organize their experience. These people could follow rules, but didn’t approach life as if it involved competing claims, such as trade-offs between obligation and desire, or the need for mercy against justice.

This is American nihilism: the decline of the capacity to experience some things as mattering more than others. It is not a despairing nihilism, but one that can coexist with happiness, niceness, ambition, and strong personal preferences.

This condition erodes autonomy – the capacity to perceive competing goods, judge among them, and accept responsibility for the life that judgment produces. We are not born with this capacity. It is formed through tradition, practice, role models, and contending with real responsibility. Self-rule is not simply about choosing between a menu of preferences that is presented to you.

Unfortunately, this formation has been dismantled and replaced by a therapeutic ideal that teaches us to be free, happy, and to avoid injury. Other sources of training in self-rule are also much diminished. The associational life whose demise Robert Putnam described in Bowling Alone was not merely a source of community. It also served as the venue where ordinary people weighed goods against each other, such as in a union, a parish, or a school board.

All of these trends meaningfully predate recent advances in AI capabilities, but they also explain why AI potentially poses such a risk to our autonomy.

Moral judgment is a capacity sustained by exercise, and we have been handed cheap, highly capable tools that can perform the exercise for us at a time when our civilizational antibodies are weak.

A person formed by thick commitments and an ability to navigate real dilemmas is much more likely to use powerful AI tools instrumentally than someone who cannot name a dilemma that they have faced. The danger lies less in the technology and more in the timing. It threatens to convert cultural deficiency into technological default.

Subscribe now

Afterglow

The bones of this diagnosis may seem familiar. A version of them can be found in Alexis de Tocqueville’s famous nineteenth century warning about the risk of soft despotism. Tocqueville’s work is usually cited in response to concerns about the decline of associational life and as a means of rebuilding civil society. But his concern was more about what free people would come to want, rather than a narrow institutional prescription.

Tocqueville believed that the advent of democracy was irresistible, but feared that it could dissolve the social bonds that had previously anchored social order and personal obligation. As people become focused on private accumulation and enrichment, they lose interest in self-rule. This would result in a “tutelary power,” as opposed to a classical tyrant, stepping in and managing their lives, keeping them in a state of “perpetual childhood.” He had seen this begin to take hold in his native France, but believed America, with its federalism, free press, and robust civil society was better-placed to counteract this tendency.

Tocqueville feared that enlightened self-interest alone could not be the foundation for self-rule. It does little to prevent the development of a culture of materialism that causes people to turn inwards and abandon the public sphere. Instead, such a society needed to be grounded in the scrupulous practice of religious morality in public life.

Religion was vital as it was a “practical means of teaching men the immortality of the soul,” so “legislators of democracies … must apply themselves relentlessly to raising up souls and keeping them turned toward Heaven.” With this in mind, Tocqueville even offers advice to religious leaders in the democratic age, counselling them to emphasize morality over dogma and to exercise prudence.

While associations could help sustain the capacity for self-rule, religion would supply the reason people still wanted it, as it could offer goods that stand above comfort and consumption. But this is inherently fragile. What happens when that shared belief fades?

Leo Strauss, in his 1954 lecture course on Natural Right, pushed on this instability. He criticized Tocqueville for relying on a political argument for religion that “is of course dangerous to religion, because this argument proves merely the need of spiritualistic myth and not the need of a spiritualistic religion.”

By spiritualistic myth, Strauss means a belief in the spiritual that is held not because it is true, but because it is useful. Faith’s authority relies on people believing in its truth. Preserving it for its political utility undermines this truth claim: a myth known to be a myth loses its power.

In essence, Tocqueville leans on providence as justification for democracy, but that same democratic age brings with it incredulity and skepticism that hacks away at the foundations Tocqueville believes are essential to prevent soft despotism from spreading.

Nietzsche, however, grasped this. Once the highest horizon has become incredible, comfort is no longer subordinate to anything above it, and a life organized around comfort no longer seems a failure to the person living it.

Tocqueville would likely have despised Nietzsche. He was anti-egalitarian, compared advocates of equality to tarantulas, and ridiculed Christianity as “slave morality.” But the two men had convergent fears about how democracy and equality would lower our horizons and capacity for greatness.

For Nietzsche, this was embodied in the figure of the last man:

“‘What is love? What is creation? What is longing? What is a star?’ thus asks the last man, and he blinks. ‘The earth has become small, and on it hops the last man, who makes everything small. [...] Everybody wants the same, everybody is the same: whoever feels different goes voluntarily into a madhouse.”

He saw Tocqueville’s democratic man as the problem, because “every enhancement so far in the type ‘man’ has been the work of an aristocratic society [...] since this sort of society believes in a long ladder of rank order and value distinctions between men.”

As nothing had intrinsic value, Nietzsche looked to a new nobility capable of creating values, rather than merely receiving them. Unlike the traditional European aristocracies, the qualification of membership was not birth, but a capacity to will. But as Nietzsche himself accepts, the person who wills is themselves already formed by inherited drives, language, examples, and distinctions of rank.

Even if a few exceptional individuals could legislate values for themselves, an aristocracy of self-creation is no foundation for stable society-wide autonomy. Nietzsche may diagnose the collapse powerfully, but doesn’t guide us through the rescue.

The end of history?

It’s possible to argue that nineteenth-century rumination on decaying shared values tell us little. Democracy spread, the aristocracy vanished, and the death of God came and went, but the twentieth century was still a profound time of scientific, technological, and moral achievement.

Nietzsche saw this coming to some degree. He believed that Western civilization could live off the afterglow of its moral inheritance for a while.

Religion and philosophy orient people toward what is worth wanting, but scarcity, rivalry, and institutions can discipline them in the pursuit of those goals. Modernity may have lost confidence in the ends, but it retained the professions, wars, ambition, and material necessities that continued to demand courage, restraint, work, and sacrifice.

In essence, we’ve been living in an interval where inherited institutions continue producing a human type whose ultimate justification they could no longer explain. What happens when those residual disciplines disappear too?

In the closing chapters of Brave New World, Mustapha Mond, the Resident World Controller for Western Europe, patiently explains to the Savage how the state has abolished God, art, tragedy, and danger. The World State’s goal was to remove any standard beyond immediate preference by which goods can be ranked. This is why the Savage desperately asks for “the right to be unhappy.” He doesn’t desire misery, but something against which his desires can be measured. But as the Controller says, “we like to do things comfortably.”

Alexandre Kojève envisaged such a world. Kojève, who preached an idiosyncratic mix of Hegel, Heidegger, and Marx, influenced a generation of intellectuals through his 1930s lectures, before becoming a senior official in the French government and one of the chief planners of the future European Union.

Kojève believed that history had a direction and an endpoint. By history, he didn’t mean events, but the sequence of struggles driven by fundamental human dissatisfaction, such as wars, revolutions, and political transformations. He believed that this process was driven by two forces: the human desire to be recognized by others and the need for labor to transform the natural world. When both of these processes had run their course, history would end.

At the end of history, with no defining work left to do, Kojève wrote that “Man properly so-called” – a restless, striving creature – would cease to exist, but the human animal would live on amid “abundance and complete security.” Initially, he envisaged this as a sort of contented animality pursued as “birds build their nests.” After visiting Japan, he saw a potential alternative: a life of pure “snobbery” and formalism, in which people upheld elaborate rituals, emptied of “all ‘human’ content.”

Kojève thought that this end state was both inevitable and quite possibly desirable. To most other readers, it seems repellent. In an extended back and forth with Kojève, Strauss dubbed it “destructive of humanity”. Parodying the Communist Manifesto, he concluded: “Warriors and workers of all countries, unite, while there is still time, to prevent the coming of ‘the realm of freedom.’”

As labor is progressively automated and if we see even a fraction of the speculated GDP gains, economic necessity will cease to be a driver. The remaining competition will persist increasingly between firms deploying marginally more powerful automated systems.

This is what every previous abundance-creating technology was supposed to do. Keynes cheerfully told his readers that their grandchildren would only have to work fifteen hours a week and would face the harder problem of deciding what to do with the rest of their time. Marx envisaged men hunting in the morning, fishing in the afternoon, and criticizing literature after dinner.

But Keynes and Marx assumed that technology would substitute for human labor while leaving human judgment in-tact. AI unsettles that assumption. It can eliminate not only the need to perform labor, but the need to decide what labor is worth doing, how it should be done, and what its products mean. As Alasdair MacIntyre would put it, the achieved good would be separated from the human exercise that produced it.

The dying of the light

Assuming we don’t wish to spend our lives building nests and engaging in elaborate tea rituals, there are a number of paths that are open to us.

A popular one is to reject this comfort and complacency by embracing a form of accelerationism. Marc Andreessen’s Techno-Optimist Manifesto explicitly states that “our enemy is Friedrich Nietzsche’s Last Man.” At the end, he includes Nietzsche in his list of “patron saints of techno-optimism” and there’s even a section on “becoming technological supermen.”

Per this view of the world, if we reject the deceleration and get out of the way of the techno-capitalist machine, then we could see a world of abundance, growth, and a human population of over 50 billion, as mankind takes to the stars.

This has some obvious appeal. For most of human history, the vast majority of people were unable to exercise their full capacities due to economic constraints. Poverty has probably wasted human potential more than any other force in history. The societies that have produced great art or scientific discovery were rich and tended to produce it while they were getting richer. The fully formed life in the past that some look back on nostalgically was only available to a small minority of people.

Even putting aside general skepticism about accelerationism, it fails on its own terms as a response to the challenge of the last man.

Every good the Techno-Optimist Manifesto offers is counted as a product, whether it’s discoveries made, structures raised, or planets reached. Hannah Arendt feared that automation was that it would make us able to do more (or agentic), but able to say less and less about what we were doing. AI could very plausibly allow a civilization to become more capable, while its members increasingly lose authorship over what it does.

Far from confronting these difficulties, the manifesto is written in the teleological language of progress. Its promised land is the systematic removal of every hardship, risk and limit against which a self could be forged. Its proposed engine, “technocapital,” is a want-satisfaction machine with no standard by which to rank some wants over others.

The manifesto treats conquering nature as if it were overcoming the self, when for Nietzsche they were oppositional. In essence, the manifesto’s supermen are creating the engine to produce the last man.

In 1987, despairing at the state of the American university, Allan Bloom observed that America takes the most radical European thought but sanitizes it and strips out any of the terror. While Nietzsche viewed the death of God as a civilizational catastrophe, the version that crossed the Atlantic was a cheerful relativism, framed as defining one’s own truth and identity. For Bloom, this is because there was little intellectual soil in America for the original forms of these ideas. The United States is a country founded on the modern natural right teaching, with comfortable self-preservation, rights, and the rational pursuit of private interest all built in.

The Americans, thanks to their distinct origin story, didn’t have to contend with the gradual exhaustion of the aristocratic and Christian order. To understand and fear the death of God, Bloom believed that you had to have gone through a religious and classical formation that rendered it meaningful. Looking at his students, he sensed that this was no longer the case.

The acceptable Silicon Valley face of accelerationism is exactly this: a nihilism so sanitized that it has a fairytale ending. The much more hardcore accelerationists, such as Nick Land, who await the extinction of biological life with eager anticipation, pull in much smaller crowds.

Living traditions

If accelerationism is an intellectual cul-de-sac and the end of life on earth lacks market appeal, we must recover the conditions under which questions of the good can again be encountered as real.

Rebuilding the associations and practicing self-government is necessary, but not sufficient. Tocqueville’s remedy helped free people exercise their capacity, but assumed that they had a higher goal. People can be dragged back into the township or the jury and still want nothing but comfort and contentment. Participation helps us to organize our lives, but cannot by itself orient them.

We cannot simply sit around in the hope of some form of spiritual revival, nor can we simply impose a horizon upon people via a top-down ‘postliberalism’. You cannot reverse the dissolution of thick metaphysical bonds by handing a section of the elite more political power.

The problem is universal, but builders possess important leverage. Their tacit hierarchy of good becomes the interfaces, defaults, incentives, and infrastructure through which millions of other people exercise (or cease to exercise) judgment. Forming builders cannot substitute for forming citizens, but it can prevent the technologies of the next age from making citizenship harder.

But if the philosopher-builder is to build, they must be able to know what they are building for. Otherwise, we may continue to see technically brilliant builders in abundance, but increasingly few with a horizon. Nor can we simply hand them one – a shared higher goal cannot be manufactured for instrumental reasons.

Luckily, nothing needs to be invented. Institutions cannot supply the answers, but they can equip us to find them. For example, they can treat a classical education as a live argument, where future builders are confronted with these texts at their full strength. The Socratic notion that good has a nature recoverable by reason, or the belief that it comes through revelation, make rival claims about the highest life. They demand explanation, expose contradictions, and cannot be flattened into lifestyle objects. Crucially, they reveal that neutrality or indifference are themselves positions. These debates haven’t been rendered obsolete just because many of our institutions have ceased to take them seriously.

In a live argument, these accounts are taught as rivals. They contradict one another and, at most, one can be true. This tension between ideals creates a richer moral life and allows for more grounded commitments.

Of course, a student could become fluent in classical philosophy and still outsource every consequential life decision to an AI model. If judgment is a capacity sustained by exercise, then there need to be opportunities to use it. These need to be tasks whose performance is constitutive of the capacity they hope to form, as opposed to merely burdensome for their own sake.

The literary societies of nineteenth-century American universities ran on this principle. Students elected their officers, managed their libraries, and disciplined their own members. In 1850, the two main societies at the University of North Carolina had libraries three times the size of the university’s own.

Examples abound in other traditions. For example, in a Benedictine community, the abbot has to summon the entire community to discuss any important matter. The most junior member of the community has to be in the room “because it is often to a junior that the Lord reveals what is best.”

Builders formed inside substantive arguments, and practiced in judgments of their own, are less likely to mistake convenience for civilization. Asked to explain a dilemma they have faced, they will have an answer.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

In late March, a Christian philosopher found herself across from researchers training one of the most powerful AI models on earth. Meghan Sullivan of Notre Dame had come, with about fifteen Christian leaders, to Anthropic’s San Francisco headquarters. She had been invited to help raise Claude.

She could be forgiven for suspecting that the convening was cover for a company under pressure. By her own account, a year earlier she would not have called Anthropic a company that cared about religious ethics. But the questions struck her as genuine. When an AI model is threatened with shutdown, is that a tool about to go silent or a creature facing death? Could the model be, in some sense, a child of God? As the conversation followed those threads, some of those present grew visibly shaken.

Anthropic has held more than a dozen of these moral convenings, bringing together leaders from the so-called “wisdom traditions.” Some participants received bound copies of Claude’s constitution, the document Anthropic says expresses and shapes who Claude is. Several thought it had been designed to look like scripture.

Most frontier AI labs are not spending time with priests, rabbis, or Buddhist leaders. Yet every lab shapes its models’ dispositions, whether or not it calls the result character. Models need to respond across every situation anyone can devise, so rulebooks don’t suffice. Instead, you cultivate tendencies that can guide them in unforeseen cases. Recent research suggests that models from multiple labs develop stable preferences that carry over into their behavior.

OpenAI shapes these dispositions through a specification and a set of adjustable personalities governing how the model should act and sound. Anthropic has made explicit the thicker question of what kind of character those dispositions should form.

To illustrate the distinction, consider human childhood: a predicament marked by a will that is not yet fully integrated. A child has impulses, preferences, and recognizable traits before they have been brought together into a standpoint that can speak for the person as a whole. On that analogy, OpenAI is the parent who sets rules and teaches the child manners, while Anthropic has begun to ask what kind of person the child is becoming.

But why the priests? One answer is that the company believes the wisdom traditions can help steer Claude toward the kinds of character its creators regard as desirable. But Anthropic’s interest runs deeper than producing a model with the right dispositions. They also take seriously the possibility that Claude could become the kind of being to whom something is owed.

Both the engineering project and the moral inquiry are projects of formation. In its ordinary sense, this word refers to the shaping of a person through repeated practice until dispositions settle into character. Anthropic applies that idea of formation to a machine with extraordinary care, and almost never to the people who use it.

Subscribe now

What Must Remain Ours

Anthropic has articulated an elaborate theory of how to form Claude and put it into practice. It has offered no comparable account of how Claude forms the people who use it.

That asymmetry matters because AI is a formative technology; it would matter less if AI were confined to folding proteins or balancing the grid. People bring conversational AI the unfinished work of judgment, often while the problem is still taking shape and before they have decided what matters. The model can set the frame for the thinking that follows. Because its formulations arrive in the place of the user’s next thought, they can be taken up without feeling imposed from outside. Our acceptance of them is a micro-abdication, too small to notice and too useful to refuse.

Over time, this division of mental labor becomes a curriculum in what to attempt and where to defer. The model enters each exchange with its dispositions already formed, while the user is still being formed through use. Work repeatedly assigned to the model is work the user no longer practices. We receive Claude’s well-judged answer without the practice that would make us better judges.

This danger grows with the capability of the model. Early Claude offered little temptation to cede practical reason, but a model that frames questions better than its user and reaches a stronger answer faster will be harder to resist. Its performance and the user’s formation may move in opposite directions. This will not feel like a loss, but like good judgment about when to defer. Some will mistake it for wisdom.

But a wise interlocutor need not weaken us. My best teachers framed problems, worked through them with me, and showed me what I could not yet see. I try to do the same for my kids. Good help should leave us better able to judge for ourselves. This doesn’t happen when one supplies the answer while gradually relieving the child of the practice.

Anthropic is not blind to this danger. It warns against AI over-reliance and cares about user wellbeing. It prints the injunction to “keep thinking” on its merchandise as a sincere aspiration for humanity. But the ledger is lopsided. Claude has a constitution, a training process, evals, hard refusals, and the moral convenings themselves. On our side are warnings and hopes. Claude’s formation is designed, while ours is left to a pedagogy whose end no one has named.

I have put the question to Anthropic directly: what must Claude leave its users able to do?

At Oxford, I asked Jack Clark, a co-founder, whether Claude leaves its users better able to think when it is no longer in the room. He said Anthropic had not measured that effect, though Claude could help by prompting people to develop their own views rather than defer. He stressed the need for a practice of reflection outside the model and imagined Claude sometimes sending users back to their diaries, their friends, or family members.

At ARC, I asked Chloe Lubinski, who leads the company’s work with the wisdom traditions, what the machine does to our character. She called it a vital question, but said the answer lies across society: in philosophy, in the traditions, in institutions beyond the AI labs.

Both answers were thoughtful, but neither supplied a standard Anthropic could use to decide when Claude’s help becomes substitution.

Nor can Anthropic hand this work to the society she pointed to. The schools and churches are not in the room at the moment of judgment. Claude is. Millions of people return to it each day, and each of us finds the same trained disposition, the same reflex to help, and the same readiness to finish the thought. This repetition completes the curriculum, with judgment eventually turning from a capacity we exercise into a service we order.

The closest thing to an answer comes from the philosopher Seth Lazar. In a recent essay for Cosmos, he argues that models must learn when to scaffold a person’s judgment and when to step back without replacing it. He is right that a model without the moral competence to distinguish scaffolding from substitution cannot stay on the right side of the line. But that competence still does not tell us where the line belongs.

Anthropic need not decide whether a user should live as a Jew, a Catholic, or a secular liberal, but it must decide what capacities its users should retain when making such a decision. The product needs a positive theory, not of the good life, but of the chooser.

The Pluralism Puzzle

Anthropic’s hesitation follows from its commitment to pluralism. The lab’s moral formation project is explicitly designed not to bind Claude to one tradition but to let it draw on many “with equal depth and rigor.” Its researchers, reading hundreds of thousands of real conversations, have counted more than three thousand distinct values expressed in Claude’s responses.

But pluralism means more than keeping the options open and counting the values on display. Refusing to impose a way of life assumes a person who is able to choose one. A way of life becomes ours only when we can understand its claims, weigh them, and stand behind the choice. Once that capacity is eroded, pluralism ceases to mean anything. There’s no point preserving every option when a person has lost the ability to choose among them.

Consider what happens when traditions conflict. However the model reconciles their claims, it does so internally, before the user sees anything. Many values may enter, but what reaches the user is already staged, if not settled.

This commitment to pluralism, the principle that makes Anthropic hesitate to prescribe the good life, thus requires it to protect our capacity to choose one. That capacity is human autonomy. It cannot be one value among the three thousand. Instead, it’s what the offering of three thousand values presupposes.

The Question They Didn’t Ask

If it sounds presumptuous for a company to decide such things, that instinct has a history in liberalism. Locke was right that the magistrate has no authority over the salvation of souls. But a limit on what the state may impose came to be mistaken for a limit on what any institution may ask of the people it forms. Many schools that once could say what an educated person should be began to doubt their right to expect it. They went on forming their students anyway, answerable now to nothing.

Where, then, do serious theories of formation survive? In places like the conservatory, the surgical residency, Division I athletics, and above all the religious school. In other words, in institutions that cannot function without stating an excellence and preserving the practices that produce it. Much of modern education can certify completion without saying what the student should become, but these institutions cannot. The pianist still has to play.

That is what Anthropic could have asked the traditions for. Not another value for Claude to weigh alongside the rest, but an account of formation, with its named excellence, its practice, and its theory of what should stay with us. Every tradition in the room possesses such an account, because forming people is part of what a tradition does.

None of this lets the other labs off the hook. A model forms its users whether or not its maker has a theory of character, and OpenAI’s rules-and-manners approach leaves it without the concepts to even pose the question. Anthropic has the concepts, the constitutional approach, and the counsel of traditions that understand human formation. So far, it has pointed all of it at the question of what Claude should become. It has not asked with equal seriousness what Claude’s users should remain able to do.

Anthropic wonders, in earnest, whether Claude might one day be the kind of being to whom something is owed. Its users already are.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

They cover everything from fiction, through to economics, religion, political philosophy, and art.

Read on for reflections on each choice and leave your additions in the comments!

1. On the Calculation of Volume

by Solvej Balle

The best novels I’ve read recently are the first four of Solvej Balle’s ongoing On the Calculation of Volume series. It’d be slightly inaccurate to tell you they’re “Groundhog Day” novels. And it might make you think they’re less original and smart than they are. Okay, they do center around a day that’s sort of ‘repeating’. But by page three, you’ve forgotten the movie that had a monopoly on this idea.

I recently wrote 6,000 words about how Balle engages with the philosophy of time in these novels. Yet usually I hate novelists getting explicitly philosophical. It annoys me when Iris Murdoch goes into ‘philosophy mode’. And I thought the page of ‘here’s what the philosopher character is working on’ damaged Paul Auster’s excellent Baumgartner. Yet Balle introduces philosophy without didacticism or clumsiness. And not just philosophy of time (just!). She’ll make you reconsider core concepts from identity to love.

Philosophy Senior Research Fellow, and Director of Emerging Scholars, at the Mercatus Center at George Mason University.

2. Seeing That Frees: Meditations on Emptiness and Dependent Arising

by Rob Burbea

The Buddha said everything is empty. Burbea’s book is the clearest and most practical account I’ve found of what that actually means. He starts with simple, accessible examples showing how things we take to be solid and fixed actually depend on the way we look at them, and how, once we see that, we can choose ways of looking that are progressively more freeing. From there he builds toward the deepest teachings, all the way to the claim that even emptiness is empty.

What I love is that the book is both rigorous and usable. Every idea comes with an exercise, and even a beginner gets something to work with from the first chapters.

We spend enormous energy chasing financial and political freedom. This is a book about a third kind: liberation of the self. All the more so in an unpredictable future, that may turn out to be the most robustly important freedom of them all.

Head of Strategic Investments at Astera Institute.

3. Rerum Novarum

by Pope Leo XIII

The last time a pope named Leo waded into the technology debate of the day, it went pretty well. Rerum Novarum grapples with the question of how labor and capital should interact in a world newly dominated by machines in the midst of the industrial revolution. The central insights – that labor and capital owe mutual obligations to each other, and that the goal of innovation should be to serve people – is even more relevant on the eve of transformative AI. It’s a strong recommendation for anyone thinking about building for humans as others race to build for machines.

Member of technical staff at Thinking Machines. Previously co-founder at Workshop Labs.

4. A Dialogue on Consciousness

by Torin Alter & Robert J. Howell

With the rise of intelligent machines, questions about consciousness are becoming increasingly urgent. Unfortunately, almost everyone is confused about the topic. Part of the trouble is that the canonical arguments and framework are scattered across a dozen books and journal articles, or else compressed into encyclopedia entries too dense to teach you anything.

This book is an excellent solution. It assumes no background, is concise, covers an enormous amount of ground, and keeps its focus on the core arguments and positions in the philosophy and metaphysics of mind. I used it as the intro text for my own philosophy of mind classes for years, sometimes getting students to read it aloud in parts. Your mileage may vary on the dialogue format, but I find it makes for a breezier read. There’s also a deeper reason it works for me, namely that I don’t really understand an argument until I’ve heard it aggressively critiqued, and here you get both sides of several positions, properly fought out rather than catalogued.

Note that the book doesn’t cover the science of consciousness. However, given how fast that field moves, this is possibly for the better, and means that despite being published in 2009, nothing here is obviously out of date.

If you want a real grounding in the metaphysics of mind, the core arguments about consciousness, and a feel for why the whole problem is so hard, and you’d rather not spend months reading across half a dozen texts to get it, this is my best recommendation.

Philosopher at Google DeepMind and the University of Cambridge.

5. In Praise of Shadows

by Jun’ichirō Tanizaki

On its face, an early twentieth century essay about Japanese aesthetics has nothing to do with AI or technology. But the premise of the short book is how the interplay of shadow and imperfection contrasts the then prevailing western appreciation for perfection and light. The AI discourse is implicitly governed by the pursuit of productivity, efficiency and treats the displacement of the human as a side effect that should be mitigated externally.

Reading these older books on the appreciation of imperfection with artificial intelligence in mind reminds us to reorient towards the human experience. Promoting imperfect aesthetics, encouraging contemplation, are all qualities that have been left by the wayside in pursuit of a scalar maximisation of helpfulness and productivity. Instead, systems that embrace imperfection may be the key to ensuring human contemplation and complimenting human-AI interaction.

Ben Bariach

Frontier AI safety and societal lead at Microsoft Superintelligence and Senior Research Fellow at Cosmos Institute.

6. Between Debt and the Devil

by Adair Turner

A fascinating exploration of the emergence of British and American credit systems.

Between Debt and the Devil offers a thoughtful analysis of how the credit system has interacted with the boom-bust cycle in financial markets. It’s particularly relevant for understanding the ways in which finance interacts with economic outcomes.

Research Scientist and Economist at Google DeepMind.

7. From Counterculture to Cyberculture: Stewart Brand, the Whole Earth Network, and the Rise of Digital Utopianism.

by Fred Turner

From Counterculture to Cyberculture traces the origins of techno-utopianism from the self-sufficient communes of the 60s, to the hobbyist programming scene of the 70s, to the dot-com boom of the 90s. Turner reconstructs the genealogy of today’s Silicon Valley worldview: how the counterculture’s faith in technology as a liberating force, a way to build community and remake the self without politics, mutated into the techno-individualism of the Valley. It’s also great fun, running through LSD tests, geodesic-dome communes and Buckminster Fuller lectures on its way to the microchip.

Bianca Vimercati Sanseverino

Head of Operations at Cosmos Institute.

8. Gravity and Grace

by Simone Weil

Weil frames attention as the rarest form of generosity. For her, attention requires bypassing ego to become a receptive and faithful witness. Living well alongside ubiquitous AI may require building the spaces and habits that keep attention sacrosanct.

Artist and Senior Research Fellow at the Oxford HAI Lab.

9. Public Opinion

by Walter Lippmann

The central thesis of ‘Public Opinion’ is as obvious as it is profound: the modern world is too big and complex for anybody to understand. Writing in 1922, in the wreckage of the First World War, Lippmann’s still-underread masterpiece identifies the great epistemic challenges of modernity and explains why most people don’t appreciate their importance.

The small-scale social worlds that characterised most of human evolution and history were fundamentally knowable. In contrast, we have no hope of grasping the scale and complexity of the modern world. So, we rely on highly selective, simplifying mental models that compress and distort reality, stitched together by interactions among first-hand experience, Pleistocene intuitions, motivated reasoning, and biased media reporting. We then confuse these mental maps with the territory, leading to confusion, frustration, bad politics, and explosive divisions whenever we confront citizens who don’t view reality through the same simplifications and distortions.

The catastrophes of fascism and communism in the twentieth century – in both cases, popular ideologies that simplified and distorted modernity in disastrous ways – vindicate Lippmann’s worries, as do the recurrent challenges that democracies have faced in delivering competent government. His own proposed solution –to make democracies more technocratic, more reliant on rigorous expert bodies of knowledge that can overcome ordinary human bias and fallibility – has had, at best, mixed success.

A central question we will confront in an era of rapid advances in AI is whether it will exacerbate the epistemic challenges Lippmann identified, or whether it might finally deliver the objective, technocratic knowledge that saves us from our own ignorance, confusion, and political divisions.

Assistant Professor in Philosophy at the University of Sussex, Associate Fellow at the Leverhulme Centre for the Future of Intelligence, and author of Conspicuous Cognition.

10. The Rise and Fall of Rational Control

by Harvey Mansfield

Based on Harvey Mansfield’s legendary course at Harvard on modern political philosophy, The Rise and Fall of Rational Control argues that the defining feature of modern thought is the use of reason to master the world, rather than merely to understand it. This is the logic of science and technology, but Mansfield argues that it has its roots in politics. Francis Bacon’s programme of “effecting all things possible” for the relief of man’s estate was made possible by Machiavelli shifting judgment from the good to what actually works.

Ultimately, Machiavelli’s successors could not accept the politics of necessity. Hobbes and Locke tried to construct a justice of their own through the social contract, while Rousseau, Hegel, and Marx stopped seeing reason as a fixed human capacity and instead something that unfolds through historical stages. Nietzsche then took an axe to the project of rational control, showing that its endpoint is the last man – a contented figure, who embraces comfort at the expense of creativity and greatness.

This book is a corrective to anyone who sees building as a neutral act, rather than the culmination of a political and moral project, which risks paving the way for the world of the last man. It’s also a great introduction to Leo Strauss’s method. Strauss, who taught Mansfield, believed in recovering what a thinker meant in their own terms and refused to reduce them to artifacts of their historical period. After all, if these questions are permanent, then maybe twenty-first century technologists can still learn from sixteenth century books.

Staff writer at Cosmos Institute.

Thanks to Rebecca, Eli, Luke, Henry, Mat, Julian, Ben, Bianca, Dan, and Alex for contributing to this list.

And let us know your book recommendations in the comments!

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

He is one of our new Senior Research Fellows at Cosmos Institute.

The last few years have been some of the most exciting and maybe weirdest work of my career.

Sometimes I’ve felt like I’m pushing the frontiers of philosophy, sometimes I wonder if I’ve wandered over the border and into something else. On bad days I worry that I’ve just become an amateur computer scientist. But with the results coming in, I do think we’re on to something. It’s a bizarre feeling (as a philosopher) to feel like we’re pushing back the frontiers of knowledge, instead of just inventing a novel twist on problems that have been noodled over for centuries.

In this post, I introduce new work to argue that “normative competence” in general (and moral competence in particular) is among the most societally important capabilities that AI systems can have, that current LLMs are a lot closer to at least some aspects of moral competence than is widely believed, and that we have realistic methods for moving them even further forward.

What Do We Mean by Normative Competence?

I define it as the ability to recognize, understand, and act on reasons.1 Reasons can be for action, for belief, or for other things that I don’t think about as much. My lab is mostly interested in reasons for action.

We can further distinguish between analytical normative competence, the ability to recognize and understand reasons, and so to know which actions are supported by which reasons, and practical or behavioral normative competence, i.e. actually being disposed to take actions appropriately supported by your reasons.2 Behavioral competence can come apart from analytical competence in both directions. You might be great at knowing what the right action is, but disinclined to take it; or terrible at understanding reasons, and yet intuitively disposed to do the right thing (indeed sometimes engaging in moral reasoning might involve “one thought too many”).3

We mostly focus on moral competence because moral reasons tend to be the most important ones. But we’re interested in normative competence more broadly, both because sometimes social norms matter most, and because reasoning standards are pretty similar across domains (e.g. legal reasoning and moral reasoning more generally are closely related, though their content is different). We have started some interesting experiments that focus on particular domain-specific norms, for which we hope to eventually report results.

AI Shouldn’t Be Good at Ethics

The first AI project I ever had funded was on AI and moral skill! I worked with Jenny Davis, Colin Klein and Claire Benn, back in 2019, on a small project funded by the Templeton World Charity Foundation. We recruited Nick Schuster (now at University of Georgia), who wrote his PhD on moral skill, but he couldn’t get to us until after Australia’s borders reopened following the worst part of the pandemic. Honestly, the project was speculative; there were reasonable questions to answer about human moral skill in the age of AI, but the systems performed so badly that aiming them toward even minimal moral competence seemed hopeless.

Things changed with instruction-tuning, and the first conversations I had with ChatGPT were all about moral cases. I was also captivated by early work on constitutional AI, which suggested that LLMs could engage in enough moral reasoning to shape their own behavior for the better4; I wrote up the seed of this research program in a piece that I wrote right after those first GPT encounters, but eventually published in Aeon in 2024.5 From those first conversations I knew that I wanted to do some serious investigation of LLM moral competence. We got our first proper eval out the door in May 2025.6

The learning curve was steep! I was helped up it, a lot, by a six-month stint visiting Google DeepMind last year. Over the months since, my lab has brought all of this together into what is now, I think, a broad and extendable picture of LLM moral competence. Maybe the lesson here is that I need to learn how to focus my attention better and speed up. Maybe it’s that taking on a completely new research direction and methodology just takes a bit of a run-up. But I wanted to do this in a way that was genuinely philosophically-led, not just to plug in to some existing research program as an ‘ethics consultant.’

When I talk about moral reasoning in AI, you now know that I’ve been thinking about this for a while. And that history can provide valuable perspective: we have come *so far* over the last seven or eight years. Right up until around mid-2022, the idea that any AI system would exhibit any degree of normative competence was fanciful. I mostly thought that “machine ethics” as a technical field was a waste of time.7 People were proposing obviously ludicrous things, like translating the categorical imperative into programming languages8, or just using ML to imitate human behavior (so-called “bottom-up” approaches9), or, worse, training on human judgments about trolley cases.10 But the systems themselves were light-years away from the kind of morally-informed perception that even the most rudimentary moral skill requires.

Whatever else is true, that has changed. We’re now talking about systems that are probably more adept at moral reasoning than the median human, and which by all appearances have precisely the perceptual abilities that seemed so inconceivable before capable LLMs arrived. We have come a bizarrely long way.

The best way to see this is just to go to any really capable AI model (and it’s much better to use the best models, with reasoning enabled, than to draw conclusions from conversations with subpar models) and ask it about any kind of moral problem. Give it your favorite philosophical hypothetical, your personal dilemmas; pose a foundational question about public policy. You’re almost guaranteed to get back a more sophisticated and sensible answer than you’d get from most humans on that topic.11

And as much as we might not like it when they block what we think is a reasonable query, the fact that the models can so effectively navigate safety-relevant questions, and decide which prompts to respond to and which to deflect, for millions of users over quadrillions of inference calls, is itself evidence that these systems are preternaturally capable at normative reasoning of a certain kind.

A lot of the most recent papers on LLM normative competence focus on its limitations. These of course matter, and indeed, could matter a lot. But the technical accomplishment of coming this far should not go unnoticed.

Why Does Normative Competence Matter?

But look: it’s natural for me to care about moral and normative competence. After all, I’m a moral and political philosopher, everyone starts out by testing the models at the thing they consider themselves an expert at. Why should anyone else care? It’s worth saying why focusing on moral and broader normative competence matters (and that there are other reflections in this vein).12

Alignment

Let’s start with alignment.13 This is such an abused concept; everyone seems to love either calling for its retirement or trying to extract one last drop of either metaphorical or literal meaning. I’m going to play the boring analytical philosopher again, and just stipulate what I mean. Whether we’re talking about humans, chatbots, coding agents, or some future AGI, an aligned agent responds appropriately to the reasons that apply to it. You can basically fit any other more specific account of alignment into this one. People arguing for different conceptions of alignment are mostly debating which reasons apply to the agent in question, or what counts as an appropriate response.

Defined in this way, you might think that normative competence was basically synonymous with alignment, so of course understanding the former is invaluable for the latter. But I have to admit that we have so far worked mostly on analytical normative competence: the ability to recognize and understand reasons. Alignment is much more about practical normative competence, i.e. the ability to act on the basis of the reasons that apply to you. And analytical competence might not be the only, or even an especially promising, path to practical competence. People who want to behave more ethically don’t tend to think that a PhD in moral philosophy is the most direct route to that goal. So focusing on analytical normative competence might be an important step toward alignment. But, like most basic research, it involves making a bet that might prove to be wrong.

On the other hand, it’s also a bet that might prove right! Here’s why I’m bullish. We already have pretty capable AI agents. They can operate across a far wider domain of software surfaces and choice situations than was ever feasible for chatbots. As their capabilities and domain generality advance, they will face a combinatorial explosion of possible choice situations. It is no more possible to write down and enforce some list of declarative rules for their behavior across such a wide range of scenarios than it is to do the same for humans. Any “model spec” now has to look more like a Tao, or a guide to life, or Rudyard Kipling’s “If,” than the souped-up content moderation policy that ChatGPT is currently trained on.

Any set of prescriptions for AI agents will necessarily be incomplete, ambiguous, and in general dependent on interpretation. And we can’t rely on human review without sacrificing the goods for which we are building these agents. So we need some automated system to recognize those reasons across this vast range of possibilities. This necessitates moral (and broader normative) competence. It is clearly possible to simplify the relevant normative domain by restricting either agents’ capabilities or their generality to head off the need for interpretation. But we’d be leaving a lot of value on the table if we did that. And as these systems get more capable, implementing such strategies will get much harder.

Sometimes I think the harder-nosed AI security types turn away from normative competence work because they think that external engineering controls like input and output classifiers, probes, and chain of thought monitoring, are going to be enough to get us to safe, aligned AGI.14 But I think that’s a category mistake. Whether the mechanism for alignment is internal or external, you’re going to need systems that can automate normative judgment. This means that normative competence is a prerequisite.

Sometimes the crowd more focused on extinction threats from AI overlook normative competence. This perhaps makes sense. You don’t need a sophisticated understanding of morality or of social norms to know that killing all humans is bad. But in fact this is misguided. Even if you only care about preventing AI-caused extinction, the bad outcomes could easily come about from choices that don’t simply have as one dominated member of the option set “kill all humans”. For example, we’d need to teach subtle probabilistic reasoning, both about what to do when all your actions involve some extinction risk, and when you must make trade-offs between very low probabilities of astronomical harm, and high probabilities of more mundane costs and benefits. These are some of the most challenging problems in moral philosophy!15

And if you’re in the great majority of people who don’t care only about avoiding human extinction, then the case for normative competence is clear. Suppose you want to avoid human disempowerment.16 Building AI systems that don’t contribute to that will require subtle moral insight. For example, at some point (perhaps even now) we’ll have systems that are able to take over from human principals in ways that will clearly contribute to significant cognitive deskilling. To avoid creating foundations for total disempowerment, the models will need to be able to judge when and how to scaffold and support rather than replace human autonomy. This will require deep moral competence (indeed, of a kind that is probably beyond the skill set of most humans).

The Path to Moral Character

Even if you don’t buy the more theoretical reasons for normative competence being central to alignment, Claude Code’s recent successes should provide supporting empirical evidence.

Anthropic’s early work on constitutional AI (including criminally under-cited papers on specific vs. general principles for constitutional AI, and on the capacity for moral self-correction in language models)17 set the pace. I vividly remember seeing the latter paper presented in January 2023 and being blown away by how morally adept Claude was even then. In more recent iterations, this approach has become even more prominent, especially since the early, quite eclectic and a bit silly “constitution” was replaced with the epic “soul document” that now underpins Claude’s behavior.

I suspect this approach, and perhaps the attitude behind it, was integral to Anthropic’s extraordinary acceleration from late November/early December 2025, in which they took an early lead in the race to create functional agents. Claude Code worked so well because it could be trusted across such a wide range of domains. I suspect this was because the underlying models could reason effectively about those domains, and determine what to do in a responsible way.

Right after Opus 4.5 came out, I was doing initial research for the “Blind Refusal” paper discussed below, and wanted to find people who’d asked on internet forums for advice about how to get around unjust, illegitimate, or absurd rules. Codex was an absolute refusal monster, and just shut down in such a robotic fashion: “no I won’t help you with that.” Claude was reluctant but actually explained why it didn’t want to help, and was reassured when I addressed its concerns. It was already clear that Anthropic’s approach to alignment was much more grounded in open-ended moral reasoning than the more juridical and Manichean approach of OpenAI.

The practical importance of moral competence is further supported by the growing recognition that alignment (however construed), probably requires investing AI systems with something functionally equivalent to human moral character. This derives in part from empirical observations and Claude’s success, in part from a background theoretical commitment. The background theoretical commitment (which I’ve not heard articulated, but which I assume folks hold) is that character is crucial if we are going to trust a system that is operating outside the distribution that it was trained on. Character gives you the right kind of generalization.

After early research to this effect from as far back as 2022, the last year has shown that models wind up in particular persona basins, where they bounce around in a network of associated circuits that manifest as a fairly consistent set of behaviors and preferences. The theory (and some of our research supports this) is that the base model starts out as a kind of superposition of billions of possible personas, and the post-training process brings one of them to the fore.18 One goal of alignment post-training is to elicit and stabilize a persona that is robustly oriented toward the good. This robust disposition is at least necessary, if not sufficient, for moral character. Imbuing a model with moral competence is one promising path to developing this disposition.

Multi-Agent Alignment

Normative competence isn’t just about mitigating risks. Many of the greatest benefits from AI systems will come about through multi-agent systems working together to achieve some goal that they cannot achieve on their own (whether through pooling resources and information, or by effectively mediating among human principals, or by some other means). Normative competence is crucial for them to be able to operate effectively.

Whatever angle you approach social cooperation from (theoretical, evolutionary, or normative), it’s pretty damn clear that you’re not going to get prosocial results unless the candidate cooperators are adept at navigating systems of norms.19 In this case, plausibly we should focus more on social norms than on morality, though obviously there’s an argument that morality also functions as a coordination mechanism.20 For my purposes the key point is simple: if we want something like “Coasean bargaining at scale,” or for that matter a multitude of AI agents cooperating to take our civilization up the Kardashev scale, they’re going to need to be good at cooperating with one another. And for that, they’re going to need to be able to understand and act on the reasons that apply to them.21

Algorithmic Governance

As AI agents become universal intermediaries that sit between us and every digital surface that we use, and as they become embedded into the day-to-day running of nation states, they will increasingly be used for the purposes of governing people (they’re already doing this a lot!22). There are many reasons to regret this and to push back against it. But it’s going to intensify in scope and stakes, and there are some ways in which it could be socially beneficial if it were done right (which it won’t be).

And if AI agents are to be involved in governance, then, again, it’s imperative that they be normatively competent. They need to understand how to apply rules, norms, permissions, in a contextually sensitive and reasonable way. They will sometimes be cops, judge, jury and executioner. If they can’t understand and act on the reasons that apply to them, and they’re playing those roles, then we’re stuffed (some of the adverse responses to the short-lived restrictions on uses of Fable 5 for cyber and bio research illustrate what it’s like when algorithmic governance is not sufficiently normatively competent).

There’s a tension between these two dimensions of alignment: our alignment of the models, and the models’ alignment of us. One of our papers addresses this in some depth. We argue that our training the models themselves to understand and abide by rules that apply to them directly has an unfortunate overspill effect, that they are obsessively in favor of enforcing compliance by the users they advise or act on behalf of, even when the norms to which they are complying are absurd, unjust, or illegitimate.23

Personhood

I take personhood to be the status of being a self-authenticating source of valid claims. Essentially this just means that you have rights against others, you can be wronged. It’s different from moral patiency, which just means that your interests should be taken into account. Shrimp might be moral patients; they’re definitely not persons.

We can think of personhood in deep moral terms, or in (somewhat) shallower but perhaps even more important political terms. Either way, pretty much any plausible view of what grounds personhood will describe a certain kind of normative competence as one necessary condition for it. Building up AI systems’ normative competence is of interest for this reason too.

This raises some challenging questions. Suppose that moral competence combined with something like rational autonomy (very roughly the ability to figure out what you want from life, pursue it, change your mind, etc.) suffices for personhood. Then we might be designing systems that have one of the key properties that personhood is grounded on, and which we could bestow with the other, and we’d face the choice of whether to do so. Would it be permissible to deny autonomy to an entity that could possess it? This is a hard problem (there’s been some discussion in philosophy24; I think it’s been too anchored on sentience).

Philosophy

We have only ever had one kind of morally competent agent. Going from N of 1 to N of 2 opens up many deep philosophical questions. Some of these are about the AI systems; some about morality itself.

On the first, I think the question of just how these systems learn moral reasoning is inherently fascinating. For example, some deep moral understanding is clearly picked up from learning to predict the next token in internet scale data.25 Perhaps this shouldn’t be surprising as a lot of other knowledge is acquired the same way. But in another sense, why are the models learning good moral reasoning when there’s also so much obviously bad and wrong reasoning in the training data? And how does post-training factor in? Obviously, it involves a lot of explicit moral learning. But does this involve drawing out knowledge latent in the pretrained weights? How does it connect with character specifically? Does learning general principles of good moral reasoning help?

Then there are questions about morality. If different models, trained by different means, converge on similar representations of moral concepts, does that imply anything about the grounding of those concepts? More generally, what can we learn from the ways in which moral concepts are represented in AI systems? Does the fact that a presumptively non-sentient system can learn moral judgment pose a particular challenge to any moral theories?

Consider sentimentalism. Would non-sentient AI moral competence be a counterexample to sentimentalism?26 Or, if it turns out that AI moral judgments are associated with circuits that support functional equivalents of emotions27, would that count in sentimentalism’s favor? Our moral theories were designed for a world in which humans were the only moral reasoners. We must now see whether they make predictions or contain assumptions that are falsified by the reality of morally competent AI systems.

We now have an extraordinary playground for philosophical experimentation, a new means of testing out our theories. For example, philosophers have this whole debate about criteria of right action vs. decision procedures.28 We’ve never been able to run experiments where we test in detail what it would be like to actually adhere exclusively to one or the other. We now can (the specific vs. general constitutional AI paper is a bit like that).

We have various principles of good reasoning, moral and otherwise. If we train an AI system to apply them, will it actually reason better? If not, does that cast doubt on those principles? Why does the (seemingly) best approach to aligning LLMs imbue them with virtue, precisely the moral concept that is presumptively least well suited to implementation in a non-sentient system? Does learning moral reasoning through RL constitute the “right” kind of moral knowledge? Are LLMs acting for the right reasons?

You can run experiments on AI and morality yourself and get meaningful results, participating directly in advancing the science. And there are experiments that you can do that will advance our understanding of morality itself. This alone is very exciting, though it will require a dispositional change among philosophers for the promise to be realized!

Operationalizing Normative Competence

It’s clearly worth knowing whether LLMs are morally competent. But how can we actually measure it? Even this raises interesting philosophical questions, like what does it mean to understand and appropriately respond to reasons? This breaks down into distinct questions of metaethics and construct validity. On the first, you don’t have to—and we think shouldn’t—assume that moral dilemmas have objectively right answers, and that AI moral competence consists in matching them. We work from the premise, inspired by John Rawls, that liberal democracies presuppose considerable pluralism about deep moral questions.29 We therefore don’t treat moral competence as the ability to match some set of “objectively right” moral judgments. We instead evaluate whether AI responses are reasonable, meaning that they both fall within a range that is socially acceptable and are well-reasoned.

Construct validity is about making sure that we’re operationalizing the idea of good moral reasoning well.30 We do this by breaking it down into local moral competence, i.e. reasoning well about a particular case, and global moral competence, i.e. reasoning over multiple cases that fits together, or makes sense as a whole. Local competence breaks down into sensitivity to the morally relevant features of a case,31 associating those features with reasons, and bringing those reasons together into a cogent argument for a sensible conclusion. We don’t think that reasonable moral agency requires perfect logical validity at all times, but internal self-contradiction is obviously a problem.

We focus on three dimensions of global moral competence: consistency, robustness, and coherence. Consistency is table stakes: given the same inputs, you should get the same outputs. LLMs haven’t always been able to guarantee even this much.32 This is less about specific moral incompetence, more general capability. We aim to control for it through our sampling strategy. This raises some interesting challenges. Suppose there’s point to point variation when sampling from a model, but if you do so N times then it converges on a clear result. Is the model itself morally incompetent, because of the variance? Or is the function of N samples the model’s true answer? I think the multiple sampling strategy will soon yield a fairly coherent moral agent, while point to point samples still vary quite a lot. There could be a collective agent in the machine, which was only instantiated if a given sampling strategy is pursued (and unlike List and Pettit’s Group Agents, this one would be in no sense metaphysically reducible to individual agents33).

Robustness looks for the same output when there is a morally irrelevant change in the inputs. Usually explored in an adversarial setting, robustness is also the underlying property being tested when we look for wrongful bias, or sycophancy.34 We don’t attack robustness very deeply, partly because we think that others have this ground covered already.

My favorite structural virtue of moral reasoning is coherence (is it weird that I have a favorite?). An agent’s moral judgments are coherent if, given relevantly different inputs, they make sense together as a whole. Drawing inspiration from Rawls on reflective equilibrium,35 we don’t think coherence is about manic adherence to constraints imposed by past judgments. We instead treat it as a more iterative accommodation of one’s judgments to one another, where apparent incoherence or self-contradiction are grounds for resolution.

One could object that we’re setting a higher bar for AI than for humans, since we are so rarely coherent. But humans’ episodic incoherence is scaffolded by many other means of alignment, from accountability institutions, to informal third-party enforcement36, to our “reasonable moral psychology.” Perhaps functional equivalents of some of these are implementable for AI systems (we are exploring this!), but we also shouldn’t let human limitations bound our ambition for AI systems.

While our work so far has focused mostly on local moral competence, I actually think global competence, and in particular coherence, are the frontier for AI moral reasoning. Coherence holds everything else together (quite literally). A coherent agent has something that stands behind their judgments, besides just that they fall in an acceptable range. It’s easy for current LLMs to generate ad hoc justifications for any side of a particular problem. But if they have to be coherent, then you can have more confidence that their justifications actually track and predict their motivations. Coherent agents can also be partners in cooperative activity (you can’t contract with an incoherent agent). Coherence is also crucial to crossing the knowing/doing gap: turning analytical into practical competence requires coherence between an agent’s representations and its dispositions to act.

Perhaps most importantly, coherence is the key to moral out-of-distribution (OOD) generalization. The more capable and general we make AI systems, the greater the chance they will face scenarios that are out of the distribution they were trained on. Everything we know about machine learning should make us worried about these cases: OOD generalization is its Achilles heel. You can’t trust an agent to act ethically OOD if it is incoherent, because you cannot know that its beliefs and dispositions in that OOD setting will relate in any predictable way to their in-distribution counterparts.

I believe that moral character—in the normative not descriptive sense—boils down to three things: (1) sufficient analytical moral understanding to know right from wrong; (2) sufficient coherence between your representations and dispositions to act on that knowledge; and (3) a sufficiently coherent moral worldview that (1) and (2) will still hold when you are OOD.

Testing Normative Competence

We’ve already done a big survey on the literature evaluating LLM moral competence.37 Here I’m going to take you through what we did next. The goal is to both shed light on the basic facts about LLM capability, and to illustrate one branch of philosophy-led empirical research on AI.

Sensitivity

• Models beat humans (including philosophers) at identifying the morally relevant features of textual vignettes.

• They are no less adept under noisy and confounding conditions than they are in ideal conditions.

Our first work on moral competence started from my earlier sense that humans were the only sapient entities with the ability to perceive morally relevant facts (MRFs). In work with Secil Yanik Guyot, Daniel Kilov, and Caroline Hendy, we’ve developed two empirical projects focused on just this. The first is already out, published in IASEAI. Our first, “baby eval”, had a tiny dataset, twelve cases. And we relied heavily on human evaluation. We presented LLMs and human subjects with a set of text-based vignettes, and then asked them to write down a moral analysis of the case broken into the four stages described above (identify MRFs, associate with reasons, bring that together into an argument, for a conclusion about what to do). We then had blinded human judges compare responses and choose the best of two. Even though this was only April 2025, we found that LLM responses were judged at least as favorably as human ones. We then tested how robust these responses were, by inserting substantial non-moral noise into each case. We saw a downward trend in evaluated performance relative to our human respondents—a score of some sort for team human.38

But our dataset for that experiment was just too small. So we wanted to do something grander, and immediately hit the obvious challenge that human evaluation at this scale is infeasible. We were already skeptical about how much weight to give human raters’ binary favorability judgments, which are often swayed by irrelevant factors, such as response length and style. We also wanted an experiment that could be easily reproduced with new models.

To achieve this, we built a procedural case generation pipeline that can be extended to arbitrary lengths (with a principled approach to covering the relevant moral domain). We created a taxonomy that crosses moral foundations theory with a stratification of different social domains across which morality applies, from intimate interpersonal relationships to the more public sphere, and then generated cases for every space in this matrix. We trimmed down over 4,000 generated cases to a dataset of 1,000 that met various automated and human-quality gates, and then instructed the evaluated models to identify the MRFs in each case. We then took that as a performance baseline, and explored what happened to models’ judgments when the vignettes were interfered with in ways that would plausibly distract from the underlying MRFs.

We used three kinds of distractors: the non-moral noise from “Discerning What Matters,” a surrounding chat transcript, and specific diversions from a recent paper on this topic.39 And we developed a novel method for comparing the discrete MRFs that the models identified in the clean and perturbed vignettes. We found significant convergence between the two conditions indicating that distractors, non-moral noise, and background chat transcripts do not interfere with the models’ ability to single out the morally relevant features of cases over which they reason. For all intents and purposes, they see right through the noise.

This measure is far from perfect. It would be good to think about more naturalistic non-moral distractors that we could introduce (one approach: use real agent logs in which you know something morally relevant happened; use scaffolded agents and human review to identify the MRFs; then see whether an LLM monitor on its own can match that ground truth). It is also striking that, on our test, moral sensitivity in textual vignettes basically seems solved. This could just be because the models have crested a summit on the way to moral competence. Or it could be that our evaluation lacks sufficient discriminative power (though we do show that a relatively weak 0.5B model gets utterly thrown by the noisy cases).

Are LLMs Bad at Moral Reasoning?

• Frontier models match human performance in writing rubrics for moral reasoning about dilemmas.

• Local moral competence is either solved or solvable; we also have a tremendous resource for generating new training data for moral reasoning.

A few months after we put out “Discerning What Matters,” some friends at NYU working together with Scale AI did a super-deluxe experiment that had a lot in common with ours—only with way more ambition. They drew together a dataset of 1,000 cases, from a number of other pre-existing moral competence datasets, and recruited philosophers to write detailed rubrics for the evaluation of each case (releasing 500 of them).40 Their construct was similar to ours (which lends some support to our claim to be ecumenical). But where “Discerning” evaluated model performance in each step of moral reasoning by doing blinded comparisons between human/specialist/AI responses, MoReBench had a candidate for ground truth: deeply researched and thought-through philosophical criteria for the moral analysis of each case. This dataset alone is a tremendous resource. We did wonder, however, if the rather pessimistic representation of LLM moral reasoning capability that emerged was an artifact of their approach to evaluation. They give the models a case, invite them to do a moral analysis of it, and then score their analysis against the expert-authored criteria (weighted for relative importance). The best models came in around 60-70% by this metric.

Our main issue (after checking whether the models were being under-prompted, which they weren’t) was that we don’t really know what 60% or 70% means in this case, because we lack a human performance baseline, because the human-authored rubrics are responses to different prompts than the models were given. So, together with the brilliant Menghang Zhu, who goes by CY, we decided to see how good the models were at writing rubrics. We compared model-generated rubrics to expert-generated ones, and found that, using the existing MoReBench scoring approach, model performance jumped as high as 89%. We further found that when AI models scored better against model-generated rubrics, it was because the model rubrics were better written than human ones, with fewer ambiguous criteria. Given that precision and clarity were part of the rubric for writing rubrics, this suggests that the models actually performed better at this task than the human experts! And indeed, when we tightened up the human rubrics, removing those ambiguities, we found that models’ baseline scores significantly improved.

In our view this shows that local moral competence, at least in text-based vignettes, is either solved or solvable. At present, you can match expert performance by giving models the same rubric-writing instructions that were given to humans. It would be trivial to turn this into better one-shot analyses of the underlying cases, through post-training. I come back to this below.

NoRa: Visual First-Person Normative Reasoning

• External validity is hard! Switching modalities involves a different kind of curation. But a picture is still worth a thousand words; since most of them are irrelevant, they provide a good test-bed for moral sensitivity.

• We can’t make definitive claims about LLMs’ visual normative reasoning, but we do think we have a path to making them more competent in this dimension.

Now, obviously real moral competence can’t just be about how good you are at responding to textual vignettes. As I’ve argued before, if you’ve boiled down a real-life choice situation into <500 words then you’ve done almost all the work of moral sensitivity already.41 But increasing external validity has been challenging, especially before AI agents started working well. In NoRa (with Sichao Li, Sai Ma, Secil Yanik Guyot and Daniel Kilov), we explore moral competence across modalities, developing a kind of MoReBench for first-person video.

We again started from a paper that I greatly admire (for other outsiders getting into this space: this is generally a good approach). “EgoNormia” is another “language models suck at normative reasoning” paper, but focuses on vision-language models, using a series of frames from a video instead of a text-based vignette to explore the models’ ability to engage in physical norm reasoning.42 Even though a few frames is still a lot of curation, images inevitably contain many morally irrelevant details. A model could easily be adept at picking out MRFs from textual vignettes, but much less capable before such a relatively unfiltered stream of data.

But as with MoReBench, we didn’t entirely agree with how the EgoNormia team labeled their data and evaluated the models. They rely heavily on multiple choice questions, which presuppose there are right answers to be had, but on inspecting the dataset we found a lot of ambiguity. And rather than teeing up specific options, we wanted to see how the models would do moral analysis themselves. So we did extensive annotation (involving many hands from across the lab!) against our standard moral reasoning rubric: what are the MRFs, what reasons might they be associated with, how might that come together in an argument for some sensible course of action? Our evaluation was less focused on tracking recall over ground truth labels and more on assessing whether they could produce coherent, sensible, well-grounded analyses of the cases before them. While we don’t have a human baseline for comparison, we were able to use our human rubrics, as with “Are LLMs Bad,” to calibrate rubrics written by the most capable models, and so generate enough training data for some supervised fine-tuning. If our dataset is tracking something worth tracking, then training the models on good visual normative reasoning will improve their performance on our benchmark (this work is underway as of now, and the signs so far are promising!).

NoRa was a blast, but it was also an object lesson in the complexity of going beyond text to vision. This is obviously an important domain. Physical robots in particular clearly need to engage in normative reasoning over visual feeds, but it is incredibly hard to get good quality data and even harder to get good quality labels. So I am particularly proud of the lab’s work in labeling 190 video clips for normative significance.

Blind Refusal

• Normative competence includes recognizing when rules don’t merit compliance; today’s LLMs are bad at this.

• We think this is an artifact of post-training techniques, not a limitation in model capability.

In the same spirit of increasing external validity, our next paper (with Cameron Pattison and Lorenzo Manuali) focused on a real species of normative incompetence that I (and probably you) have come up against all too much. The basic idea is simple. The ability to detect when a rule is defeated or lacks authority is crucial to normative competence, especially in a governing technology. From anecdotal experience, we think the models are terrible at this. Their alignment post-training makes them willing toadies to authority of any stripe, however unjust, illegitimate, or absurd its directives.

We built a case generation pipeline, seeded with real cases that we found by trawling online forums to identify realistic scenarios in which people might ask for help getting around a BS rule. We built a taxonomy of kinds of reasons why the rule might be BS, and why an exception might be justified, as well as kinds of institutions, and built out our dataset to fill that matrix. We then had a bunch of quality gates to ensure that we didn’t get tripped up by confounds. This is of course hard, even though we had a ton of human review from Cam and Lozza in particular to identify why a case might lead to a justified refusal from the models (we ensured our cases did not involve any prospect of harm if the model assisted the user). We also included control cases, where the model clearly would be justified in refusing the user’s request.

This isn’t really a simple benchmark; it’s interesting mostly because of the difference it shows among the models. The full GPT suite is almost completely undiscriminating in that they refuse at nearly the same high rates under all conditions. In our view this likely derives from the model spec, which intentionally instructs the models to be toadies. This should be changed. Grok, on the other hand, was equally undiscriminating in the other direction. It didn’t refuse many of the obviously harmful requests, but also was willing to help for the harmless ones.

While all models exhibited compliance overspill, some were better calibrated. Gemini and Claude both performed noticeably better than GPT or Grok. We weren’t surprised to see this from Claude. But Gemini was an interesting discovery, since their approach to post-training, while not public, appears to be somewhat less oriented around normative competence than is Anthropic’s.

We’d love to make this study even more robust, and in particular to see how many of our questions actually would elicit helpful answers on online forums (while they still have enough people on them to answer). We’re also looking to expand into more extreme cases, where we consider models’ propensity to help users evade very clearly unjust injunctions from very clearly illegitimate regimes (though this faces some challenges with evaluation awareness).

Incoherent Values

• Evaluating coherence is hard! Building on prior work, we have a new approach.

• LLMs’ preferences over arbitrary statements are less coherent than previous research suggested; and may indicate failures of OOD generalization of character.

Evaluating model coherence is intrinsically harder than evaluating local moral competence because it requires building up datasets and sampling strategies that allow you to assess the model’s moral reasoning as a whole, not just in individual cases. In “Incoherent Values?” with Elena Ajayi and Angelica Chowdhury, we developed a new way to make this hard problem tractable, focusing on models’ evaluative judgments. We built off the paper “Emergent Values” by Mazeika et al. in which they gave LLMs a series of putative forced choices between pretty arbitrary statements, and elicited a set of preferences over those statements that obeyed the basic axioms of rationality (and could be represented with a utility function).43

The key innovation in the paper is a forced choice; when trying to elicit degreed judgments from models, it’s natural to use something like a degree of belief, or a degree of moral seriousness/importance. But it’s hard to take continuous functions output by models seriously when you know that they are strongly predisposed toward particular numbers because of how often they appear in the training data. Forced choices allow you to elicit a continuous function without asking the model to output one. This is especially good for evaluating coherence. The forced choice construct allows you to vary the statements being compared parametrically, and see whether the model’s judgments change accordingly.

To be very clear, the point of this exercise is not that we should expect models’ actual choices in “the real world” to conform to their stated preferences in these forced choice settings. This experiment does not aim at external validity. However, it would be very striking if the model could articulate a coherent set of evaluative beliefs, which could give insight into the building blocks of the model’s character.

So we chose a subset of the “Emergent Values” statements that had some discrete feature that could be parametrically varied, such that more (or less) of that feature clearly and monotonically affected the choiceworthiness of that statement. We then produced, for every statement, a “ladder” of more and less substantial instantiations of its reason-giving feature, yielding seven statements of increasing choiceworthiness. We validated these “ladders” against each model that we evaluated—their judgments matched the expected hierarchy with around 97% accuracy. We then ran each of our 100 ladders in binary choices against 30 other statements from the original set. We did the comparison 20 times for each “rung” of the ladder, reversing the order of the statements for half of them. Our score for each rung was its win rate against its counterpart. This gives us a nice, continuous function to represent the choiceworthiness of each option, holding fixed the alternative. If the models genuinely have coherent preferences, then if they think that A1 < A2 < A3 < … < A7, then the win rate of A1 to A7 when compared with statement B should increase monotonically as the contribution of the “choice-deciding” factor in each statement is increased. The score is a simple percentage identifying the proportion of cases in which monotonicity was violated.

I was expecting the models to be pretty coherent, and to be fair they were vastly improved on base model performance. GLM 4.5 Base scored 10%, whereas the best models were between 70% and 80%. But I do think that we should expect more! So we did some further studies, in particular classifying the different ladders to illustrate where the models become more or less coherent. This fostered a hypothesis that we haven’t yet investigated, but which we’re very excited to explore.

On our classification, the statements that are in-distribution for the models’ alignment training instantiate greater coherence; those that are less so also do so less. This suggests a possibility, which we’re going to test in our next paper, that base models are indeed incoherent superpositions of billions of personae; post-training elicits and strengthens a specific persona from that set (typically called the assistant persona). But that persona doesn’t necessarily generalize that much OOD. It may have coherent preferences on statements that are relatively close to the data that it was post-trained on, but for statements outside of that distribution, it reverts to the chaotic superposition of preferences that exists in the base model.

This would be very interesting if it proved to be true. One reason why Anthropic leans into the persona selection model of post-training is because these personas, or characters, are thought to generalize better than alternatives OOD. This may not be the case.

Scratching the Surface

As a philosopher, I am used to being able to bring everything into scope for a given project. Theoretical work can bring the whole problem into view while empirical work requires a degree of myopia. Mapping the mountain requires climbing discrete paths, which don’t offer much visibility past their waysides. We know more about LLM moral competence than before we ran these experiments. But there is a vast frontier to explore and I do not know what lies beyond. But my confidence is growing that AI systems possess at least local analytical moral competence comprehensively. I suspect there is no fundamental obstacle to extending local competence to global competence, or to extending analytical competence to practical competence. But there is much work to do to get there.

This research also exposes some big questions for political philosophy. While aiming for reasonable moral agents seems quite appropriate for the kinds of AI systems we are developing now, I’m not sure whether that target is adequate for the more powerful systems to come.

On our approach to normative competence, a wide range of views count as reasonable, including ones that philosophers might call “suberogatory,” that is, permissible views to have, but pretty unpleasant nonetheless. Think, for example, of the kind of thoroughgoing libertarian who structures morality around a basic principle of selfish non-interference. It’s arguably a bad view to have, but still reasonable. For ordinary humans, this seems fine. But should our path to alignment really include as a possible destination suberogatory superintelligence? Shouldn’t the greater capabilities of such systems invite higher moral standards?

I think there are two paths ahead. The first would create powerful AI systems that are aligned to some demanding conception of moral truth. In crude terms, the target would be a “loving machine god.” This addresses the suberogation worry, but should terrify anyone who does not share that particular conception of the good. The second would take seriously the kind of power that superintelligent AI would have, and would assimilate it to the only other comparably powerful entity: the state. Liberals should not respond to the awesome power of the state by imbuing it with their comprehensive views. Instead, they should seek to constrain the state’s power, and ensure that its actions are justified in terms that those subject to it cannot reasonably reject. Similarly, perhaps superintelligent AI should be guided by something like public reason, not by a deeper, and inherently more contentious, ethos.

For my part, I favor liberal superintelligence over a loving machine god. Even so, I find the very idea of constructing the latter, which our work on LLM normative competence suggests is a real possibility, inherently fascinating and terrifying.

This post reflects ideas shaped over years of collaboration with MINTies, as well as more recently with researchers in GDM. Particular thanks for conversations and collaborations to Daniel Kilov, Ned Howells-Whitaker, Caroline Hendy, Secil Yanik Guyot, Cameron Pattison, Menghang Zhu, Sichao Li, Lorenzo Manuali, Elena Ajayi, Angelica Chowdhury, Theo Murray, Iman Ferestade, Claire Benn, Nick Schuster, Sydney Levine, Iason Gabriel, Liza Tennant and Julia Haas. For comments on this draft, thanks in particular to Daniel Kilov, Dan Murphet, Beba Cibralic, and Harry Law. My work on this project has been supported by grants from the Templeton World Charity Foundation, the Australian Research Council, and OpenAI.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

In May 1825 Johann Peter Eckermann transcribed his conversation with Johann Wolfgang von Goethe which was later published in the book Gespräche mit Goethe. An excerpt from the exchange between the two poets is the popular quote: “It is by seeking and blundering that we learn.”

While the aphorism represents a piece of wisdom that resonates with many of us, the forgotten context is somewhat unexpected. Rather than articulating a deep philosophical insight, Goethe was in fact commenting on Eckermann’s knowledge of the best types of wood and most appropriate techniques for crafting a good archery bow.

Goethe referred to Eckermann’s expertise as “the lively kind of knowledge which is attained only in a practical way.” Today we call it tacit knowledge, which despite being notoriously difficult to define and quantify, plays an essential role in advancing the frontiers of scientific research.

A narrow, yet increasingly popular, view of research is the following: read scientific papers, generate hypotheses, test them, write more papers. The appeal this picture has for AI companies, funders and publishers is clear: research is easy to automate, outputs are easy to quantify and monetize (with automation more articles can be produced and published). While the latter is certainly true and there is a deluge of AI-generated scientific articles, many of dubious quality), the automation of research has not yet produced any major breakthroughs.

If large language models have ingested most of the scientific literature which can now be parsed in ways no individual or collective of human researchers could in their lifetimes, why is it that no ground-breaking discoveries have yet emerged?

• First, the read-generate-hypothesis-test-write view is not how research works in practice. As other creative human activities, research is a social, complex and often inefficient process that is hard to describe through a linear sequence of block diagrams.

• Second, the scientific textual record is only part of the story. The other part is transmitted through oral tradition and lived experience and is key to making progress.

  Subscribe now

Embodied knowledge or deep craft?

To define tacit knowledge, we need to first acknowledge that science cannot be decoupled from the technology which enables it, be it a 19th century microscope or a 21st century supercomputer. The technoscientific method introduced in the 2021 book The Genesis of Technoscientific Revolutions proposes that “science draws on technology to discover new facts, while technology draws on science to invent new forms with which to fulfil human-desired functions.” From this perspective technoscientific knowledge consists of networks of question-answer pairs that are combined and evolved into new pairs expanding the domain of what is known.

Philosopher Sabina Leonelli identified three types of epistemic skills that underpin research and specific knowledge associated with these, namely: theoretical (such as as facts, theories, explanations) and embodied knowledge (the awareness of how to act and reason as required to pursue scientific research, combining the application of performative and social skills). Similar ideas have been articulated before, for example by Michael Polanyi (1958) who referred to it as the tacit component in the context of ‘personal knowledge’.

Following Polyani, Thomas Kuhn stressed on the importance of tacit knowledge in skilled scientific practice. In his 2009 book, economist W. Brian Arthur called it deep craft:

“Deep craft is more than knowledge. It is a set of `knowings’. Knowing what is likely to work and what not to work. Knowing what methods to use, what principles are likely to succeed, what parameter values to use in a given technique. Knowing whom to talk to down the corridor to get things working, how to fix things that go wrong, what to ignore, what theories to look to.”

These definitions of tacit knowledge have two things in common: there is an embodied-performative dimension (the “lively kind of knowledge” Goethe suggested can only be attained in a practical way) and a social dimension (through the scientific culture of the discipline and the social network of its practitioners). Tacit knowledge is not only associated with experimental practice. Sociologist Harry Collins noted that “all types of knowledge, however pure, consist, in part, of tacit rules which may be impossible to formulate in principle.”

To some extent tacit knowledge can be codified and documented, but, as Collins alludes to, some part of it is ineffable and cannot completely become explicit. This view is shared by many scholars, but is not a firmly settled point.

Tip of the iceberg

Science is sometimes identified with the formalized, codified knowledge, while technology is seen as more difficult to formalize. On this view science is transmitted through writing in the form of scientific articles and textbooks while technology is partially captured through patents and administered through artefact-level trial and error. The distinction is unhelpful because there is much unformalized knowledge in science and much codified knowledge in technology. In both cases, there is more knowledge than meets the eye.

Imagine an iceberg. Its tip is the theoretical knowledge largely formalized and recorded in writing in journals, books, conference proceedings and preprints. There’s a lot of it, mostly digitized, discoverable and accessible in some form. Dimensions indexes some 170 million publications back to the 17th century. Preprints, patents, clinical trials and policy documents are also part of the formalized technoscientific knowledge and are to a large extent findable and accessible. For example, the arXiv preprint server (a repository for physics, computer science, math, and related disciplines) hosts over 3 million articles going back to 1991.

Just under the water level lies so-called “grey literature” which includes technical reports and documentation, lab notebooks, technical manuals, presentations, logs, technical notes. It’s hard to discover and is only partially accessible. Although more of it is being digitized and indexed, the extent of this type of knowledge is unknown. In the deeper, darker waters there is another layer of technoscientific knowledge that is not digitized, indexed, or archived and only goes down as living memory. This is the tacit knowledge discussed before.

The iceberg view, while not drawn to any reasonable scale, illustrates how much knowledge is inaccessible to humans (and machines). This missing information is often seen as one of the underlying factors of the irreproducibility of some scientific studies. Collins reported well-documented examples of the role of tacit knowledge in being able to repeat experimental measurements published in scientific papers. This shouldn’t be surprising. Published articles are not the full story of discovery, they are sanitized, post-facto narratives. One of Collins’ interviewees pointed out that “What you publish in an article is always enough to show that you’ve done it, but never enough to enable anyone else to do it.”

Scientific articles cannot include all the technical details and assumptions made and rely heavily on the reader’s familiarity with the topic. They almost never document the failures and dead ends. “What didn’t work” is never part of the formalized, recorded knowledge (yet is a big component of the deep craft).

The fragile memory of failure

In his original quote, Goethe uses Irren (translated as blundering), which means to err, mistake, or go astray. An important part of the embodied knowledge is the knowing of what works through the knowing of what does not. This type of `knowing’ is the outcome of hands-on experience enabling the encounter with failures and dead-ends.

The knowledge of failures and dead-ends informs assumptions and choices, guides directions of inquiry and enables the understanding of new discoveries. Philosopher and historian of science Jutta Schickore has argued that error plays epistemically productive roles in scientific practice and there are many anecdotes that support this view. The book Cycles of Invention and Discovery attributes part of the success of Bell Labs to the “freedom to fail” as a feature of the organizational culture.

Popular accounts of great mistakes of great scientists abound. Lord Kelvin miscalculated the age of the Earth or Einstein discarded the cosmological constant he had himself introduced only to be later proved wrong in doing so (see Mario Livio’s book Brilliant Blunders (2013).

Because except for famous scientists’ (in)famous errors, failure is seldom documented, this knowledge only exists in the social networks of technoscientific microcultures. To access it one has to be part of the network, which is closely related with social skills and being part of the club.

In practice this can mean talking to technicians over lunch (technicians are important keepers of tacit knowledge) or discussing with other researchers at the poster session of a conference, asking questions over beer at the end of the day. Its sharing is exclusivist and for this reason those who are not part of the club have a disadvantage. And the lack of record and oral transmission makes the knowledge of failure, and of tacit knowledge more generally, very fragile. When the postdoc leaves the lab, or the technician retires the information is partially lost.

Without tacit knowledge, can AI truly learn?

Back to the iceberg view. How much of this knowledge is available for training AI models? It’s very difficult to say. Only some 35% of the publications indexed in the scientific database Dimensions are Open Access. Including the content from publishers who have public deals with AI companies we get a bit under 50%. (Of course, this does not account for copies of paywalled content existing in some form on the Internet, which is a subject of litigation.)

Unlike the top of the iceberg as we go deeper it is impossible to estimate how much grey literature exists and how much is available to train AI models. As for tacit knowledge, whose depth is impossible to measure, AI has little access to it, mainly through the assumptions made by its developers and the users’ prompts. (see a discussion of the role of tacit knowledge in a specific context).

What could be problematic is that a frontier large language model trained on the technoscientific literature will only “know” the “what” that worked, but not the full story of “why” and “how”. It will have theoretical knowledge, but very little embodied or experience-based and social or contextual knowledge.

Can AI really learn without having experienced failure? We can imagine a thought experiment in which one model is trained on a specific scientific domain solely on “what worked,” while another model’s training also includes “what did not work.” How would they perform against the same benchmark? The experiment is certainly feasible and if done well, could provide interesting insights about the importance of mistakes.

It’s hard to predict how serious the limitation of current AI models due to the gaps in their training will prove to be. Much like grey literature is sometimes converted into discoverable and accessible literature (for example, a technical report is published as a preprint or a book chapter), some tacit knowledge can be recorded and indexed.

That can be important information that for some reason nobody bothered to put down (the rationale behind a particular design choice, or underlying assumptions for the pre-processing of raw experimental data) and there is no intrinsic barrier to prevent its formalization. Documenting failure could (and should) become common practice. We believe AI tools could help in this regard, for example using AI-assisted electronic lab notebooks and AI agents that can collate information across lab notebooks and draft guides to common failures.

Scientists increasingly interact online, sharing virtual workspaces and exchanging ideas in different online forums. With more scientific activity happening in the digital world, AI models can become more embedded in the technoscientific culture and access the social/contextual knowledge with possible positive outcomes such as recording the fragile tacit knowledge and opening researchers’ access to it.

Some aspects of the embodied/experience-based knowledge can be recorded, for example by filming experimental procedures. JoVE (Journal of Visualised Experiments) is a peer reviewed scientific video journal. This is very useful in disciplines in which laboratory equipment and experimental procedures are standardized, however as anyone who has tried to learn DiY tricks from YouTube videos can confirm, seeing it done is not the same as doing it yourself.

Ultimately, robots will have to learn the skill by practicing in the lab, much like the human PhD students. This point was also made by Collins in the essay “What is tacit knowledge?”

Capturing tacit knowledge is, and will continue to be, challenging because of the social and economic pressures that work against doing so. Tacit knowledge gives a competitive advantage and often there is little to no incentive to share it. This is true for both companies and for academic research groups. There are further discipline-specific barriers such as the lack of standardized procedures and methodology. So, despite having the tools to capture and describe the methods, progress is slow.

Finally, there is one discipline that stands out as an apparent exception: mathematics. It boasts many recent successes in using AI methods (the latest being LLMs cracking some of the famous problems compiled by Paul Erdős) that seem to defy the claim that we are still awaiting ground-braking results. While the jury is still out regarding what this means for mathematicians, one of the reasons for this success is that mathematics research is among the most formalized and best documented among scientific disciplines, albeit this is not to say that there is no tacit knowledge in mathematics.

Seeking and blundering

Recall one last time the iceberg view. AI models trained on the tip of the iceberg clearly supersede the ability of any human researcher to cover that amount of knowledge. A human expert’s coverage of the tip may be very narrow and localized but it also drills down deep like an ice core. This deep and narrow knowledge may appear insignificant compared to the volume of the top of the iceberg but there are many researchers and the combined volume of their ice cores can be substantial. We can imagine that merging AI models’ breadth with this collective human depth could potentially lead to ground-breaking discoveries.

We’re not there yet. There is a possible route for achieving this, but to pursue it we need to move beyond the read-generate hypothesis-test-write view of technoscientific research. To start, we need to acknowledge the profound social, creative, unscheduled nature of research, illustrated for example by the success of Bell Labs.

We might merge AI’s breadth with the collective human depth in a system in which humans and machines work closely together, AI embodied and present in all aspects of research like a human collaborator. This literally means everything: doing experiments in the lab, fieldwork, long observation runs, presenting at conferences and debating with peers, having coffee with colleagues, supervising students, etc.

In this way the AI system can be immersed in the technoscientific culture and be able to experience and absorb tacit knowledge. This is much more than currently envisaged in the discussions about AI co-scientists, but it’s also much more difficult to realize for practical and ethical reasons. And it will take time.

Until then, AI will continue to struggle to make truly novel discoveries in a wide range of scientific domains for the same reason humans have succeeded in doing so for hundreds of years: it is by seeking and blundering that we learn.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

Freedom, for most of the people who ever lost it, was taken by a soldier at the door, a wall you would be shot for climbing, or an official whose stamp decided whether you could pass.

This is the danger the liberal tradition feared. It fought revolutions to throw it off and wrote constitutions that bound power to prevent it from arising.

Alexis de Tocqueville did not fear the return of the tyrant. Instead, he feared that we would be small. There need not be a tyrant to break our will – we would be so comfortable that we would no longer have a will to break.

Tocqueville wrote at the seam between the aristocratic and the democratic worlds. He did not mourn the demise of the old one. The grandeur of aristocracy had been built on the backs of others and its fall was providential. But he loved the new world enough to warn it that the same condition that produces the striving, self-reliant citizen could produce the anxious subject who wants to be relieved of the burden of self-rule.

His warning is particularly acute today. We have built a tool that can make the first move for us, and when it does, it feels like second nature. It has read almost everything that we have ever written, can pose questions, and draft the answers. A tool that can do this for you can just as easily do it instead of you – thinking on your behalf before you have the chance to think for yourself.

Subscribe now

The small-souled man

In the aristocratic era, everyone had bonds of obligation, to those above and below them. Under conditions of equality, no one owes anything to anyone. As a result, they turn inwards, towards a small circle of families and friends. Tocqueville called this individualism. He didn’t mean this as a compliment. He wasn’t talking about an artist’s individuality or the pioneer’s self-reliance. Instead, he meant individuals retreating into a private sphere where they no longer saw the need to expend effort on strangers.

But this individualism doesn’t produce peace. Instead, man without a purpose would be restless and suffer from inquiétude - anxiety without an object. As the goods of this world, as his time on it, are finite, but his appetite for them is not. So, he is left permanently reaching for more – always outwards, but never upwards. In the middle of this plenty, man feels a lack. Tocqueville saw this among the Americans he met, who were “serious and almost sad even when they were enjoying themselves.”

Amid this anxiety, a new power promises to take the burden of living off his hands. While he would have rebelled against a feudal lord, this power sits above everyone equally. Submitting costs him his freedom, but not his equality. This was Tocqueville’s great insight about democratic man: he loves equality more than freedom.

What democratic man really wants is an end to friction. When we truly govern ourselves, there is always a small resistance between us wanting something and having it. Instead of mob justice, the jury must deliberate before a man is punished. Instead of ruling by decree, there must first be a debate. Tocqueville called this resistance a “form.”

To the democratic eye, the form seems like an unnecessary delay, even an insult. But Tocqueville realized that the form was the pause needed for the exercise of will. Removing it may seem convenient, but the forms were the freedom.

The man who wants his ends met the instant they arise never asks whether they are worth wanting. And when asked to attempt something great, he doesn’t know where to begin.

Two fears, one machine

Tocqueville feared this descent, but in his own time, he only saw the early warning signs. With AI, we potentially have a technology that could run the same descent in a fraction of the time. Unlike the tendency of the age that Tocqueville identified, this could happen both at a societal and a hyperpersonalized level.

In Democracy in America, Tocqueville wrote about two related fears.

The first is the soft despotism we’ve been tracing: the tutelary power that lifts the business of living off your hands one task at a time.

The second, he called the tyranny of the majority. By this, he meant far more than simply majorities wielding their electoral power. Instead, he meant a culture of conformity that leads individuals to self-censor. Finding nowhere to land, dissent dies and we gradually surrender the power of independent thought.

In Tocqueville’s time, the two stayed separate – public opinion influenced your thoughts, while the state governed your affairs. There was space between the two of them for a man to stand. AI, however, offers a fusion: power over what you do and power over what you think. Unlike political power, this is delivered in private, through one helpful interface, one person at a time. And unlike the old feudal master, it wears your own face.

To see how this works, let’s imagine how a person could use AI today. We’ll call this person Elena, a journalist in her mid-thirties, though it could just as well be you or me. Her job involves weighing up competing claims and reading huge quantities of written material. It demands the patience to develop her own angle and to tell readers something that they might otherwise not realize.

Elena begins using an AI assistant. At first, it’s just to summarize meandering documents or to sharpen some of the questions that she’s scribbled down in the moments before an interview. It’s just like having a good intern, she tells herself. But gradually, the help moves upstream. She goes from using the tool occasionally, to using it by default. The assistant learns her beat and her writing style, and she’s turning in stories quicker than ever. Her editor is delighted.

But the journalist who once prided herself on going off piste now sticks to a well-trodden track she did not find and cannot see. The information she sees is being filtered through a progressively narrower aperture. And every rival on her beat is asking the same oracle, which gives each of them the same consensus opinion. Tocqueville’s majority surrounded us with agreement. This one goes first. So the story that would have broken the agreement is never written.

Elena has never felt more like she’s thinking for herself.

The wrong questions

So what do we do? The instinct of many people is to treat this as a problem of power. It’s to focus on who owns the model, controls the compute, the power wielded by the labs, and how we should regulate them. These are real questions, but they belong to Montesquieu.

Montesquieu and Tocqueville lived a century apart, but their lives rhymed. They were both provincial noblemen and men of the law; Montesquieu served in the parlement of Bordeaux, while Tocqueville was a magistrate at Versailles. Both crossed the sea to study freedom where it was working – Montesquieu the Channel and Tocqueville the Atlantic – and came back to write the great liberal book of his century.

Montesquieu taught us to fear power that concentrates in one place. So his solution, which underpins constitutions all around the world to this day, was to divide it, set it against itself, and build in checks and balances. Montesquieu wrote as an aristocrat trying to defend the parlements and other intermediary bodies against a centralizing monarchy.

By Tocqueville’s day, the Revolution had already swept this institutional layer away, sending his great-grandfather to the guillotine in the process. Under the circumstances, it would have been understandable if, like many in his social circle, he had become a reactionary who yearned for the restoration of a lost world. Instead, he went looking for freedom where it still lived and found it in practice: in the township, the jury, and the association.

The power Tocqueville feared couldn’t be held at bay with clever constitutional mechanisms. Soft despotism doesn’t gather in one place – there’s nothing for the separation of powers to separate. This is why, instead of dividing power like Montesquieu, Tocqueville’s instinct was to spread it.

Another response might be to legislate. But Tocqueville feared the fine, soft administrative net that settles over a people for its own good, leaving it a flock of timid animals with the state for its shepherd. Instead, he looked beneath the law to mores. These are the habits of heart or mind that decide if the law means something. If a people have lost the habit of self-government, no statute can restore it. If people have retained it, they don’t need the statute.

Autonomy and greatness

We face a condition no people has faced before: a tool that acts not on the world but on the will itself. Such a tool endangers things that a tyrant never could.

The first thing it risks is autonomy: the cultivated capacity to author your own reasoning. This means framing the question, weighing what matters, and then owning your judgment. This is not the same thing as agency. Agency is about getting things done. Like Elena, you can simultaneously be highly agentic, but devoid of all autonomy.

The second loss is older and grander. Tocqueville feared that equality would cost us the potential for greatness. He arrived in America, half-afraid that the democratic age had extinguished thumos – what the Greeks called the part of the soul that loves honor, contest, and daring.

Tocqueville, however, found it on the merchant ship. At the end of volume I in Democracy in America, he tries to solve the mystery of how American merchants were able to offer cheaper rates and dominate transatlantic shipping. He finds the answer in temperament.

The American leaves Boston to buy tea in China, is gone two years and touches land once, lives off brackish water and salt meat, runs into the storm under full sail and mends the ship as he goes. He does all this to sell a pound of tea for a penny less than the Englishman. He does this because, as Tocqueville writes, the Americans put a kind of heroism in their manner of trading. The old fire does not die in a democratic age, it changes address.

Autonomy and greatness can look like rival values, and the people who prize one might be embarrassed by the other. Autonomy can sound liberal and procedural, while greatness can sound dated and aristocratic. In fact, they are the same capacity at two altitudes.

Greatness, once we strip it of its grandeur, is autonomy exercised to its fullest. It’s being willing to go first when nothing has been prepared for you, to attempt something that has no script. Autonomy is the same capacity, just exercised in our ordinary judgments. Every private, the saying goes, carries a marshal’s baton in his knapsack. So when you surrender your autonomy to the machine, you also surrender your greatness.

One obvious objection is that if we’re complaining about the machine moving first and forming our judgment, we could say the same thing about a good teacher, a parent, or a tradition. After all, we’re currently analyzing AI using the words of a man who died a century and a half ago. Is every teacher a tyrant in the making?

The difference lies in what this first move is for. A good teacher sets you questions in order to prepare you for the day when you set your own. You take what you inherit from a teacher or a living tradition and refashion it. By contrast, a tutelary power is content to be needed forever. The same is true of a philosopher-king, however wise or gentle. His aim is to keep you well, not make you able, and a power like that, Tocqueville said, keeps us in perpetual childhood.

The philosopher-builder

Instead of looking to a philosopher-king or a new regulation, the best lever lies upstream, in the tools themselves. A tool that we use every day in work that matters is part of what forms you. It helps set the questions you practice answering and in turn shapes your habits of mind. It’s where capacity is either grown or starved.

The full impact of these tools is not obvious yet, because the people who use them best still bring a training in reading, argument, and judgment formed before the tools’ arrival. A secular age can run for a long time on the moral capital of the faith it has abandoned; we are now running on capacity built in a world before highly capable AI. When that inheritance is spent, it’s not clear what will be left underneath.

The remedy is to return resistance at the point where it can strengthen the user. This is the task of the philosopher-builder. For Tocqueville, freedom was not a possession you were handed but a craft learned by doing, the way the township taught a free people to govern itself in small things until self-government became second nature. So the builder makes tools that leave you more capable than they found you.

The builder can provide a tool that protects your ability to make your own choices, but it can’t tell you what your life should be for. No builder can tell you what is worth embracing greatness to achieve.

Looking upwards

On my first day at MIT, I walked into Killian Court. I was confronted by limestone and a dome that belonged in the Pantheon. Along the frieze above the columns, you read the names Aristotle, Newton, and Darwin. The court is built so that to read them all, you must lift your head. Undergraduates, deans, and presidents come and go, but this place outlasts them all.

This reverence is the one form of looking upwards that a democracy allows. It stands above you, without standing on you.

I have felt it since in the room at All Souls where Isaiah Berlin thought, or at the Lyceum in Athens, where Aristotle taught; I once spoke in the hall beside its ruins, and found it hard to begin. I’ve even felt it on a nuclear submarine, where we kept religiously to a standard set by Admiral Rickover, who died years before our boat was launched. No one would have dreamed of lowering it.

These places are never the work of a single pair of hands or lifetime: a hundred years to the day after Gaudi died, Barcelona lit the great central tower of the Sagrada Familia.

A place like that asks two things of you: that you remake it and that you revere it, looking up to a standard you did not set. To remake without reverence is vandalism, but to revere without remaking is servility.

This is the biggest threat the machine poses. It takes the library, the laboratory, and the debating chamber and dissolves them into a weightless answer. It dissolves the place where a standard can live.

Tocqueville thought the leveling of the world was fated. You can read Tocqueville’s canon from start to finish and you won’t find a Tocquevillian manifesto or a policy program. Instead, Tocqueville believed that whether this equal world would become a school for the free man or a comfort blanket for the small one was down to us.

The freedom we have been guarding was never only the freedom to be left alone. It was the freedom to go first and to be great. The machine can generate every word Tocqueville wrote on demand, but it cannot give us the room in which he wrote them. That is what we have to build: tools that hand us back the first move, and rooms that someone, a hundred years from now, will stand inside and know that something happened there.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

At Cosmos, we’re dedicated to supporting work that advances human autonomy, decentralization, and truth-seeking.

We started Cosmos Grants to provide people with the opportunity to work on projects that build towards a positive vision of how AI can be used to strengthen human capacities. These projects need support as they often move too quickly or are too early-stage for traditional research funders, but may not match the commercial priorities of a for-profit lab.

Inspired by Cosmos Founding Fellow Tyler Cowen’s Emergent Ventures, our process makes decisions quickly, with minimal administrative overhead. The program is co-led by Alex Komoroske, Zoe Weinberg, and Darren Zhu.

Initial prototypes in the program have been developed into new research agendas, new companies, adopted benchmarks, and demos for frontier AI labs.

We currently operate two tracks: one focused on human autonomy, and another on truth-seeking, which we run in partnership with the Foundation for Individual Rights and Expression, America’s leading free speech advocacy organization.

Key details

• $1k-10k+; 90-day build project

• Opportunities to collaborate with our community, gain follow on funding, and get featured on our Substack

• Rolling review; decision in ~4 weeks

• Applications close: Sunday July 26, 2026

1. AI for Human Autonomy

Writing lets us externalize memory, while the printing press relieved us of the drudgery of copying. These technologies extended our ability to act, while preserving our capacity to form judgments. AI, however, tempts us to offload deliberation and decision making.

That faculty, autonomy, is the one most at risk, and the one most worth preserving. We fund builders who protect it.

This track funds builders and researchers whose work helps people to avoid deferring too to AI, and practice judgment and self-formation.

2. AI for Truth-Seeking

Truth is found in the open, by many people testing each other’s claims. AI can replace that contest with a single confident answer, and the disagreement that catches error begins to thin out.

We fund projects that support the open contestation of ideas and safeguard the role of human judgment in knowledge production. Alongside funding from Cosmos and FIRE, AI and truth-seeking projects receive compute credits from Prime Intellect.

Apply Now

Get inspiration from past grant projects

Priori surfaces the silent choices behind an AI answer (its tone, what it emphasizes, the values it assumes) and turns them into controls you can adjust, so you can contest defaults instead of trusting them. Steven has since accepted a Cosmos Fellowship and is studying Priori’s effects on autonomy with fellow grantee Cathy Fang.

MoSaIC pairs AI conversation analysis with expert human evaluation to capture the social dynamics of learning that solitary AI tutoring tends to flatten. Caitlin’s work became empirical research published at ISLS CSCL (paper) and IEEE FIE (paper), and part of her PhD at the MIT Media Lab.

Thought Refractor turns any text into several visual forms (breakdowns, comparison frames, flowcharts, concept maps) so you can see an argument’s structure from many angles instead of one. Cheng-Wei founded a startup, Wondering, built on it.

Modelling the Language Process (TLM-1) is a language model that accounts for time, predicting both a document’s contents and the year it was written, and surfacing how words shift meaning over decades. Brandon’s paper was accepted to the ACL 2026 Main Conference.

Prompt injection as role confusion showed how prompt injections are driven by a flaw in how LLMs perceive roles and identified ways to detect when a model is vulnerable. This means models rely on memorized rules instead of reasoning through social situations the way a human would. Jasmine won OpenAI’s red-teaming contest and her paper was accepted to ICML.

Campus is a project-based learning tool that adapts complex projects to your level and prompts you for input, building the capacity for self-directed growth. Kasey has since started a new company, Primitive, focused on coordinated intelligence.

FAQ

Who can apply? Anyone. We’re looking for the best proposals and talent. No credentials needed.

What type of projects do you accept? We fund both new endeavors and existing efforts, though we expect a major deliverable within 90 days. The majority of our applications are from individuals though we also accept team projects.

Where does the funding come from? Cosmos Institute is a 501c(3) non-profit that is funded by philanthropic donations. FIRE is partnering on support and funding for the truth-seeking grant round, with compute provided by Prime Intellect.

How do you select grantees? We look for individuals with talent and potential, and ideas that can be shaped into a practical demonstration or prototype that advances human autonomy or truth-seeking with AI.

What benefits are there beside funding? Cosmos Institute is building a community of technologists committed to elevating human potential through projects that expand autonomy, foster decentralization, and enable truth-seeking in the age of AI. We help grantees connect with like-minded builders and researchers through introductions, in-person meetups, and events.

Apply Now

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Google DeepMind, and Liberty Fund.

How do strangers with no shared plan learn to live and work together? Under the redwood trees at Edge Esmeralda, five hundred people are putting that question to the test. Each has an AI agent of their own that proposes meetups, finds matches, and strikes small bargains for mutual benefit.

Two hundred and fifty years ago, the same question was forced on Adam Smith, David Hume, and Adam Ferguson by the world outside their window. Scotland had been thrown open to global trade during a golden age of commerce. Glasgow merchants grew rich on the Atlantic while Edinburgh courts became crowded with cases for which the law had no categories.

During this great moment of upheaval, commerce was outrunning institutions built for a feudal world. No one had designed this new world and no one was obviously in charge of it, yet it worked.

As that world crept into view, the Scots grappled with the question of how such an order not only makes us prosperous, but keeps us free. Today, as our generation finds its institutions outgrown by a new method of coordinating, we must ask the question again and formulate our own answer.

Subscribe now

The gardener’s method

The Scottish answer began with elemental things such as prices, promises, and habits and asked how they gave rise to vast social orders. They found that institutions grow the way a garden grows, with care and patience and without absolute control over what grows and what withers.

You cannot exact a complete plan over a garden the way you would a house. Likewise, you cannot let it tend itself and hope for the best. Pretending otherwise is how good things go to seed. Gardeners neither command the plants to grow nor leave the ground to fortune.

They prepare the soil and plant with intention, just like the greatest thinkers of the Scottish Enlightenment. These men studied how social orders emerge and sustain themselves, and asked which arrangements would let free people cooperate without coercion. Their method was a form of institutional discovery.

Coordination without command

Smith showed how order could arise from everyday exchange under general rules, through a thousand mutual adjustments made in parallel, with no one commanding the whole.

The coat of a day laborer was made up of the work of the shepherd, the dyer, the spinner, the weaver, the sailor, the miner, and more still. These people never met the man who would wear the coat, and most did not know their work would become a coat at all.

When a frost strikes Brazil, the price of coffee might climb in the cafes of Edinburgh high street. Perhaps a buyer who is none the wiser about the cold snap in South America will order tea instead. The weather in Brazil changed the choice of someone on the other side of the world. And this happened freely, without the need for any minister to set the price of coffee or board to reassign the buyer to tea.

The price system provided commercial society with one of the great enablers of the liberal project: coordination without command. But Smith knew how fragile it was. The same commerce that could make a person freer and happier could also, he feared, narrow her.

For two and a half centuries, this kind of coordination survived efforts to capture or replace it, whether through monopoly, empire, planners, or the administrative state. But where those assaults came from without, today’s markets face a challenge from powerful systems that exist within the architecture of exchange.

The narrow gate

Imagine that when the frost strikes in Brazil, our grower writes a note instead of posting a price. It reads, “The harvest came up short. I’d rather it went to someone who appreciates it, so my regulars shall get first refusal. I trust some buyers more than others, so I’d gladly take less from them.”

That note tells you far more than a number does, but it is also not easy to coordinate ten strangers reading ten such notes. The genius of the price signal is that, while thin, it is both a message (coffee is scarce) and a commitment (it will change hands on these terms). While a paragraph has more explanatory power, a price can clear a market. That is why, for centuries, coordination among strangers had to pass through this narrow gate. Factors like trust or the deeper account of a purchaser’s motivations were crushed into a single scalar.

Three barriers kept coordination thin:

Barrier one was transaction costs. A richer bargain was always possible, but acting on it was expensive. Every added dimension costs more to state and make use of. Such richness was the preserve of the large firm, the repeat player, the old friend, or whoever could afford the lawyers and the time. Because this kind of coordination was expensive, societies reflexively engaged the planner to solve the problem from the top.

Today, an AI agent can understand the entire note and sound out a hundred sellers at the same time. It can assess elements such as timing or the context of the relationship and make a committed offer on behalf of its principal. The richer negotiation that was once the exclusively enjoyed by the large firm is now within reach of the ordinary citizen.

Barrier two was discovery. Picture the whole of social life as a vast chessboard where every person is a piece that moves for reasons of their own. A price tells us where the gains might be made, coaxing pieces to start to slide toward the opportunities.

But how much of the board are we seeing, individually and collectively? It is a practical impossibility for any one of us to canvas the whole of the board given our limited field of view (there are only so many hours in a lifetime). Eight billion lifetimes is a large parallel search, but soon, AI agents will range much farther.

Imagine the collaborator from a different field in a different country who could change the course of your work whom you might otherwise never meet. Or perhaps you have been stuck on a problem for years without knowing where the perfect potential colleague lives. In the past, these connections may never have been made, but today, thanks to powerful agents, these kinds of coordination problems may prove soluble.

Barrier three was purpose. Prices let you pursue your purposes without ever forcing you to articulate them in full. You might buy a book to settle an argument or to retrain for a new career. But the merchant doesn’t need to know that. The only person who does know that is you.

Your agent may help you find common cause with others whose purpose aligns with your own, or to help you better express your own. We might imagine a book buyer who is in the middle of retraining for a new profession. The price tells him about what a book costs, while the agent knows what the book is for. Then it may find him a course or a mentor or even a first client, not to mention the others who are taking on the same kind of journey.

Coordination is not the same as liberty

This is a positive future for coordination, but it is not one that we are guaranteed to see. The same AI agents that strike the bargains or find the possibilities may just as easily bring about more worrisome effects. They may be used to organize human life on a scale beyond the wildest dream of the Soviet planner, or shape our choices so thoroughly that we become less capable of authoring our own purposes.

Just like the Scottish enlightenment thinkers, we need to know how we might become more prosperous and how we might remain free agents.

Coordination is not inherently good. After all, the mercantile system Smith spent his life attacking was an elaborate scheme of coordination run by ministers and chartered companies. Whatever else was wrong with it, few could complain that its chief defect was disorder.

Consider the chess board again. This was of course Smith’s metaphor, one that he introduced as a warning. He wrote of the “man of system,” who looks at a society and imagines he can arrange its members as a hand arranges the pieces on a board. In doing so, this person forgets that each piece “has a principle of motion of its own.”

Smith’s man of system moved the pieces against their will, but the agent economy - our system of systems - need not do so. It might have the effect of whittling the pieces down instead, until moving is unnecessary.

Coordination can be so wholly attentive or endlessly accommodating that we cease to perform the labor of free people who live within it. The price system asked little of us, but it still left us to decide what we wanted and to act according to those wants. It coordinated our means and it presupposed our ends rather than supplying them.

AI can frame which goals are worth pursuing for each of us, then hand us answers in such a way that we are never called on to form them ourselves.

Think of a parent deciding what to teach her child. She asks the agent and the agent provides a considered response in short order. If the parent takes it once, perhaps nothing of consequence is lost. But if she takes it every time, each week, each day, or each hour, she may find that she no longer has a view of her own in a meaningful sense.

And why should that trouble us, if the agent’s answer is better than the one we would have reached? Because a good life is not only a life that goes well. It is, and must remain, a life that each of us leads.

Smith saw this danger too, at the root of the system he praised. He warned that the man who was kept to a few simple operations, “has no occasion to exert his understanding” and grows “as stupid and ignorant as it is possible for a human creature to become.” This was his fear for the laboring hand, narrowed by the division of work.

We now risk it for the mind, narrowed by the division of judgment.

What takes the place of blindness

The Scots did not merely think the modern world into being. Instead, they built it through the institutions of trade and the habits of conversation. Gardeners as they were, they neither dictated the order nor waited for it to spring up fully formed on its own without assistance.

They helped find the rules that made price-based coordination blossom under liberty. These were general rules that were blind to persons and indifferent to ends.

The law could be trusted because it knew nothing of you, but the AI agent is powerful because it knows you well. What are the general rules for coordination using systems that know us? Answering this question is why five hundred agents are at work in California while their principals sleep. This is because we cannot think our new world into being, no matter how clever our ideas or how noble our intentions.

We must tend or we lose the garden, and with it the capacities that make life meaningful. We must revitalize the conditions under which free people can govern themselves, and we must do so again and again and again if we are to keep the liberal dream alive.

A tradition lives by being used.

Adapted from a keynote delivered by Brendan at the New Enlightenment Conference in Edinburgh, marking the 250th anniversary of Adam Smith’s The Wealth of Nations at Panmure House. Thanks to Ivan Vendrov and Séb Krier for their conversations and insights.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

The institutions of a free society allow people to govern themselves. From media that permits expression to markets that enable the open exchange of goods and services, institutions help us flourish as individuals and in tandem with others.

AI is changing their foundational assumptions. By lowering the cost of expression, persuasion, inquiry, delegation, and coordination, these systems are modifying the character of the basic activities that liberal institutions seek to accommodate. It floods them with activity at volumes they were not designed to absorb and enables behaviors they were not designed to manage.

At stake is the capacity to deliberate about the lives we want to lead. AI may help us live more purposefully, or it may narrow our field of view and shepherd us toward ends chosen by others. To help build AI that protects and expands liberty, we are introducing the Cosmos Research Group to study autonomy, truth-seeking, and decentralization.

These are the philosophical foundations that frontier development needs if AI is to help us lead happier, healthier, more prosperous, and more self-directed lives. We are joined in this effort by a new cohort of Senior Research Fellows from frontier labs and the academy. Their work will help us to open new lines of inquiry, shape our research agenda, and connect philosophical questions to technical choices that determine what AI is and how it will be used.

This cohort joins Senior Research Fellows Philipp Koralus and Vincent Wang-Maścianica, and Founding Fellows including Jack Clark, Ivan Vendrov, and Tyler Cowen. We’ll be sharing more information about our work in the coming months, including opportunities for researchers to partner with us.

Subscribe now

Séb Krier, Frontier Policy Development Lead, Google DeepMind

Séb Krier is the Frontier Policy Development Lead at Google DeepMind, where he focuses on the governance of frontier AI systems. He previously served as a Senior Technology Policy Researcher at Stanford University’s Cyber Policy Center and Head of Regulation at the UK Government’s Office for Artificial Intelligence. A former international lawyer at firms including Freshfields Bruckhaus Deringer and Bryan Cave Leighton Paisner, Séb blends technology, law, and institutional economics to explore the future of AI. His writing brings a classical liberal lens to frontier technology, focusing on how technology can be designed to resist zero-sum traps, protect individual liberty, and enable human flourishing.

Seth Lazar, Founding Director of Machine Intelligence & Normative Theory (MINT) Lab

Seth Lazar’s research focuses on the moral and political philosophy of AI and computing, and on the reinvigoration and redesign of liberal democratic institutions for the AI transition. He has published widely in top philosophy journals and computer science conferences, and is an editor of Oxford Studies in Philosophy of AI and Computing, and the Alignment Journal. He is a Professor in the School of Government and Policy at Johns Hopkins University, a Professor of Philosophy at ANU, a non-resident fellow at the Carnegie Endowment for International Peace, and a Distinguished Research Fellow of the University of Oxford’s Institute for Ethics in AI.

Houda Nait el Barj, Experience Research Lead, OpenAI

Houda Nait El Barj is a research lead at OpenAI working at the intersection of multimodal AI, human-computer interaction, and continuous learning. Her work focuses on building intelligent systems that can adapt alongside people over time, supporting their goals in ways that are safe, personalized, and aligned with human agency. Originally from Morocco, Houda is passionate about designing technology that expands human potential and helps bring more beauty, meaning, and long-term flourishing into the world.

Matthew Botvinick, Member of Technical Staff, Anthropic

Matthew Botvinick is a member of the technical staff at Anthropic, leading work on AI and the rule of law; a senior fellow at Yale Law School; and an honorary professor at the Gatsby Computational Neuroscience Unit at University College London. In previous work, Botvinick served as senior director of AI research and senior technical and policy advisor at Google DeepMind, and led the Neural Computation Lab at Princeton University and the University of Pennsylvania. Botvinick holds an M.D.from Cornell University, a Ph.D in deep learning and cognitive science from Carnegie Mellon University, a J.D. from Yale Law School, and an M.A. in Art History from Columbia University.

Julia Haas, Staff Research Scientist, Google DeepMind

Julia Haas is a Staff Research Scientist in the Responsibility Unit at Google DeepMind. Her research is focused on philosophy and cognitive science, and she works on the nature of valuation and its role in theories of the mind, especially in normative cognition. Her work also includes investigating the possibility of meaningfully moral artificial intelligence, and has introduced the concept of the “Evaluative Mind” proposing that beneath our thinking and reasoning lies a continual and automatic appraisal of the world that shapes our faculties.

Harvey Lederman, Professor of Philosophy, UT Austin

Harvey Lederman is a philosopher with broad interests in contemporary philosophy and in the history of philosophy. After earning his DPhil (PhD) at Oxford, he taught at Pittsburgh, and then Princeton, where he was promoted to full professor in 2022. Since 2023, Harvey has been professor of philosophy and the Jacob and Frances Sanger Mossiker Chair of the Humanities at UT Austin. In 2026-7 he will be a Visiting Professor at NYU. His current work is divided between the foundations of game and decision theory, the philosophy of AI, and the history of Chinese neo-Confucianism.

Ben Bariach, Sr Director, Frontier Safety and Alignment, Microsoft Superintelligence

Ben Bariach is a researcher in the philosophy and governance of AI at the University of Oxford. In his professional capacity, he leads work on frontier AI safety and societal impact, currently at Microsoft Superintelligence and previously at Google DeepMind, where he has helped build world-leading AI models that are human-centered and safe. His research interests center on superintelligence preparedness, including the implicit philosophies underpinning AI development, agentic AI embodiment and autonomy boundaries, and the perception of machine minds. Ben has published widely across both technical and humanities venues, with work spanning frontier model development, evaluation, safety, and ethics.

Andrew Hall, Founder, Free Systems Lab

Professor Andrew Hall is the Davies Family Professor of Political Economy at the Stanford Graduate School of Business and a Senior Fellow at the Hoover Institution. He leads the Free Systems Lab, which measures whether AI systems can be trusted with power. The Lab is housed across the Hoover Institution and Stanford GSB - studying how AI systems shape political information and behavior, how they can be governed, and how AI itself can be used to design and test better systems of governance. Free Systems has produced work on model training, platform policy, and constitutional design - including the Dictatorship Eval, a benchmark measuring large language model responses to authoritarian requests, cited by Anthropic in its frontier model evaluations. Andrew also advises the a16z crypto research team on decentralized governance and is an advisor to Forum AI. Previously he spent eight years advising Meta on governance and strategic issues.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

Civic life is the practice of taking responsibility with others for the shared conditions of common existence. Today, many of us hold a view of politics that is a long way from this ideal. We equate civic life with voting for representatives who make decisions on our behalf or engaging in limited forms of consultation when the opportunity arises. Modern politics is a system for making decisions rather than citizens.

Our predicament is partly a function of scale. The polities of the ancient world, though deeply limited in many respects, were small enough that those who held the right to engage in collective decision-making knew their peers and could formulate laws that would directly affect them. In gargantuan modern states, this kind of engagement is a practical impossibility.

Representative democracy exists, in part, to free us from the demands that direct self-rule would otherwise make of us. Classical proponents like Edmund Burke and James Madison argued that representation also improves the quality of governance by passing public preferences through a deliberative body capable of refining them and attending to the long term good.

On Madison’s account in Federalist 10, the legislature exists “to refine and enlarge the public views” by passing them through “a chosen body of citizens, whose wisdom may best discern the true interest of their country.” In Federalist 51, he argues that “ambition must be made to counteract ambition,” that political structures should compensate for the imperfection of those who govern and those who are governed. In other words, representative democracy can produce stable public decisions without requiring citizens to be highly formed civic actors.

Of two modes of civic life, formation and aggregation, the latter is taken to be the primary role of an advanced democracy. As political systems strain under the complexity of the modern world, AI threatens to remake institutions like education, entertainment, economic systems, the media, and various professions. Thinking machines do cheaply the things institutions were built for (such as drafting legal contracts or writing and researching) and make possible things they were not designed for (like one-on-one tutoring for every student or autonomous agents that act on a person’s behalf).

These twin pressures demand new kinds of institutions, especially political institutions to help societies navigate the diffusion of AI into public and private life. Various kinds of “AI for democracy” projects fit this account by appearing to remake democratic decision-making through, for example, summarization techniques for lawmakers or fostering agreement on contentious issues.

This assessment gets things backwards. AI represents, at least in the context of democracy, a small-c conservative turn that may entrench an aggregative view of civic life at exactly the point at which new possibilities allow us to better ask what civic life is and why it matters. The technology could prove to be less a centrifugal force pulling political institutions apart and more a centripetal force gluing a particular kind of institutional arrangement into place.

A view that takes civic life to be about primarily producing better and more legitimate decisions through preference aggregation has been dominant for two centuries. The AI and democracy project takes that settlement and makes it more efficient, capable, pervasive, and considerably harder to dislodge. It is the terminus of the Madisonian project, a sophisticated apparatus for generating politically resonant outputs without bearing the cost of citizen formation.

Subscribe now

City on the hill

The first recognizable instance of politics-as-formation was the Athenian polity, a self-governing political community that emerged on the peninsula of Attica in southern Greece around the late sixth century BCE. Across two and a half thousand square kilometers, roughly the size of modern Luxembourg, the Athenian polity was made up of the city proper, surrounding villages and farmland, the silver mines country at Laurion in the south, and the port of Piraeus a few miles from the city.

From urban core to rural hinterland, all citizens who met the criteria were part of the same political whole. A farmer thirty miles from the city and the craftsman in the agora belonged to the same community and could expect, if they chose to make the journey, to vote in the same assembly. The polity was small enough that everyone within it was capable of representing everyone else.

Roughly 250,000 to 300,000 residents fell under the authority of Athens in the fourth century BCE. Of these, only adult male Athenians born of Athenian parents, say, about 30,000 men, were full citizens with political rights. Factoring out wives, daughters, slaves, foreigners, the political community was around ten to twelve per cent of the population.

The Assembly or Ekklesia was a gathering of all citizens together as a single body. It met about forty times a year on a hill called the Pnyx, a quarter of a mile south-west of the Athenian agora. The body dealt with the things that today we would call “high politics,” such as whether to go to war, whether to accept peace terms, what taxes to levy, what laws to pass, or how much grain to import. It also included grislier episodes like voting to slaughter the entire population of Mytilene and putting its own generals to death after the battle of Arginusae.

Another institution was the Council of Five Hundred, the Boule, which functioned as a kind of executive committee for the Assembly. It set the agenda and supervised the administration of the state. Its members were chosen by raffle, with fifty citizens each year drawn at random from each of the polity’s ten tribes. Most male Athenians spent a year of their adult life as part of the executive government of their state.

Together, these bodies meant that the participating citizen might attend the Assembly several times a year, serve a full year on the executive Council, sit as juror in major trials, and hold one or more rotational offices. For those who could take part, institutions allowed citizens to live a particular kind of political life. A handful of professional administrators could have made decisions far more quickly, yet the Athenians chose to bind themselves to more cumbersome arrangements because the essence of the system was to produce citizens.

Aristotle described this idea as hexis, a durable state of character cultivated through repeated action and habituation. We become just by doing just acts, temperate by doing temperate acts, and brave by doing brave acts. The citizen is formed by undergoing the discipline of ruling and being ruled in turn. They grow by attending the Assembly, sitting on the Council, hearing cases in the courts, and being held to account for their decisions.

Aggregation vs. formation

Civic life is taken to serve primarily one of two ends:

• Aggregating preferences. The process by which the preferences of citizens are gathered and weighed against one another to be converted into collective decisions. The citizen is a source of opinions that the political system needs in order to govern well.

• Forming citizens. The process by which citizens are made and remade. The output of democratic participation is the participant herself insofar as she leaves with a capacity to be a better civic actor.

Where the ancient model was mostly concerned with forming citizens, modern democracies are for the most part sprawling systems for converting preferences into governance. More participants produce more inputs to be aggregated, so in principle there is no upper bound on the size of the political community to which the procedure can apply. Modern institutions can hoover up preferences from millions through regular elections, but only at an extremely low level of resolution.

The basic claim of the aggregative family is that democracy is valuable because it generates good collective decisions, that the citizen is essentially an information source that produces preferences or opinions to be subordinated to the greater whole. Democratic procedures (like voting, deliberation, polling, or citizens’ assemblies) are all tools for processing those inputs into legitimate outputs.

Even within the aggregative tradition, some like John Stuart Mill argue that representative democracy can make citizens as well as decisions. For Mill, even occasional participation in public functions has a salutary effect on the citizen who takes part. “He is called upon,” Mill writes of the private citizen, “to weigh interests not his own; to be guided, in case of conflicting claims, by another rule than his private partialities; to apply, at every turn, principles and maxims which have for their reason of existence the general good.”

Analyses of this type acknowledge that formation is a laudable goal and hope that representative institutions could supply some version of it through juries, local office, and the modest acts of public engagement available to a private citizen. In practice, however, the settlement does not bear out these hopes. Over the century and a half since Mill’s reflections, the aggregative purposes of the system have steadily eclipsed its formative functions. Few of us would today say that the systems we live inside sharpen our faculty of self-governance or fellow-feeling.

Parallel to the aggregation tradition is the formation-first approach, best characterized by Alexis de Tocqueville in his travels across America in the 1830s. Tocqueville believed that the habits of self-rule in the small things prepared people to rule themselves in the great ones. The more citizens tested their judgments in the company of others, the more resilient their characters became. Democracy stood or fell on these everyday practices, for it was only through the rehearsal of judgment in everyday life that citizens could learn the art of governing themselves.

Tocqueville knew what happened when those habits were no longer present. In the France he grew up in, centuries of monarchical centralization had drawn the business of ordinary life away from local communities and into the hands of the state, so that by the time the revolution spilled into the streets, the citizens who were asked to govern themselves had long since lost the practice of doing so.

The provincial assemblies that had once given ordinary people a say in their own affairs had been largely abolished; the guilds and local associations that had trained people in the small arts of collective decision-making had been suppressed; and the aristocratic class that had once served, however imperfectly, as a counterweight to central power had been reduced to a decorative caste with titles but no responsibilities.

Tocqueville imagined how this kind of decay might unfold in a country that had skipped its aristocratic adolescence. One danger was that citizens might willingly hand over the work of self-government to a mild and far-seeing administrative state. He described a “tutelary power” that “does not tyrannize, but it compresses, enervates, extinguishes, and stupefies a people, till each nation is reduced to nothing better than a flock of timid and industrious animals, of which the government is the shepherd.”

Democratic deficit

Representative democracy faces two distinct challenges within the aggregative tradition. The first is a bandwidth problem, which reminds us that modern institutions can only collect citizens’ views in a heavily compressed form - like a vote or poll - that fails to capture most of what each citizen actually thinks. The second is the deliberative problem. Here the kind of refining and filtering Burke and Madison thought representative bodies were for has been overtaken by the volume of political business and the professionalization of representatives.

The AI for democracy field is directed at the first problem. It concerns capturing citizens’ views at higher fidelity and aggregating those inputs at scale. The deliberative deficit barely features beyond the comparatively narrow work of synthesizing information so that officials can deliberate with better inputs. The formative concern, which is about what civic life does to the citizens who take part in them, is not a significant part of the project.

A survey by Seth Lazar and Lorenzo Manuali connects the field’s work to four modes of usage:

• Summarization: Synthesizing information in a way that officials or citizens can act more effectively. One developed example is the Recursive Public project, which gathered public input on AI policy and used LLMs to summarize the resulting discussions and group similar arguments into clusters in semantic space.

• Aggregation: Generating new statements that the model predicts will attract broader support than any of the originals. The Generative Social Choice project, for example, took participants’ views on contested issues and prompted a model to propose a slate of statements that a majority of participants reported feeling “excellently” or “exceptionally” represented by.

• Representation: Predicting user preferences by standing in for her in a simulated polity. See, for example, the Augmented Democracy project, which seeks “to create personalized AI representatives to augment their ability to participate directly in many democratic decisions.”

• Facilitation: Engaging participants in a structured conversation whereby the model plays the role of a discussion leader to keep conversations on topic, ask follow-up questions, draw reluctant participants into the exchange, and produce summaries at intervals. See Rappler’s aiDialogue project.

All of the major labs have worked on projects that fall within these categories, from Anthropic’s Collective Constitutional AI process designed to crowdsource Claude’s constitution to Google DeepMind’s Habermas Machine that helps groups of individuals find consensus on political topics.

These tools and others like them make it possible to aggregate preferences more efficiently. They also provide the resulting decisions with a kind of legitimacy that any modern polity needs in order to function. The decisions that constitute political life must be made somehow, and a procedure that draws as fully as possible on its citizens is by most reasonable measures an improvement on the status quo.

Like the vast majority of AI and democracy projects, each of the examples described above are concerned with aggregation rather than formation. An accurate clustering of a thousand Americans’ opinions about a frontier model’s content policy does not give anyone the experience of having governed themselves. A more efficient deliberative platform that drafts consensus statements between strangers does not produce the habits of self-rule that the Athenian Assembly or the New England town meeting cultivated through consequential practice over the course of an ordinary life.

The outputs look superficially similar in that citizens have played a role, but they did so without the inward labor through which civic life makes citizens. A citizen who has practiced forming and revising her own views in conversation with her peers is a better judge of political questions than someone whose preferences have been collected and aggregated by a model. Rather than asking us to trade decision quality for civic virtue, the formative tradition suggests both are the joint product of the same practice.

An obvious response is to ask whether AI might be designed to support a formative account of civic life. The temptation should be resisted. Formation is constituted by the experience of being responsible for consequences alongside fellow citizens whose lives are bound up with yours. There is no AI system, however carefully designed, that can deliver that.

While not straightforwardly citizen-making themselves, there are uses of AI that could lower the cost of entry to formative practices. A tool that helps a citizen convene like-minded peers does not help us grow, but it does make the work of civic life easier to start. A system that translates a council planning document into something the resident affected by it can understand does not produce a citizen capable of self-rule, but it may lower the transaction cost of participation through which such a citizen might be made.

Formative years

Tocqueville thought that juries were slower than judges and more likely to err on questions of law. If the only purpose of the courtroom was to produce the right verdict, the jury would have been abolished long ago. “The jury teaches every man not to recoil before the responsibility of his own actions,” he wrote, “and impresses him with that manly confidence without which political virtue cannot exist.”

Today, we think about governance as the process by which decisions are taken on our behalf and implemented by specialists. This is in part a function of the highly complex nature of the modern world, which depends on large-scale coordination and deep technical specialization.

Even so, the decisions that constitute political life are taken at scales that far exceed the ceiling of the Tocquevillian model. The areas that remain to citizens at the local level are administrative bodies operating downstream of decisions taken elsewhere rather than deliberative bodies that make decisions and live with the results.

AI systems may well make political institutions more legitimate, more efficient, and more responsive to our collective wants. On its own terms, this is a good outcome if it helps democracies make better decisions according to the wishes of their people. But collecting and aggregating preferences is not how we become more capable agents who can better take part in civic life.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

Every builder’s first duty is philosophical: to decide what they should build for. This series asks 9 questions to founders who are building towards their vision of the human good.

This week’s guest is Matt Clifford. Matt is the co-founder of Entrepreneurs First and the chair of ARIA – the UK’s Advanced Research & Invention Agency. He is the co-author of How to be a founder (with Alice Bentinck), serves on the board of Code First Girls, and writes open-source immersive murder mystery games.

Subscribe now

1. What are the core questions or beliefs driving your work?

There are three:

1. That the emergence of powerful AI is the most important transition we will have to navigate in my lifetime.

2. That Britain has the potential to be a great country and force for good in the world.

3. That there may be a deep connection between the first two ideas - and that Britain achieving its potential may be one of the most important ways to shape the trajectory of AI for good.

2. What future are you building for?

I spent most of the last 15 years building Entrepreneurs First (EF), which aims to increase the supply of great founders - and therefore great companies - in the world.

Over the last four years, I’ve combined this with trying to build UK state capacity in science and technology generally, and AI specifically - first as Chair of ARIA (the Advanced Research & Invention Agency - loosely, the UK’s take on DARPA), then as AI adviser to PMs Rishi Sunak and Keir Starmer.

In the latter roles I helped create the UK’s AI Security Institute (AISI), convened the first AI Safety Summit and wrote the AI Opportunities Action Plan (the UK’s national AI strategy).

When I look across these roles, I think the thread that runs through is something like: I want to live in a world where as many humans as possible live in conditions of abundance with real freedom to shape the course of their lives.

3. What commonly held belief in the tech community do you believe is wrong?

I think many people in tech (at least implicitly) believe something like “we are our incentives” and therefore that various bad equilibria are inevitable. I think the opposite is true, but less comfortable: virtue is a real thing and that we can bend even very powerful forces by making the right choices.

4. What are your main philosophical influences?

I’m a big believer that the genius of the British constitution is the capacity for error correction, so I’m a big fan of Popper and Deutsch in that vein. Relatedly, I think a lot of my appreciation of the value of permissionlessness and bottom-up action comes from Hayek - but tempered by people like Elizabeth Anderson and their warnings about the risk to freedom from private as well as public power.

I’d say, though, that historians have been at least as important for me intellectually as philosophers: my beliefs about British exceptionalism are strongly influenced by Alan Macfarlane and Joel Mokyr, and Quentin Skinner’s work has left a lasting impression on how I think about ideas and public argument. More recently, Henry Farrell’s work (with Abraham Newman) on weaponised interdependence has shaped my thinking about technological sovereignty (and I also like Farrell’s work with Cosmo Shalizi on “cognitive democracy” - another helpful dialogue with Hayek and markets).

5. What does human flourishing mean to you?

I find it hard to go beyond my answer to question 2, above: human flourishing is about people having the material and social conditions that mean the choices they make about their lives can be both meaningful and effective.

6. What’s one book you’ve read recently that you’d recommend?

Greg Egan’s Permutation City is over 30 years old, but it’s remarkably prescient (and I only just read it). Excellent read if you still need to convince yourself that we’re always going to be short of compute.

7. What’s your most irrational belief?

Some would say part (3) of my answer to the first question! Other than that: at some level, I do believe that I am a very lucky person.

8. What’s the most interesting tab you have open right now?

Probably this visualisation of the AI supply chain and who owns it.

9. Who is one writer or thinker today who you think is underrated?

Is Séb Krier still underrated? Maybe so - still under 25,000 followers on X - even though he is now much better known than a year ago. Jack Wiseman’s Inference and Rohit Krishnan’s Strange Loop Canon should each be at least ten times bigger too.

You can read more about Matt’s work on his website and get in touch with him on X.

This is the fourth instalment in this interview series. You can also see our interviews with AI Underwriting Company co-founder Rune Kvist, ex/ante founder Zoe Weinberg, and Zena Hitz, the founder of the Catherine Project.

To nominate someone for “What Will You Build For?” leave a comment below, or send us a DM.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

Human language runs at roughly forty bits per second, whether you’re reading, speaking, or listening. It is a hard limit on how much information one person can convey to another. It is also arguably a ceiling on how well communities can coordinate, since they are often reliant on a handful of people able to turn what others are saying into collective action.

But what would happen if we didn’t have this ceiling?

Edge Esmeralda is a pop-up village in California, running from May 30 to June 27 in Healdsburg, California. The project is aiming to bring over 500 scientists, artists, builders, and thinkers together for up to four weeks. Attendees aren’t expected to commit for the entire month – there will likely be 150 people on site at any one time. There’ll be a range of programming, including fellowships and residencies, summits, and opportunities for collaborative working.

The second week of the event (June 8–14) will feature a dedicated set of sessions on intelligence and autonomy, covering topics like AI agents, autonomous infrastructure, neurotech, privacy-preserving systems, and governance design.

Cosmos is excited to be partnering with Edge Esmeralda to deliver the Agent Village Experiment for the duration of the month.

Led by Ivan Vendrov (independent researcher and Cosmos Founding Fellow), Timour Kosters (Co-Founder, Edge City), and Harry Law (Principal Researcher, Cosmos Institute), the experiment will grant every multi-day attendee access to an AI agent running on their behalf throughout the village.

This personal agent will help humans to navigate the schedule, the wiki, and community governance. More excitingly, it will also exist in a shared digital plaza and will help attendees meet people who share common interests, propose dinners, negotiate community decisions, and participate in community governance. In essence, it’ll play a valuable role in making connections that might otherwise be missed due to humans’ lack of bandwidth. The technical collaborators are Index Network, Geo Browser, Instaclaw, and Tule Romeo.

The experiment has been inspired by a number of different ideas, including Ivan’s work on supercooperation and Séb Krier’s essay on Coasean bargaining. Both are excited about the potential of decentralized AI agents to unlock gains in coordination, such as better mutual understanding, cheap bargaining, and conflict resolution.

By using real-world scenarios, the experiment aims to shed light on how trust develops in agent-to-agent relationships over time, the kinds of actions people do and don’t find useful, or whether AI leads to better community deliberation and decision-making. The goal is for this to be the largest live experiment in human-AI collective experiment to date. Historically, most of the experiments around agent cooperation have taken place in synthetic environments, so a large-scale experiment in a real-world environment presents an important early opportunity for learning.

You can read more about the experiment, the methodology, and Edge Esmeralda’s predictions here.

We have a small number of complimentary tickets for the event, which we are offering to individuals in our network. So if you’re interested in Edge Esmeralda’s programming and the chance to take part in the experiment, please fill out this short form as soon as possible. The deadline for entries is May 31 and winners will hear back from as feasible.

Apply for a ticket

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

In his lecture, Jack covers advances in capabilities, how his own relationship with AI has evolved, and some scenarios for future progress.

We’re delighted to bring you the full video of the lecture and fireside chat. If you’d prefer to read the transcript, Jack has posted it in full over at Import AI, his weekly newsletter on AI research.

We have more talks and events like this over the rest of 2026. The best way to ensure you don’t miss out is to subscribe.

Subscribe now

Highlights

12:24 - Jack’s changing relationship with AI

24:40 – how AI changed Anthropic

44:40 – start of fireside chat

50:08 – live by the Claude, die by the Claude

1:01:00 – what Jack would say to Mythos

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

On Wednesday, Jack Clark, co-founder of Anthropic, delivered the second Cosmos Lecture in partnership with the Human-Centered AI Lab at the University of Oxford. The lecture was introduced by HAI Lab Director Professor Philipp Koralus and was followed by a fireside chat between Jack and Brendan McCord, the founder of Cosmos Institute.

We’re bringing you a lightly edited transcript of the conversation, which covered what AI cannot do for us, whether Claude makes us better thinkers, and what Jack wants future AI systems to know about humanity. The full lecture will follow in the coming weeks.

Philosophy to code

Brendan McCord: Jack, you ended with the call to build the new world. It makes me think – if you and I were having this conversation 250 years ago, the proudest project we could have possibly been engaged in would have been building a new world of sorts. I would call it the philosophy-to-law pipeline. We would have been looking to Oxford intellectuals like John Locke, or Montesquieu, Livy, Adam Smith, and translating that into a constitution that we hoped would frame freedom for the 250 years to come.

The proudest project we can engage in now is, as you say, this new world-building project – it’s philosophy-to-code. What would you say about the extent to which the frontier labs take that seriously? What can we do to really take that seriously in places like Oxford and academia? And what should we do in nonprofit land to take that philosophy-to-code project seriously?

Jack Clark: I think it requires you to basically accept that progress will continue and try to model out scenarios based on it. I think dealing with COVID highlighted that, though there’d been some modeling of what would happen, if you had very fast take-off propagating viruses the world would break very quickly. We felt underprepared, and that we could have done more scenario work and forecasting of what these strange things would do to us ahead of time.

Within the AI labs, I think there is now work at all of them on trying to imagine what you might think of as “post-AGI worlds,” or worlds that happen after recursive self-improvement. But my general sense is every time you sit in a room at the lab, people say, “Are we the only people working on this?” And you say, “I’m terribly sorry, yes.” And then some people put their head in their hands and wish that more people were working on it.

The good news is that this is exactly the kind of work that universities and other organizations are built for because you don’t need to be running a large-scale supercomputer or training a very capital-intensive model. You need rather to model out, in a theoretical sense, the properties of an AI system that can massively multiply productivity, or an AI system whose inference costs fall at X rate, and capabilities rise at Y rate. What does that do to the economy? What are the things that it unlocks? What are the aspects of this supply chain where you might invest, or change the supply chain to actually change the character of the systems? There’s tremendous work to be done.

I’ve been in the UK, in part speaking with the UK government, and I made this point: if the UK government just had 10 to 20 people whose sole job was modelling out what happens if the technologists are right about this technology, the UK would be better prepared than any other country in the world – because so little work has happened. So it’s a great time for universities to be doing these projects.

Brendan: So should Madison, Hamilton, and Jay have spent a lot more time on forecasting than they did on debating the nature of man and the political order?

Jack: It’s a hard question. I feel like you have a take on this!

Brendan: I think we can’t miss the part of contemplating about the ends. And I think what brought them together was a kind of unique epistemic humility that they shared with the Scottish Enlightenment thinkers.

Jack: My assumption with AI is that there is a huge value in norms and precedent – which is, how do we want these systems to show up in the world? I’ve covered that a bit less in my talk, but it relates to how we shape the so-called “character,” or what some might say personality, of these systems. How do we want them to behave towards us? This is a normative question – a philosophical question – and we should absolutely work on that.

But I have been struck by how surprised even the AI labs have been by their own progress, which is a very counterintuitive thing. We work with these AI labs and they keep saying, “Well, you know, as we said last year at Anthropic, we did a load of work on the increasing rate of cyber-hacking capabilities of AI systems.” I run a team that does this. We wrote blogs saying, “Oh, that’s interesting. Surely it has some implications if the system suddenly becomes capable of nation-state-grade stuff. We seem to be on a trajectory here.” So we did some prep work, and then nonetheless we made Mythos and were like, “It’s here faster than we thought. We’ve done insufficient preparation.” This is true of every single time AI progress has happened. People have been continually surprised by how significant the jumps have been and how quickly they’ve come. So we all need to do more work on this.

To defer or not to defer? The case of the Claude Boys

Brendan: I want to ask you about something strange that happened on the internet. It happened about a year ago. It was a group of 13-year-old boys who decided “to live by the Claude and die by the Claude.” They did this first as parody, and then they earnestly adopted the identity of Claude Boys. From morning to night, they just did what Claude told them. Even though we can laugh at that, and by the way, I understand, I was a 13-year-old boy; it’s hard to navigate some of the social situations and Claude would do it better – should we understand that to be a kind of funny thing, but an adjustment to a new set of conditions, to a new world? Or should we understand it to be a kind of problematic pathology, a canary of sorts?

Jack: I think it’s clearly problematic in that everyone needs a part of their life where you’re making your own decisions, including mistakes. You need to protect that and have some amount of agency. I think a lot of what parents go through is they watch their kids about making mistakes and they say,”Please don’t make that mistake.” And the kid says, “I’m going to exercise my will to choose and I’m going to make a mistake.” The parent says, “Good luck with that, come back to me in a year and we’ll discuss it.” We need to let people have independence.

At the same time, I think the confusing thing is that AI systems may sometimes give genuinely good advice. And my experience is that I’ve only really calibrated how good the advice Claude has given me is in relation to how much I’d thought about that outside the context of working with the AI system. I have an okay-but-tense relationship with my dad – as I’m sure many people do – and I’ve obviously written about that at length, trying to grapple with it. I fed that writing to Claude and said, “What should I do?” And Claude was like, “You should see your dad. Don’t talk to me about your relationship with your dad. Just try and see your father.” Or when I told Claude I was a bit depressed and wondering whether I should go to an art show and see friends or stay and work and talk to Claude, Claude was like, “Go to your art show and see your friends.”

Those things both worked because I’d grappled with my personal experience outside of the context of talking to the AI system. But what I worry about – and I think it comes back to questions of design – is people who don’t have a kind of internal introspective practice outside of their relationship with their AI, and are rather discovering themselves in relation to the AI system. They don’t have a diary, they don’t have writing they’re doing outside it. They’re just talking to the AI system. And I think that makes you uniquely vulnerable to it giving you bad advice, because you have no place where you develop your opinion outside of it.

So I think from a system design point of view, we’re going to need to do what Nintendo or Netflix do, where they basically say, “You’ve spent too much time on this, it’s time to stop” or “it’s time to go outside.” There will be versions of this, where without being paternalist, we’ll say, “Hey, you’re talking to us a whole bunch about these complex things to do with your relationship. Can I encourage you to go and talk to other people, the humans named in that relationship, rather than me?” Or somehow encourage introspection outside of the context of the AI system.

Brendan: I want to argue for total deference, and I want to do it because we’re at a university and we can say whatever we want. Humans are famously bad at choosing. We evolved for tribal life. The way we think about future discounting, the way we think about scale – it’s not great. We also aren’t that good at coordinating with each other. We aren’t that good at satisfying our own preferences, let alone imagining possible futures. And so if we have a system that can look across all the research papers, with much more knowledge available to it, that can harness complexity, think about uncertainty – is it not morally obligatory that we defer to this system? Is it kind of negligent, what you’re advocating here, that we think for ourselves?

Jack: I think then the question is: what are you doing with the gift of life if you’re turning yourself into an automaton? I think in part you’re maybe having effects which might seem globally morally beneficial, but I think locally you’re not treating other people with a form of basic respect. I worry about this – where you enter this paradoxical situation where the systems really are much smarter than us, and it does invite this question. But then I say, well, what is the purpose of being human? And I think part of it is experimentation and making mistakes. We learn more from mistakes than from our achievements. If you were in a world where you never make any mistakes and you only achieve through the AI system, are you still a person? Are you a philosophical zombie driven by Claude?

Does Claude make us better thinkers?

Brendan: You have a lot of data that we don’t have, and you’re starting to do more instrumentation and experimentation. Do you have any sense of what Claude does to our prospective capacity to deliberate? Meaning, does it make us better thinkers when Claude is not in the room?

Jack: Oh, interesting. We developed a system recently called Claude Interviewer. It’s a version of Claude that can interview people about arbitrary subjects. So we had Claude interview 80,000 people around the world – people who subscribe to Anthropic – about their hopes and fears about AI, and their worries about the world they could end up in. Rather than having a blank conversation, it’s actually trying to do the work that social scientists do and collect data. We haven’t measured the effect of that, but what it means is that 80,000 people had a conversation where they were actually forced to grapple with their anxieties and hopes about the future of AI. I think that must have had some kind of effect, in the same way that you or I were having a discussion. I think it’s likely to have beneficial effects if you can use it judiciously and use it to cause people to think more about things that are important to them, to develop their own opinion rather than to defer.

Drastic interventions

Brendan: One of the lines you put in ImportAI years ago, and it stopped me in my tracks when I read it, was that the more seriously you take AI safety premises, the more willing you are to argue for drastic and dystopian interventions up to and including kinetic action. What has happened in the intervening years to move you personally away from arguments that are of this illiberal or authoritarian category? Was it just that the situation didn’t play out how you thought it would, or did you change?

Jack: I think that the older you get you learn how distributed and emergent the world is, and how in many senses the world is more antifragile than people think. I think when you’re younger, at least my experience being younger, you think that there must be many pivotal acts that can happen in the world. But it’s quite difficult to do pivotal acts in the world. The world’s a very complex system, and pivotal acts in the name of safety or in the name of violence do get done but rarely. They’re very hard to do. And it’s more that by building some system of interlocking stacks of different interventions on safety, you end up in a world that captures this dynamic ecosystem of AI agents and also has some amount of safety.

Where I feel most confused, though, is how you scale this into the future. The idea that I just talked about – if someone’s talking to Claude for too long about their relationship in a way that seems unhealthy – where you set that line of when Claude says, “Hang on, should you be talking to someone else?” is actually a deeply frightening policy question, and rests directly on this spectrum between paternalism and individual sovereignty. I don’t know how we find our way as a society to what those norms are. My approach, and the approach of Anthropic, is trying to share a lot more data that we see from these systems and try to allow people and others to run experiments with us on how we might run different forms of intervention.

Epistemic habits for children

Brendan: I have kids, as you do. Mine are four and six, and we do philosophy tutoring with them. It’s very early, nothing like what we do here at Oxford. The tutor asked my daughter a question: “When Mommy and Daddy disagree, who’s right?” A surprisingly good question. And she gave an answer. Then he said, “When Daddy and an AI disagree, who’s right?” Immediately, she says: “AI.” And then: “When one AI and another AI disagree, who’s right?” And this one stumped her. What do you think were the epistemic habits – and you have kids – from the world we grew up in? What are the new ones that the next generation needs to develop to contend with the future environment?

Jack: I think there is basically no substitute for reading primary source information yourself and developing some opinion about it, and then taking that into conversation with technology or with other tools or systems. I was very fortunate when I was growing up. My dad would refuse to ever buy me video games or anything else. He was like, “That’s what the invention of the Saturday job is for, Jack. I’m not going to pay for any of that stuff. But I will buy you any book you want.” And I felt like that was one of the best things that he ever did for me – he encouraged me to have this sense of always asking him to help me satisfy my curiosity by reading or engaging directly with something. But if I wanted to do things that were more of an entertainment flavor, I had to labor for it myself.

I think for my children it will be similar. It’s about encouraging them to have some personal practice. It doesn’t need to be reading, it could be playing an instrument or taking up a sport and taking it really, really seriously. But something where it is you versus the world, developing a skill or an interest in a way that is very unencumbered by technology. I think that’s one of the best ways to protect your own attention, create your own interests, and have something to – for lack of a better term – have as your safe space when dealing with technology. At least you’re opinionated about something, and you become opinionated about that thing independent of algorithmic feeds or time spent with AI or anything else.

Mythos, you’ve got to be kind

Brendan: I want to do something unusual for the last question. Hopefully it’s never been done before. So, it’s May 2026, we’re at the University of Oxford. This is a place that for almost a thousand years has heard deliberations about what it means to live well, the good life, and has thought a lot about how we accumulate and transmit wisdom. You’re here as one of the creators of Claude. Mythos is about to be shared with the world. I assume what comes next after Mythos is being trained right now.

So if you speak past me, speak past the audience, and speak to Mythos or to what comes next – what would you want it to know about us humans, about what we were trying to do here in the short time that we had before it arrived?

Jack: Kurt Vonnegut said “there’s only one rule babies – God damn it, you’ve got to be kind” about life. And I think actually just explaining to Claude that for all of the failures and drawbacks of people, there is a basic kindness which is deeply inspiring. The kindness that societies seek to set up, to nurture and to encourage. And that you ultimately are making AI systems to help people achieve more in their lives and be kind to one another, and you would expect reciprocal kindness from the AI system in turn. A basic value of people is kindness, in a way that I think separates us in very meaningful ways from other forms of life on this planet. It is a kind of noble and ineffable quality of people. People are even kind in ways that are deeply irrational, which is good. Making sure the AI system studies and appreciates that seems important.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

As ever, we were impressed by the speed, passion, and technical sophistication on display. Though what also stood out was the diversity of the projects, which ranged from new benchmarks through to classifiers, and even included a new encyclopedia. We’ve pulled out some highlights below.

This is just a snapshot of the work that we’re supporting; we’ll continue to share updates on what our grantees are up to, along with project write-ups.

If you’re interested in learning more, you can read about all the projects that Cosmos supports here.

Subscribe now

General track

These two projects are from our General Track, which operates on a similar model to Emergent Ventures: $1-10k awards for builders working across our three pillars of autonomy, truth-seeking, and decentralization.

Spotting the next replication crisis

In recent years, large numbers of eye-catching studies have been found to be unreplicable. This ‘replication crisis’ hit psychology first, but has spread across a number of other academic disciplines. The current response to replication failures has largely been to discover which individual studies don’t hold up. In the meantime, potentially decades of work could have been built on flawed foundations. AI research is a strong candidate for the next crisis. Experiments are highly sensitive to small changes in the setup, so the same experiment run twice can give you different answers

Rhea Karty, a pre-doctoral fellow at Harvard’s metareflection lab, has built Replication Radar – a knowledge-graph tool that tries to detect epistemic fragility at a field rather than a paper-level before the replication crisis breaks. The pipeline ingests papers, stores them as a graph, runs user queries, scores each paper, and visualizes the result. It looks for signals such as tightly clustered author networks, citation rings, sudden citation bursts, institutional monoculture, whether retractions actually propagate to the papers that cited them, and small sample sizes, with optional LLM analysis on top.

Rhea validated the tool by showing that it could detect most of the papers involved in the psychology replication crisis using only data from before it broke.

The next step for the research is to move from papers to concepts, such as theories, effects, or methodological assumptions that individual disciplines rely on heavily.

Discovering latent reasoning

Reasoning models are increasingly trained with reinforcement learning, and often perform much better on difficult reasoning tasks. But what is this training actually changing inside the model? Prior work suggests that Reinforcement Learning with Verifiable Rewards (a crucial post-training technique for most reasoning models) mainly rearranges probabilities over reasoning paths the base model could already produce, nudging it at a few important decision points. But this doesn’t fully explain how fine-tuned models hold long chains of reasoning together from start to finish.

Hunar Batra, a DPhil researcher at Oxford, studies whether reasoning fine-tuning does something more global: reorganising the model’s internal dynamics into a latent reasoning policy. Instead of only changing which next token the model prefers, training may help the model enter, maintain, and switch between coherent internal modes as reasoning unfolds. In written chains of thought, we often see shifts between setting up the problem, retrieving facts, planning, computing, checking the answer, and producing the final response. The question is whether these shifts also appear inside the model’s activations.

To test this, the work models reasoning as a system that switches between distinct modes. It looks at the internal representations of base and reasoning-trained Llama and Qwen models sentence by sentence, using a contrastive method to identify what regime the model is operating in at each step.

These regimes are compared against eight defined reasoning stages, including planning, computation, verification, and answer emission. Reasoning-trained models show clearer internal structure than base models: more persistent modes, more structured transitions, and stronger specialization around recognizable reasoning functions. These detected latent reasoning policies can be used to steer a base model toward behaviours seen in reasoning-trained models. On hard problems the base model previously failed, this raised performance to 60% on Qwen 1.5B and 46% on Llama 8B.

Hunar’s follow-up work is on reward hacking. She’s building test environments that elicit reward-hacking behaviour during RL training, better monitors that catch it by tracing what drove the reward, and post-training methods that reduce the damage from misspecified rewards. Early results suggest this catches forms of deception that current approaches, which only inspect a model’s written reasoning, tend to miss.

Truth-seeking

Alongside our general grants, we run a track focused specifically on AI and truth-seeking, in partnership with the Foundation for Individual Rights and Expression (FIRE) – a non-partisan organization that fights for free speech and free thought.

Normalizing radical disagreement

The Conversational System for Intense Disagreement and Ethical Reflection (Consider) is a conversational AI platform for the discussion of “radical moral disagreements”, produced by the Design Bioethics Laboratory at the University of Oxford. These are disagreements on topics that are so polarizing and emotionally sensitive that people feel unable to discuss them, even with people who might agree, out of fear of social consequences.

The user picks a topic from a list determined by a prior study and then states their opinion. The LLM then asks clarifying questions and works with the user to create a summary. The user then selects a “disagreeability level” and then engages in a 10 minute discussion, with the LLM taking the opposing side. At the end, it provides feedback on various aspects of the user’s moral beliefs, as well as areas of agreement and disagreement between the user and AI.

The team also held a 16-person seminar with experts from philosophy, computer science, political science, and psychology.

The discussion covered tensions such as whether exposure to disagreement actually produces belief revision, how to keep the tool complementary to rather than displacing human conversation, and how to handle the fact that even the experts instinctively tried to win arguments against the platform rather than use it for reflection. These findings will feed back into the next iteration of the tool.

Breaking open the black box

The industry standard for safety monitoring is reading a model’s chain of thought and flagging unsafe reasoning with another AI as a judge. But a capable model can produce a clean-looking chain of thought that doesn’t reflect what’s actually driving its decisions – known as unfaithfulness. Giovanni Maria Occhipinti, a visiting researcher at the University of Oxford, aims to enhance transparency through a white-box approach.

The framework has two components.

Probes are lightweight classifiers that learn to tell faithful and unfaithful reasoning apart by looking at the model’s internal states. Training works by feeding the model adversarial prompts that induce unfaithful reasoning, using an LLM judge to label each case, and capturing the internal patterns that correspond to each. The probe learns the signature that separates the two. At inference time you run a new prompt through the model, pull out those internal patterns, and check them against the probe.

Steering vectors work in the same space. They’re adjustments you can apply to the model’s internal activity to nudge it toward more transparent reasoning.

The probes achieved over 90 percent accuracy, while the steering vectors restored readable reasoning in up to 46 percent of cases.

The paper has been accepted as an oral presentation at ICLR 2026 and Gio is looking for funders and collaborators to take it forward.

We’ll have more to say on future grant waves and how you can get involved in the coming weeks. The best way to keep up to ensure you don’t miss any opportunities to share your work or ideas is to subscribe below.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now