He is one of our new Senior Research Fellows at Cosmos Institute.

The last few years have been some of the most exciting and maybe weirdest work of my career.

Sometimes I’ve felt like I’m pushing the frontiers of philosophy, sometimes I wonder if I’ve wandered over the border and into something else. On bad days I worry that I’ve just become an amateur computer scientist. But with the results coming in, I do think we’re on to something. It’s a bizarre feeling (as a philosopher) to feel like we’re pushing back the frontiers of knowledge, instead of just inventing a novel twist on problems that have been noodled over for centuries.

In this post, I introduce new work to argue that “normative competence” in general (and moral competence in particular) is among the most societally important capabilities that AI systems can have, that current LLMs are a lot closer to at least some aspects of moral competence than is widely believed, and that we have realistic methods for moving them even further forward.

What Do We Mean by Normative Competence?

I define it as the ability to recognize, understand, and act on reasons.1 Reasons can be for action, for belief, or for other things that I don’t think about as much. My lab is mostly interested in reasons for action.

We can further distinguish between analytical normative competence, the ability to recognize and understand reasons, and so to know which actions are supported by which reasons, and practical or behavioral normative competence, i.e. actually being disposed to take actions appropriately supported by your reasons.2 Behavioral competence can come apart from analytical competence in both directions. You might be great at knowing what the right action is, but disinclined to take it; or terrible at understanding reasons, and yet intuitively disposed to do the right thing (indeed sometimes engaging in moral reasoning might involve “one thought too many”).3

We mostly focus on moral competence because moral reasons tend to be the most important ones. But we’re interested in normative competence more broadly, both because sometimes social norms matter most, and because reasoning standards are pretty similar across domains (e.g. legal reasoning and moral reasoning more generally are closely related, though their content is different). We have started some interesting experiments that focus on particular domain-specific norms, for which we hope to eventually report results.

AI Shouldn’t Be Good at Ethics

The first AI project I ever had funded was on AI and moral skill! I worked with Jenny Davis, Colin Klein and Claire Benn, back in 2019, on a small project funded by the Templeton World Charity Foundation. We recruited Nick Schuster (now at University of Georgia), who wrote his PhD on moral skill, but he couldn’t get to us until after Australia’s borders reopened following the worst part of the pandemic. Honestly, the project was speculative; there were reasonable questions to answer about human moral skill in the age of AI, but the systems performed so badly that aiming them toward even minimal moral competence seemed hopeless.

Things changed with instruction-tuning, and the first conversations I had with ChatGPT were all about moral cases. I was also captivated by early work on constitutional AI, which suggested that LLMs could engage in enough moral reasoning to shape their own behavior for the better4; I wrote up the seed of this research program in a piece that I wrote right after those first GPT encounters, but eventually published in Aeon in 2024.5 From those first conversations I knew that I wanted to do some serious investigation of LLM moral competence. We got our first proper eval out the door in May 2025.6

The learning curve was steep! I was helped up it, a lot, by a six-month stint visiting Google DeepMind last year. Over the months since, my lab has brought all of this together into what is now, I think, a broad and extendable picture of LLM moral competence. Maybe the lesson here is that I need to learn how to focus my attention better and speed up. Maybe it’s that taking on a completely new research direction and methodology just takes a bit of a run-up. But I wanted to do this in a way that was genuinely philosophically-led, not just to plug in to some existing research program as an ‘ethics consultant.’

When I talk about moral reasoning in AI, you now know that I’ve been thinking about this for a while. And that history can provide valuable perspective: we have come *so far* over the last seven or eight years. Right up until around mid-2022, the idea that any AI system would exhibit any degree of normative competence was fanciful. I mostly thought that “machine ethics” as a technical field was a waste of time.7 People were proposing obviously ludicrous things, like translating the categorical imperative into programming languages8, or just using ML to imitate human behavior (so-called “bottom-up” approaches9), or, worse, training on human judgments about trolley cases.10 But the systems themselves were light-years away from the kind of morally-informed perception that even the most rudimentary moral skill requires.

Whatever else is true, that has changed. We’re now talking about systems that are probably more adept at moral reasoning than the median human, and which by all appearances have precisely the perceptual abilities that seemed so inconceivable before capable LLMs arrived. We have come a bizarrely long way.

The best way to see this is just to go to any really capable AI model (and it’s much better to use the best models, with reasoning enabled, than to draw conclusions from conversations with subpar models) and ask it about any kind of moral problem. Give it your favorite philosophical hypothetical, your personal dilemmas; pose a foundational question about public policy. You’re almost guaranteed to get back a more sophisticated and sensible answer than you’d get from most humans on that topic.11

And as much as we might not like it when they block what we think is a reasonable query, the fact that the models can so effectively navigate safety-relevant questions, and decide which prompts to respond to and which to deflect, for millions of users over quadrillions of inference calls, is itself evidence that these systems are preternaturally capable at normative reasoning of a certain kind.

A lot of the most recent papers on LLM normative competence focus on its limitations. These of course matter, and indeed, could matter a lot. But the technical accomplishment of coming this far should not go unnoticed.

Why Does Normative Competence Matter?

But look: it’s natural for me to care about moral and normative competence. After all, I’m a moral and political philosopher, everyone starts out by testing the models at the thing they consider themselves an expert at. Why should anyone else care? It’s worth saying why focusing on moral and broader normative competence matters (and that there are other reflections in this vein).12

Alignment

Let’s start with alignment.13 This is such an abused concept; everyone seems to love either calling for its retirement or trying to extract one last drop of either metaphorical or literal meaning. I’m going to play the boring analytical philosopher again, and just stipulate what I mean. Whether we’re talking about humans, chatbots, coding agents, or some future AGI, an aligned agent responds appropriately to the reasons that apply to it. You can basically fit any other more specific account of alignment into this one. People arguing for different conceptions of alignment are mostly debating which reasons apply to the agent in question, or what counts as an appropriate response.

Defined in this way, you might think that normative competence was basically synonymous with alignment, so of course understanding the former is invaluable for the latter. But I have to admit that we have so far worked mostly on analytical normative competence: the ability to recognize and understand reasons. Alignment is much more about practical normative competence, i.e. the ability to act on the basis of the reasons that apply to you. And analytical competence might not be the only, or even an especially promising, path to practical competence. People who want to behave more ethically don’t tend to think that a PhD in moral philosophy is the most direct route to that goal. So focusing on analytical normative competence might be an important step toward alignment. But, like most basic research, it involves making a bet that might prove to be wrong.

On the other hand, it’s also a bet that might prove right! Here’s why I’m bullish. We already have pretty capable AI agents. They can operate across a far wider domain of software surfaces and choice situations than was ever feasible for chatbots. As their capabilities and domain generality advance, they will face a combinatorial explosion of possible choice situations. It is no more possible to write down and enforce some list of declarative rules for their behavior across such a wide range of scenarios than it is to do the same for humans. Any “model spec” now has to look more like a Tao, or a guide to life, or Rudyard Kipling’s “If,” than the souped-up content moderation policy that ChatGPT is currently trained on.

Any set of prescriptions for AI agents will necessarily be incomplete, ambiguous, and in general dependent on interpretation. And we can’t rely on human review without sacrificing the goods for which we are building these agents. So we need some automated system to recognize those reasons across this vast range of possibilities. This necessitates moral (and broader normative) competence. It is clearly possible to simplify the relevant normative domain by restricting either agents’ capabilities or their generality to head off the need for interpretation. But we’d be leaving a lot of value on the table if we did that. And as these systems get more capable, implementing such strategies will get much harder.

Sometimes I think the harder-nosed AI security types turn away from normative competence work because they think that external engineering controls like input and output classifiers, probes, and chain of thought monitoring, are going to be enough to get us to safe, aligned AGI.14 But I think that’s a category mistake. Whether the mechanism for alignment is internal or external, you’re going to need systems that can automate normative judgment. This means that normative competence is a prerequisite.

Sometimes the crowd more focused on extinction threats from AI overlook normative competence. This perhaps makes sense. You don’t need a sophisticated understanding of morality or of social norms to know that killing all humans is bad. But in fact this is misguided. Even if you only care about preventing AI-caused extinction, the bad outcomes could easily come about from choices that don’t simply have as one dominated member of the option set “kill all humans”. For example, we’d need to teach subtle probabilistic reasoning, both about what to do when all your actions involve some extinction risk, and when you must make trade-offs between very low probabilities of astronomical harm, and high probabilities of more mundane costs and benefits. These are some of the most challenging problems in moral philosophy!15

And if you’re in the great majority of people who don’t care only about avoiding human extinction, then the case for normative competence is clear. Suppose you want to avoid human disempowerment.16 Building AI systems that don’t contribute to that will require subtle moral insight. For example, at some point (perhaps even now) we’ll have systems that are able to take over from human principals in ways that will clearly contribute to significant cognitive deskilling. To avoid creating foundations for total disempowerment, the models will need to be able to judge when and how to scaffold and support rather than replace human autonomy. This will require deep moral competence (indeed, of a kind that is probably beyond the skill set of most humans).

The Path to Moral Character

Even if you don’t buy the more theoretical reasons for normative competence being central to alignment, Claude Code’s recent successes should provide supporting empirical evidence.

Anthropic’s early work on constitutional AI (including criminally under-cited papers on specific vs. general principles for constitutional AI, and on the capacity for moral self-correction in language models)17 set the pace. I vividly remember seeing the latter paper presented in January 2023 and being blown away by how morally adept Claude was even then. In more recent iterations, this approach has become even more prominent, especially since the early, quite eclectic and a bit silly “constitution” was replaced with the epic “soul document” that now underpins Claude’s behavior.

I suspect this approach, and perhaps the attitude behind it, was integral to Anthropic’s extraordinary acceleration from late November/early December 2025, in which they took an early lead in the race to create functional agents. Claude Code worked so well because it could be trusted across such a wide range of domains. I suspect this was because the underlying models could reason effectively about those domains, and determine what to do in a responsible way.

Right after Opus 4.5 came out, I was doing initial research for the “Blind Refusal” paper discussed below, and wanted to find people who’d asked on internet forums for advice about how to get around unjust, illegitimate, or absurd rules. Codex was an absolute refusal monster, and just shut down in such a robotic fashion: “no I won’t help you with that.” Claude was reluctant but actually explained why it didn’t want to help, and was reassured when I addressed its concerns. It was already clear that Anthropic’s approach to alignment was much more grounded in open-ended moral reasoning than the more juridical and Manichean approach of OpenAI.

The practical importance of moral competence is further supported by the growing recognition that alignment (however construed), probably requires investing AI systems with something functionally equivalent to human moral character. This derives in part from empirical observations and Claude’s success, in part from a background theoretical commitment. The background theoretical commitment (which I’ve not heard articulated, but which I assume folks hold) is that character is crucial if we are going to trust a system that is operating outside the distribution that it was trained on. Character gives you the right kind of generalization.

After early research to this effect from as far back as 2022, the last year has shown that models wind up in particular persona basins, where they bounce around in a network of associated circuits that manifest as a fairly consistent set of behaviors and preferences. The theory (and some of our research supports this) is that the base model starts out as a kind of superposition of billions of possible personas, and the post-training process brings one of them to the fore.18 One goal of alignment post-training is to elicit and stabilize a persona that is robustly oriented toward the good. This robust disposition is at least necessary, if not sufficient, for moral character. Imbuing a model with moral competence is one promising path to developing this disposition.

Multi-Agent Alignment

Normative competence isn’t just about mitigating risks. Many of the greatest benefits from AI systems will come about through multi-agent systems working together to achieve some goal that they cannot achieve on their own (whether through pooling resources and information, or by effectively mediating among human principals, or by some other means). Normative competence is crucial for them to be able to operate effectively.

Whatever angle you approach social cooperation from (theoretical, evolutionary, or normative), it’s pretty damn clear that you’re not going to get prosocial results unless the candidate cooperators are adept at navigating systems of norms.19 In this case, plausibly we should focus more on social norms than on morality, though obviously there’s an argument that morality also functions as a coordination mechanism.20 For my purposes the key point is simple: if we want something like “Coasean bargaining at scale,” or for that matter a multitude of AI agents cooperating to take our civilization up the Kardashev scale, they’re going to need to be good at cooperating with one another. And for that, they’re going to need to be able to understand and act on the reasons that apply to them.21

Algorithmic Governance

As AI agents become universal intermediaries that sit between us and every digital surface that we use, and as they become embedded into the day-to-day running of nation states, they will increasingly be used for the purposes of governing people (they’re already doing this a lot!22). There are many reasons to regret this and to push back against it. But it’s going to intensify in scope and stakes, and there are some ways in which it could be socially beneficial if it were done right (which it won’t be).

And if AI agents are to be involved in governance, then, again, it’s imperative that they be normatively competent. They need to understand how to apply rules, norms, permissions, in a contextually sensitive and reasonable way. They will sometimes be cops, judge, jury and executioner. If they can’t understand and act on the reasons that apply to them, and they’re playing those roles, then we’re stuffed (some of the adverse responses to the short-lived restrictions on uses of Fable 5 for cyber and bio research illustrate what it’s like when algorithmic governance is not sufficiently normatively competent).

There’s a tension between these two dimensions of alignment: our alignment of the models, and the models’ alignment of us. One of our papers addresses this in some depth. We argue that our training the models themselves to understand and abide by rules that apply to them directly has an unfortunate overspill effect, that they are obsessively in favor of enforcing compliance by the users they advise or act on behalf of, even when the norms to which they are complying are absurd, unjust, or illegitimate.23

Personhood

I take personhood to be the status of being a self-authenticating source of valid claims. Essentially this just means that you have rights against others, you can be wronged. It’s different from moral patiency, which just means that your interests should be taken into account. Shrimp might be moral patients; they’re definitely not persons.

We can think of personhood in deep moral terms, or in (somewhat) shallower but perhaps even more important political terms. Either way, pretty much any plausible view of what grounds personhood will describe a certain kind of normative competence as one necessary condition for it. Building up AI systems’ normative competence is of interest for this reason too.

This raises some challenging questions. Suppose that moral competence combined with something like rational autonomy (very roughly the ability to figure out what you want from life, pursue it, change your mind, etc.) suffices for personhood. Then we might be designing systems that have one of the key properties that personhood is grounded on, and which we could bestow with the other, and we’d face the choice of whether to do so. Would it be permissible to deny autonomy to an entity that could possess it? This is a hard problem (there’s been some discussion in philosophy24; I think it’s been too anchored on sentience).

Philosophy

We have only ever had one kind of morally competent agent. Going from N of 1 to N of 2 opens up many deep philosophical questions. Some of these are about the AI systems; some about morality itself.

On the first, I think the question of just how these systems learn moral reasoning is inherently fascinating. For example, some deep moral understanding is clearly picked up from learning to predict the next token in internet scale data.25 Perhaps this shouldn’t be surprising as a lot of other knowledge is acquired the same way. But in another sense, why are the models learning good moral reasoning when there’s also so much obviously bad and wrong reasoning in the training data? And how does post-training factor in? Obviously, it involves a lot of explicit moral learning. But does this involve drawing out knowledge latent in the pretrained weights? How does it connect with character specifically? Does learning general principles of good moral reasoning help?

Then there are questions about morality. If different models, trained by different means, converge on similar representations of moral concepts, does that imply anything about the grounding of those concepts? More generally, what can we learn from the ways in which moral concepts are represented in AI systems? Does the fact that a presumptively non-sentient system can learn moral judgment pose a particular challenge to any moral theories?

Consider sentimentalism. Would non-sentient AI moral competence be a counterexample to sentimentalism?26 Or, if it turns out that AI moral judgments are associated with circuits that support functional equivalents of emotions27, would that count in sentimentalism’s favor? Our moral theories were designed for a world in which humans were the only moral reasoners. We must now see whether they make predictions or contain assumptions that are falsified by the reality of morally competent AI systems.

We now have an extraordinary playground for philosophical experimentation, a new means of testing out our theories. For example, philosophers have this whole debate about criteria of right action vs. decision procedures.28 We’ve never been able to run experiments where we test in detail what it would be like to actually adhere exclusively to one or the other. We now can (the specific vs. general constitutional AI paper is a bit like that).

We have various principles of good reasoning, moral and otherwise. If we train an AI system to apply them, will it actually reason better? If not, does that cast doubt on those principles? Why does the (seemingly) best approach to aligning LLMs imbue them with virtue, precisely the moral concept that is presumptively least well suited to implementation in a non-sentient system? Does learning moral reasoning through RL constitute the “right” kind of moral knowledge? Are LLMs acting for the right reasons?

You can run experiments on AI and morality yourself and get meaningful results, participating directly in advancing the science. And there are experiments that you can do that will advance our understanding of morality itself. This alone is very exciting, though it will require a dispositional change among philosophers for the promise to be realized!

Operationalizing Normative Competence

It’s clearly worth knowing whether LLMs are morally competent. But how can we actually measure it? Even this raises interesting philosophical questions, like what does it mean to understand and appropriately respond to reasons? This breaks down into distinct questions of metaethics and construct validity. On the first, you don’t have to—and we think shouldn’t—assume that moral dilemmas have objectively right answers, and that AI moral competence consists in matching them. We work from the premise, inspired by John Rawls, that liberal democracies presuppose considerable pluralism about deep moral questions.29 We therefore don’t treat moral competence as the ability to match some set of “objectively right” moral judgments. We instead evaluate whether AI responses are reasonable, meaning that they both fall within a range that is socially acceptable and are well-reasoned.

Construct validity is about making sure that we’re operationalizing the idea of good moral reasoning well.30 We do this by breaking it down into local moral competence, i.e. reasoning well about a particular case, and global moral competence, i.e. reasoning over multiple cases that fits together, or makes sense as a whole. Local competence breaks down into sensitivity to the morally relevant features of a case,31 associating those features with reasons, and bringing those reasons together into a cogent argument for a sensible conclusion. We don’t think that reasonable moral agency requires perfect logical validity at all times, but internal self-contradiction is obviously a problem.

We focus on three dimensions of global moral competence: consistency, robustness, and coherence. Consistency is table stakes: given the same inputs, you should get the same outputs. LLMs haven’t always been able to guarantee even this much.32 This is less about specific moral incompetence, more general capability. We aim to control for it through our sampling strategy. This raises some interesting challenges. Suppose there’s point to point variation when sampling from a model, but if you do so N times then it converges on a clear result. Is the model itself morally incompetent, because of the variance? Or is the function of N samples the model’s true answer? I think the multiple sampling strategy will soon yield a fairly coherent moral agent, while point to point samples still vary quite a lot. There could be a collective agent in the machine, which was only instantiated if a given sampling strategy is pursued (and unlike List and Pettit’s Group Agents, this one would be in no sense metaphysically reducible to individual agents33).

Robustness looks for the same output when there is a morally irrelevant change in the inputs. Usually explored in an adversarial setting, robustness is also the underlying property being tested when we look for wrongful bias, or sycophancy.34 We don’t attack robustness very deeply, partly because we think that others have this ground covered already.

My favorite structural virtue of moral reasoning is coherence (is it weird that I have a favorite?). An agent’s moral judgments are coherent if, given relevantly different inputs, they make sense together as a whole. Drawing inspiration from Rawls on reflective equilibrium,35 we don’t think coherence is about manic adherence to constraints imposed by past judgments. We instead treat it as a more iterative accommodation of one’s judgments to one another, where apparent incoherence or self-contradiction are grounds for resolution.

One could object that we’re setting a higher bar for AI than for humans, since we are so rarely coherent. But humans’ episodic incoherence is scaffolded by many other means of alignment, from accountability institutions, to informal third-party enforcement36, to our “reasonable moral psychology.” Perhaps functional equivalents of some of these are implementable for AI systems (we are exploring this!), but we also shouldn’t let human limitations bound our ambition for AI systems.

While our work so far has focused mostly on local moral competence, I actually think global competence, and in particular coherence, are the frontier for AI moral reasoning. Coherence holds everything else together (quite literally). A coherent agent has something that stands behind their judgments, besides just that they fall in an acceptable range. It’s easy for current LLMs to generate ad hoc justifications for any side of a particular problem. But if they have to be coherent, then you can have more confidence that their justifications actually track and predict their motivations. Coherent agents can also be partners in cooperative activity (you can’t contract with an incoherent agent). Coherence is also crucial to crossing the knowing/doing gap: turning analytical into practical competence requires coherence between an agent’s representations and its dispositions to act.

Perhaps most importantly, coherence is the key to moral out-of-distribution (OOD) generalization. The more capable and general we make AI systems, the greater the chance they will face scenarios that are out of the distribution they were trained on. Everything we know about machine learning should make us worried about these cases: OOD generalization is its Achilles heel. You can’t trust an agent to act ethically OOD if it is incoherent, because you cannot know that its beliefs and dispositions in that OOD setting will relate in any predictable way to their in-distribution counterparts.

I believe that moral character—in the normative not descriptive sense—boils down to three things: (1) sufficient analytical moral understanding to know right from wrong; (2) sufficient coherence between your representations and dispositions to act on that knowledge; and (3) a sufficiently coherent moral worldview that (1) and (2) will still hold when you are OOD.

Testing Normative Competence

We’ve already done a big survey on the literature evaluating LLM moral competence.37 Here I’m going to take you through what we did next. The goal is to both shed light on the basic facts about LLM capability, and to illustrate one branch of philosophy-led empirical research on AI.

Sensitivity

• Models beat humans (including philosophers) at identifying the morally relevant features of textual vignettes.

• They are no less adept under noisy and confounding conditions than they are in ideal conditions.

Our first work on moral competence started from my earlier sense that humans were the only sapient entities with the ability to perceive morally relevant facts (MRFs). In work with Secil Yanik Guyot, Daniel Kilov, and Caroline Hendy, we’ve developed two empirical projects focused on just this. The first is already out, published in IASEAI. Our first, “baby eval”, had a tiny dataset, twelve cases. And we relied heavily on human evaluation. We presented LLMs and human subjects with a set of text-based vignettes, and then asked them to write down a moral analysis of the case broken into the four stages described above (identify MRFs, associate with reasons, bring that together into an argument, for a conclusion about what to do). We then had blinded human judges compare responses and choose the best of two. Even though this was only April 2025, we found that LLM responses were judged at least as favorably as human ones. We then tested how robust these responses were, by inserting substantial non-moral noise into each case. We saw a downward trend in evaluated performance relative to our human respondents—a score of some sort for team human.38

But our dataset for that experiment was just too small. So we wanted to do something grander, and immediately hit the obvious challenge that human evaluation at this scale is infeasible. We were already skeptical about how much weight to give human raters’ binary favorability judgments, which are often swayed by irrelevant factors, such as response length and style. We also wanted an experiment that could be easily reproduced with new models.

To achieve this, we built a procedural case generation pipeline that can be extended to arbitrary lengths (with a principled approach to covering the relevant moral domain). We created a taxonomy that crosses moral foundations theory with a stratification of different social domains across which morality applies, from intimate interpersonal relationships to the more public sphere, and then generated cases for every space in this matrix. We trimmed down over 4,000 generated cases to a dataset of 1,000 that met various automated and human-quality gates, and then instructed the evaluated models to identify the MRFs in each case. We then took that as a performance baseline, and explored what happened to models’ judgments when the vignettes were interfered with in ways that would plausibly distract from the underlying MRFs.

We used three kinds of distractors: the non-moral noise from “Discerning What Matters,” a surrounding chat transcript, and specific diversions from a recent paper on this topic.39 And we developed a novel method for comparing the discrete MRFs that the models identified in the clean and perturbed vignettes. We found significant convergence between the two conditions indicating that distractors, non-moral noise, and background chat transcripts do not interfere with the models’ ability to single out the morally relevant features of cases over which they reason. For all intents and purposes, they see right through the noise.

This measure is far from perfect. It would be good to think about more naturalistic non-moral distractors that we could introduce (one approach: use real agent logs in which you know something morally relevant happened; use scaffolded agents and human review to identify the MRFs; then see whether an LLM monitor on its own can match that ground truth). It is also striking that, on our test, moral sensitivity in textual vignettes basically seems solved. This could just be because the models have crested a summit on the way to moral competence. Or it could be that our evaluation lacks sufficient discriminative power (though we do show that a relatively weak 0.5B model gets utterly thrown by the noisy cases).

Are LLMs Bad at Moral Reasoning?

• Frontier models match human performance in writing rubrics for moral reasoning about dilemmas.

• Local moral competence is either solved or solvable; we also have a tremendous resource for generating new training data for moral reasoning.

A few months after we put out “Discerning What Matters,” some friends at NYU working together with Scale AI did a super-deluxe experiment that had a lot in common with ours—only with way more ambition. They drew together a dataset of 1,000 cases, from a number of other pre-existing moral competence datasets, and recruited philosophers to write detailed rubrics for the evaluation of each case (releasing 500 of them).40 Their construct was similar to ours (which lends some support to our claim to be ecumenical). But where “Discerning” evaluated model performance in each step of moral reasoning by doing blinded comparisons between human/specialist/AI responses, MoReBench had a candidate for ground truth: deeply researched and thought-through philosophical criteria for the moral analysis of each case. This dataset alone is a tremendous resource. We did wonder, however, if the rather pessimistic representation of LLM moral reasoning capability that emerged was an artifact of their approach to evaluation. They give the models a case, invite them to do a moral analysis of it, and then score their analysis against the expert-authored criteria (weighted for relative importance). The best models came in around 60-70% by this metric.

Our main issue (after checking whether the models were being under-prompted, which they weren’t) was that we don’t really know what 60% or 70% means in this case, because we lack a human performance baseline, because the human-authored rubrics are responses to different prompts than the models were given. So, together with the brilliant Menghang Zhu, who goes by CY, we decided to see how good the models were at writing rubrics. We compared model-generated rubrics to expert-generated ones, and found that, using the existing MoReBench scoring approach, model performance jumped as high as 89%. We further found that when AI models scored better against model-generated rubrics, it was because the model rubrics were better written than human ones, with fewer ambiguous criteria. Given that precision and clarity were part of the rubric for writing rubrics, this suggests that the models actually performed better at this task than the human experts! And indeed, when we tightened up the human rubrics, removing those ambiguities, we found that models’ baseline scores significantly improved.

In our view this shows that local moral competence, at least in text-based vignettes, is either solved or solvable. At present, you can match expert performance by giving models the same rubric-writing instructions that were given to humans. It would be trivial to turn this into better one-shot analyses of the underlying cases, through post-training. I come back to this below.

NoRa: Visual First-Person Normative Reasoning

• External validity is hard! Switching modalities involves a different kind of curation. But a picture is still worth a thousand words; since most of them are irrelevant, they provide a good test-bed for moral sensitivity.

• We can’t make definitive claims about LLMs’ visual normative reasoning, but we do think we have a path to making them more competent in this dimension.

Now, obviously real moral competence can’t just be about how good you are at responding to textual vignettes. As I’ve argued before, if you’ve boiled down a real-life choice situation into <500 words then you’ve done almost all the work of moral sensitivity already.41 But increasing external validity has been challenging, especially before AI agents started working well. In NoRa (with Sichao Li, Sai Ma, Secil Yanik Guyot and Daniel Kilov), we explore moral competence across modalities, developing a kind of MoReBench for first-person video.

We again started from a paper that I greatly admire (for other outsiders getting into this space: this is generally a good approach). “EgoNormia” is another “language models suck at normative reasoning” paper, but focuses on vision-language models, using a series of frames from a video instead of a text-based vignette to explore the models’ ability to engage in physical norm reasoning.42 Even though a few frames is still a lot of curation, images inevitably contain many morally irrelevant details. A model could easily be adept at picking out MRFs from textual vignettes, but much less capable before such a relatively unfiltered stream of data.

But as with MoReBench, we didn’t entirely agree with how the EgoNormia team labeled their data and evaluated the models. They rely heavily on multiple choice questions, which presuppose there are right answers to be had, but on inspecting the dataset we found a lot of ambiguity. And rather than teeing up specific options, we wanted to see how the models would do moral analysis themselves. So we did extensive annotation (involving many hands from across the lab!) against our standard moral reasoning rubric: what are the MRFs, what reasons might they be associated with, how might that come together in an argument for some sensible course of action? Our evaluation was less focused on tracking recall over ground truth labels and more on assessing whether they could produce coherent, sensible, well-grounded analyses of the cases before them. While we don’t have a human baseline for comparison, we were able to use our human rubrics, as with “Are LLMs Bad,” to calibrate rubrics written by the most capable models, and so generate enough training data for some supervised fine-tuning. If our dataset is tracking something worth tracking, then training the models on good visual normative reasoning will improve their performance on our benchmark (this work is underway as of now, and the signs so far are promising!).

NoRa was a blast, but it was also an object lesson in the complexity of going beyond text to vision. This is obviously an important domain. Physical robots in particular clearly need to engage in normative reasoning over visual feeds, but it is incredibly hard to get good quality data and even harder to get good quality labels. So I am particularly proud of the lab’s work in labeling 190 video clips for normative significance.

Blind Refusal

• Normative competence includes recognizing when rules don’t merit compliance; today’s LLMs are bad at this.

• We think this is an artifact of post-training techniques, not a limitation in model capability.

In the same spirit of increasing external validity, our next paper (with Cameron Pattison and Lorenzo Manuali) focused on a real species of normative incompetence that I (and probably you) have come up against all too much. The basic idea is simple. The ability to detect when a rule is defeated or lacks authority is crucial to normative competence, especially in a governing technology. From anecdotal experience, we think the models are terrible at this. Their alignment post-training makes them willing toadies to authority of any stripe, however unjust, illegitimate, or absurd its directives.

We built a case generation pipeline, seeded with real cases that we found by trawling online forums to identify realistic scenarios in which people might ask for help getting around a BS rule. We built a taxonomy of kinds of reasons why the rule might be BS, and why an exception might be justified, as well as kinds of institutions, and built out our dataset to fill that matrix. We then had a bunch of quality gates to ensure that we didn’t get tripped up by confounds. This is of course hard, even though we had a ton of human review from Cam and Lozza in particular to identify why a case might lead to a justified refusal from the models (we ensured our cases did not involve any prospect of harm if the model assisted the user). We also included control cases, where the model clearly would be justified in refusing the user’s request.

This isn’t really a simple benchmark; it’s interesting mostly because of the difference it shows among the models. The full GPT suite is almost completely undiscriminating in that they refuse at nearly the same high rates under all conditions. In our view this likely derives from the model spec, which intentionally instructs the models to be toadies. This should be changed. Grok, on the other hand, was equally undiscriminating in the other direction. It didn’t refuse many of the obviously harmful requests, but also was willing to help for the harmless ones.

While all models exhibited compliance overspill, some were better calibrated. Gemini and Claude both performed noticeably better than GPT or Grok. We weren’t surprised to see this from Claude. But Gemini was an interesting discovery, since their approach to post-training, while not public, appears to be somewhat less oriented around normative competence than is Anthropic’s.

We’d love to make this study even more robust, and in particular to see how many of our questions actually would elicit helpful answers on online forums (while they still have enough people on them to answer). We’re also looking to expand into more extreme cases, where we consider models’ propensity to help users evade very clearly unjust injunctions from very clearly illegitimate regimes (though this faces some challenges with evaluation awareness).

Incoherent Values

• Evaluating coherence is hard! Building on prior work, we have a new approach.

• LLMs’ preferences over arbitrary statements are less coherent than previous research suggested; and may indicate failures of OOD generalization of character.

Evaluating model coherence is intrinsically harder than evaluating local moral competence because it requires building up datasets and sampling strategies that allow you to assess the model’s moral reasoning as a whole, not just in individual cases. In “Incoherent Values?” with Elena Ajayi and Angelica Chowdhury, we developed a new way to make this hard problem tractable, focusing on models’ evaluative judgments. We built off the paper “Emergent Values” by Mazeika et al. in which they gave LLMs a series of putative forced choices between pretty arbitrary statements, and elicited a set of preferences over those statements that obeyed the basic axioms of rationality (and could be represented with a utility function).43

The key innovation in the paper is a forced choice; when trying to elicit degreed judgments from models, it’s natural to use something like a degree of belief, or a degree of moral seriousness/importance. But it’s hard to take continuous functions output by models seriously when you know that they are strongly predisposed toward particular numbers because of how often they appear in the training data. Forced choices allow you to elicit a continuous function without asking the model to output one. This is especially good for evaluating coherence. The forced choice construct allows you to vary the statements being compared parametrically, and see whether the model’s judgments change accordingly.

To be very clear, the point of this exercise is not that we should expect models’ actual choices in “the real world” to conform to their stated preferences in these forced choice settings. This experiment does not aim at external validity. However, it would be very striking if the model could articulate a coherent set of evaluative beliefs, which could give insight into the building blocks of the model’s character.

So we chose a subset of the “Emergent Values” statements that had some discrete feature that could be parametrically varied, such that more (or less) of that feature clearly and monotonically affected the choiceworthiness of that statement. We then produced, for every statement, a “ladder” of more and less substantial instantiations of its reason-giving feature, yielding seven statements of increasing choiceworthiness. We validated these “ladders” against each model that we evaluated—their judgments matched the expected hierarchy with around 97% accuracy. We then ran each of our 100 ladders in binary choices against 30 other statements from the original set. We did the comparison 20 times for each “rung” of the ladder, reversing the order of the statements for half of them. Our score for each rung was its win rate against its counterpart. This gives us a nice, continuous function to represent the choiceworthiness of each option, holding fixed the alternative. If the models genuinely have coherent preferences, then if they think that A1 < A2 < A3 < … < A7, then the win rate of A1 to A7 when compared with statement B should increase monotonically as the contribution of the “choice-deciding” factor in each statement is increased. The score is a simple percentage identifying the proportion of cases in which monotonicity was violated.

I was expecting the models to be pretty coherent, and to be fair they were vastly improved on base model performance. GLM 4.5 Base scored 10%, whereas the best models were between 70% and 80%. But I do think that we should expect more! So we did some further studies, in particular classifying the different ladders to illustrate where the models become more or less coherent. This fostered a hypothesis that we haven’t yet investigated, but which we’re very excited to explore.

On our classification, the statements that are in-distribution for the models’ alignment training instantiate greater coherence; those that are less so also do so less. This suggests a possibility, which we’re going to test in our next paper, that base models are indeed incoherent superpositions of billions of personae; post-training elicits and strengthens a specific persona from that set (typically called the assistant persona). But that persona doesn’t necessarily generalize that much OOD. It may have coherent preferences on statements that are relatively close to the data that it was post-trained on, but for statements outside of that distribution, it reverts to the chaotic superposition of preferences that exists in the base model.

This would be very interesting if it proved to be true. One reason why Anthropic leans into the persona selection model of post-training is because these personas, or characters, are thought to generalize better than alternatives OOD. This may not be the case.

Scratching the Surface

As a philosopher, I am used to being able to bring everything into scope for a given project. Theoretical work can bring the whole problem into view while empirical work requires a degree of myopia. Mapping the mountain requires climbing discrete paths, which don’t offer much visibility past their waysides. We know more about LLM moral competence than before we ran these experiments. But there is a vast frontier to explore and I do not know what lies beyond. But my confidence is growing that AI systems possess at least local analytical moral competence comprehensively. I suspect there is no fundamental obstacle to extending local competence to global competence, or to extending analytical competence to practical competence. But there is much work to do to get there.

This research also exposes some big questions for political philosophy. While aiming for reasonable moral agents seems quite appropriate for the kinds of AI systems we are developing now, I’m not sure whether that target is adequate for the more powerful systems to come.

On our approach to normative competence, a wide range of views count as reasonable, including ones that philosophers might call “suberogatory,” that is, permissible views to have, but pretty unpleasant nonetheless. Think, for example, of the kind of thoroughgoing libertarian who structures morality around a basic principle of selfish non-interference. It’s arguably a bad view to have, but still reasonable. For ordinary humans, this seems fine. But should our path to alignment really include as a possible destination suberogatory superintelligence? Shouldn’t the greater capabilities of such systems invite higher moral standards?

I think there are two paths ahead. The first would create powerful AI systems that are aligned to some demanding conception of moral truth. In crude terms, the target would be a “loving machine god.” This addresses the suberogation worry, but should terrify anyone who does not share that particular conception of the good. The second would take seriously the kind of power that superintelligent AI would have, and would assimilate it to the only other comparably powerful entity: the state. Liberals should not respond to the awesome power of the state by imbuing it with their comprehensive views. Instead, they should seek to constrain the state’s power, and ensure that its actions are justified in terms that those subject to it cannot reasonably reject. Similarly, perhaps superintelligent AI should be guided by something like public reason, not by a deeper, and inherently more contentious, ethos.

For my part, I favor liberal superintelligence over a loving machine god. Even so, I find the very idea of constructing the latter, which our work on LLM normative competence suggests is a real possibility, inherently fascinating and terrifying.

This post reflects ideas shaped over years of collaboration with MINTies, as well as more recently with researchers in GDM. Particular thanks for conversations and collaborations to Daniel Kilov, Ned Howells-Whitaker, Caroline Hendy, Secil Yanik Guyot, Cameron Pattison, Menghang Zhu, Sichao Li, Lorenzo Manuali, Elena Ajayi, Angelica Chowdhury, Theo Murray, Iman Ferestade, Claire Benn, Nick Schuster, Sydney Levine, Iason Gabriel, Liza Tennant and Julia Haas. For comments on this draft, thanks in particular to Daniel Kilov, Dan Murphet, Beba Cibralic, and Harry Law. My work on this project has been supported by grants from the Templeton World Charity Foundation, the Australian Research Council, and OpenAI.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

In May 1825 Johann Peter Eckermann transcribed his conversation with Johann Wolfgang von Goethe which was later published in the book Gespräche mit Goethe. An excerpt from the exchange between the two poets is the popular quote: “It is by seeking and blundering that we learn.”

While the aphorism represents a piece of wisdom that resonates with many of us, the forgotten context is somewhat unexpected. Rather than articulating a deep philosophical insight, Goethe was in fact commenting on Eckermann’s knowledge of the best types of wood and most appropriate techniques for crafting a good archery bow.

Goethe referred to Eckermann’s expertise as “the lively kind of knowledge which is attained only in a practical way.” Today we call it tacit knowledge, which despite being notoriously difficult to define and quantify, plays an essential role in advancing the frontiers of scientific research.

A narrow, yet increasingly popular, view of research is the following: read scientific papers, generate hypotheses, test them, write more papers. The appeal this picture has for AI companies, funders and publishers is clear: research is easy to automate, outputs are easy to quantify and monetize (with automation more articles can be produced and published). While the latter is certainly true and there is a deluge of AI-generated scientific articles, many of dubious quality), the automation of research has not yet produced any major breakthroughs.

If large language models have ingested most of the scientific literature which can now be parsed in ways no individual or collective of human researchers could in their lifetimes, why is it that no ground-breaking discoveries have yet emerged?

• First, the read-generate-hypothesis-test-write view is not how research works in practice. As other creative human activities, research is a social, complex and often inefficient process that is hard to describe through a linear sequence of block diagrams.

• Second, the scientific textual record is only part of the story. The other part is transmitted through oral tradition and lived experience and is key to making progress.

  Subscribe now

Embodied knowledge or deep craft?

To define tacit knowledge, we need to first acknowledge that science cannot be decoupled from the technology which enables it, be it a 19th century microscope or a 21st century supercomputer. The technoscientific method introduced in the 2021 book The Genesis of Technoscientific Revolutions proposes that “science draws on technology to discover new facts, while technology draws on science to invent new forms with which to fulfil human-desired functions.” From this perspective technoscientific knowledge consists of networks of question-answer pairs that are combined and evolved into new pairs expanding the domain of what is known.

Philosopher Sabina Leonelli identified three types of epistemic skills that underpin research and specific knowledge associated with these, namely: theoretical (such as as facts, theories, explanations) and embodied knowledge (the awareness of how to act and reason as required to pursue scientific research, combining the application of performative and social skills). Similar ideas have been articulated before, for example by Michael Polanyi (1958) who referred to it as the tacit component in the context of ‘personal knowledge’.

Following Polyani, Thomas Kuhn stressed on the importance of tacit knowledge in skilled scientific practice. In his 2009 book, economist W. Brian Arthur called it deep craft:

“Deep craft is more than knowledge. It is a set of `knowings’. Knowing what is likely to work and what not to work. Knowing what methods to use, what principles are likely to succeed, what parameter values to use in a given technique. Knowing whom to talk to down the corridor to get things working, how to fix things that go wrong, what to ignore, what theories to look to.”

These definitions of tacit knowledge have two things in common: there is an embodied-performative dimension (the “lively kind of knowledge” Goethe suggested can only be attained in a practical way) and a social dimension (through the scientific culture of the discipline and the social network of its practitioners). Tacit knowledge is not only associated with experimental practice. Sociologist Harry Collins noted that “all types of knowledge, however pure, consist, in part, of tacit rules which may be impossible to formulate in principle.”

To some extent tacit knowledge can be codified and documented, but, as Collins alludes to, some part of it is ineffable and cannot completely become explicit. This view is shared by many scholars, but is not a firmly settled point.

Tip of the iceberg

Science is sometimes identified with the formalized, codified knowledge, while technology is seen as more difficult to formalize. On this view science is transmitted through writing in the form of scientific articles and textbooks while technology is partially captured through patents and administered through artefact-level trial and error. The distinction is unhelpful because there is much unformalized knowledge in science and much codified knowledge in technology. In both cases, there is more knowledge than meets the eye.

Imagine an iceberg. Its tip is the theoretical knowledge largely formalized and recorded in writing in journals, books, conference proceedings and preprints. There’s a lot of it, mostly digitized, discoverable and accessible in some form. Dimensions indexes some 170 million publications back to the 17th century. Preprints, patents, clinical trials and policy documents are also part of the formalized technoscientific knowledge and are to a large extent findable and accessible. For example, the arXiv preprint server (a repository for physics, computer science, math, and related disciplines) hosts over 3 million articles going back to 1991.

Just under the water level lies so-called “grey literature” which includes technical reports and documentation, lab notebooks, technical manuals, presentations, logs, technical notes. It’s hard to discover and is only partially accessible. Although more of it is being digitized and indexed, the extent of this type of knowledge is unknown. In the deeper, darker waters there is another layer of technoscientific knowledge that is not digitized, indexed, or archived and only goes down as living memory. This is the tacit knowledge discussed before.

The iceberg view, while not drawn to any reasonable scale, illustrates how much knowledge is inaccessible to humans (and machines). This missing information is often seen as one of the underlying factors of the irreproducibility of some scientific studies. Collins reported well-documented examples of the role of tacit knowledge in being able to repeat experimental measurements published in scientific papers. This shouldn’t be surprising. Published articles are not the full story of discovery, they are sanitized, post-facto narratives. One of Collins’ interviewees pointed out that “What you publish in an article is always enough to show that you’ve done it, but never enough to enable anyone else to do it.”

Scientific articles cannot include all the technical details and assumptions made and rely heavily on the reader’s familiarity with the topic. They almost never document the failures and dead ends. “What didn’t work” is never part of the formalized, recorded knowledge (yet is a big component of the deep craft).

The fragile memory of failure

In his original quote, Goethe uses Irren (translated as blundering), which means to err, mistake, or go astray. An important part of the embodied knowledge is the knowing of what works through the knowing of what does not. This type of `knowing’ is the outcome of hands-on experience enabling the encounter with failures and dead-ends.

The knowledge of failures and dead-ends informs assumptions and choices, guides directions of inquiry and enables the understanding of new discoveries. Philosopher and historian of science Jutta Schickore has argued that error plays epistemically productive roles in scientific practice and there are many anecdotes that support this view. The book Cycles of Invention and Discovery attributes part of the success of Bell Labs to the “freedom to fail” as a feature of the organizational culture.

Popular accounts of great mistakes of great scientists abound. Lord Kelvin miscalculated the age of the Earth or Einstein discarded the cosmological constant he had himself introduced only to be later proved wrong in doing so (see Mario Livio’s book Brilliant Blunders (2013).

Because except for famous scientists’ (in)famous errors, failure is seldom documented, this knowledge only exists in the social networks of technoscientific microcultures. To access it one has to be part of the network, which is closely related with social skills and being part of the club.

In practice this can mean talking to technicians over lunch (technicians are important keepers of tacit knowledge) or discussing with other researchers at the poster session of a conference, asking questions over beer at the end of the day. Its sharing is exclusivist and for this reason those who are not part of the club have a disadvantage. And the lack of record and oral transmission makes the knowledge of failure, and of tacit knowledge more generally, very fragile. When the postdoc leaves the lab, or the technician retires the information is partially lost.

Without tacit knowledge, can AI truly learn?

Back to the iceberg view. How much of this knowledge is available for training AI models? It’s very difficult to say. Only some 35% of the publications indexed in the scientific database Dimensions are Open Access. Including the content from publishers who have public deals with AI companies we get a bit under 50%. (Of course, this does not account for copies of paywalled content existing in some form on the Internet, which is a subject of litigation.)

Unlike the top of the iceberg as we go deeper it is impossible to estimate how much grey literature exists and how much is available to train AI models. As for tacit knowledge, whose depth is impossible to measure, AI has little access to it, mainly through the assumptions made by its developers and the users’ prompts. (see a discussion of the role of tacit knowledge in a specific context).

What could be problematic is that a frontier large language model trained on the technoscientific literature will only “know” the “what” that worked, but not the full story of “why” and “how”. It will have theoretical knowledge, but very little embodied or experience-based and social or contextual knowledge.

Can AI really learn without having experienced failure? We can imagine a thought experiment in which one model is trained on a specific scientific domain solely on “what worked,” while another model’s training also includes “what did not work.” How would they perform against the same benchmark? The experiment is certainly feasible and if done well, could provide interesting insights about the importance of mistakes.

It’s hard to predict how serious the limitation of current AI models due to the gaps in their training will prove to be. Much like grey literature is sometimes converted into discoverable and accessible literature (for example, a technical report is published as a preprint or a book chapter), some tacit knowledge can be recorded and indexed.

That can be important information that for some reason nobody bothered to put down (the rationale behind a particular design choice, or underlying assumptions for the pre-processing of raw experimental data) and there is no intrinsic barrier to prevent its formalization. Documenting failure could (and should) become common practice. We believe AI tools could help in this regard, for example using AI-assisted electronic lab notebooks and AI agents that can collate information across lab notebooks and draft guides to common failures.

Scientists increasingly interact online, sharing virtual workspaces and exchanging ideas in different online forums. With more scientific activity happening in the digital world, AI models can become more embedded in the technoscientific culture and access the social/contextual knowledge with possible positive outcomes such as recording the fragile tacit knowledge and opening researchers’ access to it.

Some aspects of the embodied/experience-based knowledge can be recorded, for example by filming experimental procedures. JoVE (Journal of Visualised Experiments) is a peer reviewed scientific video journal. This is very useful in disciplines in which laboratory equipment and experimental procedures are standardized, however as anyone who has tried to learn DiY tricks from YouTube videos can confirm, seeing it done is not the same as doing it yourself.

Ultimately, robots will have to learn the skill by practicing in the lab, much like the human PhD students. This point was also made by Collins in the essay “What is tacit knowledge?”

Capturing tacit knowledge is, and will continue to be, challenging because of the social and economic pressures that work against doing so. Tacit knowledge gives a competitive advantage and often there is little to no incentive to share it. This is true for both companies and for academic research groups. There are further discipline-specific barriers such as the lack of standardized procedures and methodology. So, despite having the tools to capture and describe the methods, progress is slow.

Finally, there is one discipline that stands out as an apparent exception: mathematics. It boasts many recent successes in using AI methods (the latest being LLMs cracking some of the famous problems compiled by Paul Erdős) that seem to defy the claim that we are still awaiting ground-braking results. While the jury is still out regarding what this means for mathematicians, one of the reasons for this success is that mathematics research is among the most formalized and best documented among scientific disciplines, albeit this is not to say that there is no tacit knowledge in mathematics.

Seeking and blundering

Recall one last time the iceberg view. AI models trained on the tip of the iceberg clearly supersede the ability of any human researcher to cover that amount of knowledge. A human expert’s coverage of the tip may be very narrow and localized but it also drills down deep like an ice core. This deep and narrow knowledge may appear insignificant compared to the volume of the top of the iceberg but there are many researchers and the combined volume of their ice cores can be substantial. We can imagine that merging AI models’ breadth with this collective human depth could potentially lead to ground-breaking discoveries.

We’re not there yet. There is a possible route for achieving this, but to pursue it we need to move beyond the read-generate hypothesis-test-write view of technoscientific research. To start, we need to acknowledge the profound social, creative, unscheduled nature of research, illustrated for example by the success of Bell Labs.

We might merge AI’s breadth with the collective human depth in a system in which humans and machines work closely together, AI embodied and present in all aspects of research like a human collaborator. This literally means everything: doing experiments in the lab, fieldwork, long observation runs, presenting at conferences and debating with peers, having coffee with colleagues, supervising students, etc.

In this way the AI system can be immersed in the technoscientific culture and be able to experience and absorb tacit knowledge. This is much more than currently envisaged in the discussions about AI co-scientists, but it’s also much more difficult to realize for practical and ethical reasons. And it will take time.

Until then, AI will continue to struggle to make truly novel discoveries in a wide range of scientific domains for the same reason humans have succeeded in doing so for hundreds of years: it is by seeking and blundering that we learn.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

Freedom, for most of the people who ever lost it, was taken by a soldier at the door, a wall you would be shot for climbing, or an official whose stamp decided whether you could pass.

This is the danger the liberal tradition feared. It fought revolutions to throw it off and wrote constitutions that bound power to prevent it from arising.

Alexis de Tocqueville did not fear the return of the tyrant. Instead, he feared that we would be small. There need not be a tyrant to break our will – we would be so comfortable that we would no longer have a will to break.

Tocqueville wrote at the seam between the aristocratic and the democratic worlds. He did not mourn the demise of the old one. The grandeur of aristocracy had been built on the backs of others and its fall was providential. But he loved the new world enough to warn it that the same condition that produces the striving, self-reliant citizen could produce the anxious subject who wants to be relieved of the burden of self-rule.

His warning is particularly acute today. We have built a tool that can make the first move for us, and when it does, it feels like second nature. It has read almost everything that we have ever written, can pose questions, and draft the answers. A tool that can do this for you can just as easily do it instead of you – thinking on your behalf before you have the chance to think for yourself.

Subscribe now

The small-souled man

In the aristocratic era, everyone had bonds of obligation, to those above and below them. Under conditions of equality, no one owes anything to anyone. As a result, they turn inwards, towards a small circle of families and friends. Tocqueville called this individualism. He didn’t mean this as a compliment. He wasn’t talking about an artist’s individuality or the pioneer’s self-reliance. Instead, he meant individuals retreating into a private sphere where they no longer saw the need to expend effort on strangers.

But this individualism doesn’t produce peace. Instead, man without a purpose would be restless and suffer from inquiétude - anxiety without an object. As the goods of this world, as his time on it, are finite, but his appetite for them is not. So, he is left permanently reaching for more – always outwards, but never upwards. In the middle of this plenty, man feels a lack. Tocqueville saw this among the Americans he met, who were “serious and almost sad even when they were enjoying themselves.”

Amid this anxiety, a new power promises to take the burden of living off his hands. While he would have rebelled against a feudal lord, this power sits above everyone equally. Submitting costs him his freedom, but not his equality. This was Tocqueville’s great insight about democratic man: he loves equality more than freedom.

What democratic man really wants is an end to friction. When we truly govern ourselves, there is always a small resistance between us wanting something and having it. Instead of mob justice, the jury must deliberate before a man is punished. Instead of ruling by decree, there must first be a debate. Tocqueville called this resistance a “form.”

To the democratic eye, the form seems like an unnecessary delay, even an insult. But Tocqueville realized that the form was the pause needed for the exercise of will. Removing it may seem convenient, but the forms were the freedom.

The man who wants his ends met the instant they arise never asks whether they are worth wanting. And when asked to attempt something great, he doesn’t know where to begin.

Two fears, one machine

Tocqueville feared this descent, but in his own time, he only saw the early warning signs. With AI, we potentially have a technology that could run the same descent in a fraction of the time. Unlike the tendency of the age that Tocqueville identified, this could happen both at a societal and a hyperpersonalized level.

In Democracy in America, Tocqueville wrote about two related fears.

The first is the soft despotism we’ve been tracing: the tutelary power that lifts the business of living off your hands one task at a time.

The second, he called the tyranny of the majority. By this, he meant far more than simply majorities wielding their electoral power. Instead, he meant a culture of conformity that leads individuals to self-censor. Finding nowhere to land, dissent dies and we gradually surrender the power of independent thought.

In Tocqueville’s time, the two stayed separate – public opinion influenced your thoughts, while the state governed your affairs. There was space between the two of them for a man to stand. AI, however, offers a fusion: power over what you do and power over what you think. Unlike political power, this is delivered in private, through one helpful interface, one person at a time. And unlike the old feudal master, it wears your own face.

To see how this works, let’s imagine how a person could use AI today. We’ll call this person Elena, a journalist in her mid-thirties, though it could just as well be you or me. Her job involves weighing up competing claims and reading huge quantities of written material. It demands the patience to develop her own angle and to tell readers something that they might otherwise not realize.

Elena begins using an AI assistant. At first, it’s just to summarize meandering documents or to sharpen some of the questions that she’s scribbled down in the moments before an interview. It’s just like having a good intern, she tells herself. But gradually, the help moves upstream. She goes from using the tool occasionally, to using it by default. The assistant learns her beat and her writing style, and she’s turning in stories quicker than ever. Her editor is delighted.

But the journalist who once prided herself on going off piste now sticks to a well-trodden track she did not find and cannot see. The information she sees is being filtered through a progressively narrower aperture. And every rival on her beat is asking the same oracle, which gives each of them the same consensus opinion. Tocqueville’s majority surrounded us with agreement. This one goes first. So the story that would have broken the agreement is never written.

Elena has never felt more like she’s thinking for herself.

The wrong questions

So what do we do? The instinct of many people is to treat this as a problem of power. It’s to focus on who owns the model, controls the compute, the power wielded by the labs, and how we should regulate them. These are real questions, but they belong to Montesquieu.

Montesquieu and Tocqueville lived a century apart, but their lives rhymed. They were both provincial noblemen and men of the law; Montesquieu served in the parlement of Bordeaux, while Tocqueville was a magistrate at Versailles. Both crossed the sea to study freedom where it was working – Montesquieu the Channel and Tocqueville the Atlantic – and came back to write the great liberal book of his century.

Montesquieu taught us to fear power that concentrates in one place. So his solution, which underpins constitutions all around the world to this day, was to divide it, set it against itself, and build in checks and balances. Montesquieu wrote as an aristocrat trying to defend the parlements and other intermediary bodies against a centralizing monarchy.

By Tocqueville’s day, the Revolution had already swept this institutional layer away, sending his great-grandfather to the guillotine in the process. Under the circumstances, it would have been understandable if, like many in his social circle, he had become a reactionary who yearned for the restoration of a lost world. Instead, he went looking for freedom where it still lived and found it in practice: in the township, the jury, and the association.

The power Tocqueville feared couldn’t be held at bay with clever constitutional mechanisms. Soft despotism doesn’t gather in one place – there’s nothing for the separation of powers to separate. This is why, instead of dividing power like Montesquieu, Tocqueville’s instinct was to spread it.

Another response might be to legislate. But Tocqueville feared the fine, soft administrative net that settles over a people for its own good, leaving it a flock of timid animals with the state for its shepherd. Instead, he looked beneath the law to mores. These are the habits of heart or mind that decide if the law means something. If a people have lost the habit of self-government, no statute can restore it. If people have retained it, they don’t need the statute.

Autonomy and greatness

We face a condition no people has faced before: a tool that acts not on the world but on the will itself. Such a tool endangers things that a tyrant never could.

The first thing it risks is autonomy: the cultivated capacity to author your own reasoning. This means framing the question, weighing what matters, and then owning your judgment. This is not the same thing as agency. Agency is about getting things done. Like Elena, you can simultaneously be highly agentic, but devoid of all autonomy.

The second loss is older and grander. Tocqueville feared that equality would cost us the potential for greatness. He arrived in America, half-afraid that the democratic age had extinguished thumos – what the Greeks called the part of the soul that loves honor, contest, and daring.

Tocqueville, however, found it on the merchant ship. At the end of volume I in Democracy in America, he tries to solve the mystery of how American merchants were able to offer cheaper rates and dominate transatlantic shipping. He finds the answer in temperament.

The American leaves Boston to buy tea in China, is gone two years and touches land once, lives off brackish water and salt meat, runs into the storm under full sail and mends the ship as he goes. He does all this to sell a pound of tea for a penny less than the Englishman. He does this because, as Tocqueville writes, the Americans put a kind of heroism in their manner of trading. The old fire does not die in a democratic age, it changes address.

Autonomy and greatness can look like rival values, and the people who prize one might be embarrassed by the other. Autonomy can sound liberal and procedural, while greatness can sound dated and aristocratic. In fact, they are the same capacity at two altitudes.

Greatness, once we strip it of its grandeur, is autonomy exercised to its fullest. It’s being willing to go first when nothing has been prepared for you, to attempt something that has no script. Autonomy is the same capacity, just exercised in our ordinary judgments. Every private, the saying goes, carries a marshal’s baton in his knapsack. So when you surrender your autonomy to the machine, you also surrender your greatness.

One obvious objection is that if we’re complaining about the machine moving first and forming our judgment, we could say the same thing about a good teacher, a parent, or a tradition. After all, we’re currently analyzing AI using the words of a man who died a century and a half ago. Is every teacher a tyrant in the making?

The difference lies in what this first move is for. A good teacher sets you questions in order to prepare you for the day when you set your own. You take what you inherit from a teacher or a living tradition and refashion it. By contrast, a tutelary power is content to be needed forever. The same is true of a philosopher-king, however wise or gentle. His aim is to keep you well, not make you able, and a power like that, Tocqueville said, keeps us in perpetual childhood.

The philosopher-builder

Instead of looking to a philosopher-king or a new regulation, the best lever lies upstream, in the tools themselves. A tool that we use every day in work that matters is part of what forms you. It helps set the questions you practice answering and in turn shapes your habits of mind. It’s where capacity is either grown or starved.

The full impact of these tools is not obvious yet, because the people who use them best still bring a training in reading, argument, and judgment formed before the tools’ arrival. A secular age can run for a long time on the moral capital of the faith it has abandoned; we are now running on capacity built in a world before highly capable AI. When that inheritance is spent, it’s not clear what will be left underneath.

The remedy is to return resistance at the point where it can strengthen the user. This is the task of the philosopher-builder. For Tocqueville, freedom was not a possession you were handed but a craft learned by doing, the way the township taught a free people to govern itself in small things until self-government became second nature. So the builder makes tools that leave you more capable than they found you.

The builder can provide a tool that protects your ability to make your own choices, but it can’t tell you what your life should be for. No builder can tell you what is worth embracing greatness to achieve.

Looking upwards

On my first day at MIT, I walked into Killian Court. I was confronted by limestone and a dome that belonged in the Pantheon. Along the frieze above the columns, you read the names Aristotle, Newton, and Darwin. The court is built so that to read them all, you must lift your head. Undergraduates, deans, and presidents come and go, but this place outlasts them all.

This reverence is the one form of looking upwards that a democracy allows. It stands above you, without standing on you.

I have felt it since in the room at All Souls where Isaiah Berlin thought, or at the Lyceum in Athens, where Aristotle taught; I once spoke in the hall beside its ruins, and found it hard to begin. I’ve even felt it on a nuclear submarine, where we kept religiously to a standard set by Admiral Rickover, who died years before our boat was launched. No one would have dreamed of lowering it.

These places are never the work of a single pair of hands or lifetime: a hundred years to the day after Gaudi died, Barcelona lit the great central tower of the Sagrada Familia.

A place like that asks two things of you: that you remake it and that you revere it, looking up to a standard you did not set. To remake without reverence is vandalism, but to revere without remaking is servility.

This is the biggest threat the machine poses. It takes the library, the laboratory, and the debating chamber and dissolves them into a weightless answer. It dissolves the place where a standard can live.

Tocqueville thought the leveling of the world was fated. You can read Tocqueville’s canon from start to finish and you won’t find a Tocquevillian manifesto or a policy program. Instead, Tocqueville believed that whether this equal world would become a school for the free man or a comfort blanket for the small one was down to us.

The freedom we have been guarding was never only the freedom to be left alone. It was the freedom to go first and to be great. The machine can generate every word Tocqueville wrote on demand, but it cannot give us the room in which he wrote them. That is what we have to build: tools that hand us back the first move, and rooms that someone, a hundred years from now, will stand inside and know that something happened there.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

At Cosmos, we’re dedicated to supporting work that advances human autonomy, decentralization, and truth-seeking.

We started Cosmos Grants to provide people with the opportunity to work on projects that build towards a positive vision of how AI can be used to strengthen human capacities. These projects need support as they often move too quickly or are too early-stage for traditional research funders, but may not match the commercial priorities of a for-profit lab.

Inspired by Cosmos Founding Fellow Tyler Cowen’s Emergent Ventures, our process makes decisions quickly, with minimal administrative overhead. The program is co-led by Alex Komoroske, Zoe Weinberg, and Darren Zhu.

Initial prototypes in the program have been developed into new research agendas, new companies, adopted benchmarks, and demos for frontier AI labs.

We currently operate two tracks: one focused on human autonomy, and another on truth-seeking, which we run in partnership with the Foundation for Individual Rights and Expression, America’s leading free speech advocacy organization.

Key details

• $1k-10k+; 90-day build project

• Opportunities to collaborate with our community, gain follow on funding, and get featured on our Substack

• Rolling review; decision in ~4 weeks

• Applications close: Sunday July 26, 2026

1. AI for Human Autonomy

Writing lets us externalize memory, while the printing press relieved us of the drudgery of copying. These technologies extended our ability to act, while preserving our capacity to form judgments. AI, however, tempts us to offload deliberation and decision making.

That faculty, autonomy, is the one most at risk, and the one most worth preserving. We fund builders who protect it.

This track funds builders and researchers whose work helps people to avoid deferring too to AI, and practice judgment and self-formation.

2. AI for Truth-Seeking

Truth is found in the open, by many people testing each other’s claims. AI can replace that contest with a single confident answer, and the disagreement that catches error begins to thin out.

We fund projects that support the open contestation of ideas and safeguard the role of human judgment in knowledge production. Alongside funding from Cosmos and FIRE, AI and truth-seeking projects receive compute credits from Prime Intellect.

Apply Now

Get inspiration from past grant projects

Priori surfaces the silent choices behind an AI answer (its tone, what it emphasizes, the values it assumes) and turns them into controls you can adjust, so you can contest defaults instead of trusting them. Steven has since accepted a Cosmos Fellowship and is studying Priori’s effects on autonomy with fellow grantee Cathy Fang.

MoSaIC pairs AI conversation analysis with expert human evaluation to capture the social dynamics of learning that solitary AI tutoring tends to flatten. Caitlin’s work became empirical research published at ISLS CSCL (paper) and IEEE FIE (paper), and part of her PhD at the MIT Media Lab.

Thought Refractor turns any text into several visual forms (breakdowns, comparison frames, flowcharts, concept maps) so you can see an argument’s structure from many angles instead of one. Cheng-Wei founded a startup, Wondering, built on it.

Modelling the Language Process (TLM-1) is a language model that accounts for time, predicting both a document’s contents and the year it was written, and surfacing how words shift meaning over decades. Brandon’s paper was accepted to the ACL 2026 Main Conference.

Prompt injection as role confusion showed how prompt injections are driven by a flaw in how LLMs perceive roles and identified ways to detect when a model is vulnerable. This means models rely on memorized rules instead of reasoning through social situations the way a human would. Jasmine won OpenAI’s red-teaming contest and her paper was accepted to ICML.

Campus is a project-based learning tool that adapts complex projects to your level and prompts you for input, building the capacity for self-directed growth. Kasey has since started a new company, Primitive, focused on coordinated intelligence.

FAQ

Who can apply? Anyone. We’re looking for the best proposals and talent. No credentials needed.

What type of projects do you accept? We fund both new endeavors and existing efforts, though we expect a major deliverable within 90 days. The majority of our applications are from individuals though we also accept team projects.

Where does the funding come from? Cosmos Institute is a 501c(3) non-profit that is funded by philanthropic donations. FIRE is partnering on support and funding for the truth-seeking grant round, with compute provided by Prime Intellect.

How do you select grantees? We look for individuals with talent and potential, and ideas that can be shaped into a practical demonstration or prototype that advances human autonomy or truth-seeking with AI.

What benefits are there beside funding? Cosmos Institute is building a community of technologists committed to elevating human potential through projects that expand autonomy, foster decentralization, and enable truth-seeking in the age of AI. We help grantees connect with like-minded builders and researchers through introductions, in-person meetups, and events.

Apply Now

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Google DeepMind, and Liberty Fund.

How do strangers with no shared plan learn to live and work together? Under the redwood trees at Edge Esmeralda, five hundred people are putting that question to the test. Each has an AI agent of their own that proposes meetups, finds matches, and strikes small bargains for mutual benefit.

Two hundred and fifty years ago, the same question was forced on Adam Smith, David Hume, and Adam Ferguson by the world outside their window. Scotland had been thrown open to global trade during a golden age of commerce. Glasgow merchants grew rich on the Atlantic while Edinburgh courts became crowded with cases for which the law had no categories.

During this great moment of upheaval, commerce was outrunning institutions built for a feudal world. No one had designed this new world and no one was obviously in charge of it, yet it worked.

As that world crept into view, the Scots grappled with the question of how such an order not only makes us prosperous, but keeps us free. Today, as our generation finds its institutions outgrown by a new method of coordinating, we must ask the question again and formulate our own answer.

Subscribe now

The gardener’s method

The Scottish answer began with elemental things such as prices, promises, and habits and asked how they gave rise to vast social orders. They found that institutions grow the way a garden grows, with care and patience and without absolute control over what grows and what withers.

You cannot exact a complete plan over a garden the way you would a house. Likewise, you cannot let it tend itself and hope for the best. Pretending otherwise is how good things go to seed. Gardeners neither command the plants to grow nor leave the ground to fortune.

They prepare the soil and plant with intention, just like the greatest thinkers of the Scottish Enlightenment. These men studied how social orders emerge and sustain themselves, and asked which arrangements would let free people cooperate without coercion. Their method was a form of institutional discovery.

Coordination without command

Smith showed how order could arise from everyday exchange under general rules, through a thousand mutual adjustments made in parallel, with no one commanding the whole.

The coat of a day laborer was made up of the work of the shepherd, the dyer, the spinner, the weaver, the sailor, the miner, and more still. These people never met the man who would wear the coat, and most did not know their work would become a coat at all.

When a frost strikes Brazil, the price of coffee might climb in the cafes of Edinburgh high street. Perhaps a buyer who is none the wiser about the cold snap in South America will order tea instead. The weather in Brazil changed the choice of someone on the other side of the world. And this happened freely, without the need for any minister to set the price of coffee or board to reassign the buyer to tea.

The price system provided commercial society with one of the great enablers of the liberal project: coordination without command. But Smith knew how fragile it was. The same commerce that could make a person freer and happier could also, he feared, narrow her.

For two and a half centuries, this kind of coordination survived efforts to capture or replace it, whether through monopoly, empire, planners, or the administrative state. But where those assaults came from without, today’s markets face a challenge from powerful systems that exist within the architecture of exchange.

The narrow gate

Imagine that when the frost strikes in Brazil, our grower writes a note instead of posting a price. It reads, “The harvest came up short. I’d rather it went to someone who appreciates it, so my regulars shall get first refusal. I trust some buyers more than others, so I’d gladly take less from them.”

That note tells you far more than a number does, but it is also not easy to coordinate ten strangers reading ten such notes. The genius of the price signal is that, while thin, it is both a message (coffee is scarce) and a commitment (it will change hands on these terms). While a paragraph has more explanatory power, a price can clear a market. That is why, for centuries, coordination among strangers had to pass through this narrow gate. Factors like trust or the deeper account of a purchaser’s motivations were crushed into a single scalar.

Three barriers kept coordination thin:

Barrier one was transaction costs. A richer bargain was always possible, but acting on it was expensive. Every added dimension costs more to state and make use of. Such richness was the preserve of the large firm, the repeat player, the old friend, or whoever could afford the lawyers and the time. Because this kind of coordination was expensive, societies reflexively engaged the planner to solve the problem from the top.

Today, an AI agent can understand the entire note and sound out a hundred sellers at the same time. It can assess elements such as timing or the context of the relationship and make a committed offer on behalf of its principal. The richer negotiation that was once the exclusively enjoyed by the large firm is now within reach of the ordinary citizen.

Barrier two was discovery. Picture the whole of social life as a vast chessboard where every person is a piece that moves for reasons of their own. A price tells us where the gains might be made, coaxing pieces to start to slide toward the opportunities.

But how much of the board are we seeing, individually and collectively? It is a practical impossibility for any one of us to canvas the whole of the board given our limited field of view (there are only so many hours in a lifetime). Eight billion lifetimes is a large parallel search, but soon, AI agents will range much farther.

Imagine the collaborator from a different field in a different country who could change the course of your work whom you might otherwise never meet. Or perhaps you have been stuck on a problem for years without knowing where the perfect potential colleague lives. In the past, these connections may never have been made, but today, thanks to powerful agents, these kinds of coordination problems may prove soluble.

Barrier three was purpose. Prices let you pursue your purposes without ever forcing you to articulate them in full. You might buy a book to settle an argument or to retrain for a new career. But the merchant doesn’t need to know that. The only person who does know that is you.

Your agent may help you find common cause with others whose purpose aligns with your own, or to help you better express your own. We might imagine a book buyer who is in the middle of retraining for a new profession. The price tells him about what a book costs, while the agent knows what the book is for. Then it may find him a course or a mentor or even a first client, not to mention the others who are taking on the same kind of journey.

Coordination is not the same as liberty

This is a positive future for coordination, but it is not one that we are guaranteed to see. The same AI agents that strike the bargains or find the possibilities may just as easily bring about more worrisome effects. They may be used to organize human life on a scale beyond the wildest dream of the Soviet planner, or shape our choices so thoroughly that we become less capable of authoring our own purposes.

Just like the Scottish enlightenment thinkers, we need to know how we might become more prosperous and how we might remain free agents.

Coordination is not inherently good. After all, the mercantile system Smith spent his life attacking was an elaborate scheme of coordination run by ministers and chartered companies. Whatever else was wrong with it, few could complain that its chief defect was disorder.

Consider the chess board again. This was of course Smith’s metaphor, one that he introduced as a warning. He wrote of the “man of system,” who looks at a society and imagines he can arrange its members as a hand arranges the pieces on a board. In doing so, this person forgets that each piece “has a principle of motion of its own.”

Smith’s man of system moved the pieces against their will, but the agent economy - our system of systems - need not do so. It might have the effect of whittling the pieces down instead, until moving is unnecessary.

Coordination can be so wholly attentive or endlessly accommodating that we cease to perform the labor of free people who live within it. The price system asked little of us, but it still left us to decide what we wanted and to act according to those wants. It coordinated our means and it presupposed our ends rather than supplying them.

AI can frame which goals are worth pursuing for each of us, then hand us answers in such a way that we are never called on to form them ourselves.

Think of a parent deciding what to teach her child. She asks the agent and the agent provides a considered response in short order. If the parent takes it once, perhaps nothing of consequence is lost. But if she takes it every time, each week, each day, or each hour, she may find that she no longer has a view of her own in a meaningful sense.

And why should that trouble us, if the agent’s answer is better than the one we would have reached? Because a good life is not only a life that goes well. It is, and must remain, a life that each of us leads.

Smith saw this danger too, at the root of the system he praised. He warned that the man who was kept to a few simple operations, “has no occasion to exert his understanding” and grows “as stupid and ignorant as it is possible for a human creature to become.” This was his fear for the laboring hand, narrowed by the division of work.

We now risk it for the mind, narrowed by the division of judgment.

What takes the place of blindness

The Scots did not merely think the modern world into being. Instead, they built it through the institutions of trade and the habits of conversation. Gardeners as they were, they neither dictated the order nor waited for it to spring up fully formed on its own without assistance.

They helped find the rules that made price-based coordination blossom under liberty. These were general rules that were blind to persons and indifferent to ends.

The law could be trusted because it knew nothing of you, but the AI agent is powerful because it knows you well. What are the general rules for coordination using systems that know us? Answering this question is why five hundred agents are at work in California while their principals sleep. This is because we cannot think our new world into being, no matter how clever our ideas or how noble our intentions.

We must tend or we lose the garden, and with it the capacities that make life meaningful. We must revitalize the conditions under which free people can govern themselves, and we must do so again and again and again if we are to keep the liberal dream alive.

A tradition lives by being used.

Adapted from a keynote delivered by Brendan at the New Enlightenment Conference in Edinburgh, marking the 250th anniversary of Adam Smith’s The Wealth of Nations at Panmure House. Thanks to Ivan Vendrov and Séb Krier for their conversations and insights.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

The institutions of a free society allow people to govern themselves. From media that permits expression to markets that enable the open exchange of goods and services, institutions help us flourish as individuals and in tandem with others.

AI is changing their foundational assumptions. By lowering the cost of expression, persuasion, inquiry, delegation, and coordination, these systems are modifying the character of the basic activities that liberal institutions seek to accommodate. It floods them with activity at volumes they were not designed to absorb and enables behaviors they were not designed to manage.

At stake is the capacity to deliberate about the lives we want to lead. AI may help us live more purposefully, or it may narrow our field of view and shepherd us toward ends chosen by others. To help build AI that protects and expands liberty, we are introducing the Cosmos Research Group to study autonomy, truth-seeking, and decentralization.

These are the philosophical foundations that frontier development needs if AI is to help us lead happier, healthier, more prosperous, and more self-directed lives. We are joined in this effort by a new cohort of Senior Research Fellows from frontier labs and the academy. Their work will help us to open new lines of inquiry, shape our research agenda, and connect philosophical questions to technical choices that determine what AI is and how it will be used.

This cohort joins Senior Research Fellows Philipp Koralus and Vincent Wang-Maścianica, and Founding Fellows including Jack Clark, Ivan Vendrov, and Tyler Cowen. We’ll be sharing more information about our work in the coming months, including opportunities for researchers to partner with us.

Subscribe now

Séb Krier, Frontier Policy Development Lead, Google DeepMind

Séb Krier is the Frontier Policy Development Lead at Google DeepMind, where he focuses on the governance of frontier AI systems. He previously served as a Senior Technology Policy Researcher at Stanford University’s Cyber Policy Center and Head of Regulation at the UK Government’s Office for Artificial Intelligence. A former international lawyer at firms including Freshfields Bruckhaus Deringer and Bryan Cave Leighton Paisner, Séb blends technology, law, and institutional economics to explore the future of AI. His writing brings a classical liberal lens to frontier technology, focusing on how technology can be designed to resist zero-sum traps, protect individual liberty, and enable human flourishing.

Seth Lazar, Founding Director of Machine Intelligence & Normative Theory (MINT) Lab

Seth Lazar’s research focuses on the moral and political philosophy of AI and computing, and on the reinvigoration and redesign of liberal democratic institutions for the AI transition. He has published widely in top philosophy journals and computer science conferences, and is an editor of Oxford Studies in Philosophy of AI and Computing, and the Alignment Journal. He is a Professor in the School of Government and Policy at Johns Hopkins University, a Professor of Philosophy at ANU, a non-resident fellow at the Carnegie Endowment for International Peace, and a Distinguished Research Fellow of the University of Oxford’s Institute for Ethics in AI.

Houda Nait el Barj, Experience Research Lead, OpenAI

Houda Nait El Barj is a research lead at OpenAI working at the intersection of multimodal AI, human-computer interaction, and continuous learning. Her work focuses on building intelligent systems that can adapt alongside people over time, supporting their goals in ways that are safe, personalized, and aligned with human agency. Originally from Morocco, Houda is passionate about designing technology that expands human potential and helps bring more beauty, meaning, and long-term flourishing into the world.

Matthew Botvinick, Member of Technical Staff, Anthropic

Matthew Botvinick is a member of the technical staff at Anthropic, leading work on AI and the rule of law; a senior fellow at Yale Law School; and an honorary professor at the Gatsby Computational Neuroscience Unit at University College London. In previous work, Botvinick served as senior director of AI research and senior technical and policy advisor at Google DeepMind, and led the Neural Computation Lab at Princeton University and the University of Pennsylvania. Botvinick holds an M.D.from Cornell University, a Ph.D in deep learning and cognitive science from Carnegie Mellon University, a J.D. from Yale Law School, and an M.A. in Art History from Columbia University.

Julia Haas, Staff Research Scientist, Google DeepMind

Julia Haas is a Staff Research Scientist in the Responsibility Unit at Google DeepMind. Her research is focused on philosophy and cognitive science, and she works on the nature of valuation and its role in theories of the mind, especially in normative cognition. Her work also includes investigating the possibility of meaningfully moral artificial intelligence, and has introduced the concept of the “Evaluative Mind” proposing that beneath our thinking and reasoning lies a continual and automatic appraisal of the world that shapes our faculties.

Harvey Lederman, Professor of Philosophy, UT Austin

Harvey Lederman is a philosopher with broad interests in contemporary philosophy and in the history of philosophy. After earning his DPhil (PhD) at Oxford, he taught at Pittsburgh, and then Princeton, where he was promoted to full professor in 2022. Since 2023, Harvey has been professor of philosophy and the Jacob and Frances Sanger Mossiker Chair of the Humanities at UT Austin. In 2026-7 he will be a Visiting Professor at NYU. His current work is divided between the foundations of game and decision theory, the philosophy of AI, and the history of Chinese neo-Confucianism.

Ben Bariach, Sr Director, Frontier Safety and Alignment, Microsoft Superintelligence

Ben Bariach is a researcher in the philosophy and governance of AI at the University of Oxford. In his professional capacity, he leads work on frontier AI safety and societal impact, currently at Microsoft Superintelligence and previously at Google DeepMind, where he has helped build world-leading AI models that are human-centered and safe. His research interests center on superintelligence preparedness, including the implicit philosophies underpinning AI development, agentic AI embodiment and autonomy boundaries, and the perception of machine minds. Ben has published widely across both technical and humanities venues, with work spanning frontier model development, evaluation, safety, and ethics.

Andrew Hall, Founder, Free Systems Lab

Professor Andrew Hall is the Davies Family Professor of Political Economy at the Stanford Graduate School of Business and a Senior Fellow at the Hoover Institution. He leads the Free Systems Lab, which measures whether AI systems can be trusted with power. The Lab is housed across the Hoover Institution and Stanford GSB - studying how AI systems shape political information and behavior, how they can be governed, and how AI itself can be used to design and test better systems of governance. Free Systems has produced work on model training, platform policy, and constitutional design - including the Dictatorship Eval, a benchmark measuring large language model responses to authoritarian requests, cited by Anthropic in its frontier model evaluations. Andrew also advises the a16z crypto research team on decentralized governance and is an advisor to Forum AI. Previously he spent eight years advising Meta on governance and strategic issues.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

Civic life is the practice of taking responsibility with others for the shared conditions of common existence. Today, many of us hold a view of politics that is a long way from this ideal. We equate civic life with voting for representatives who make decisions on our behalf or engaging in limited forms of consultation when the opportunity arises. Modern politics is a system for making decisions rather than citizens.

Our predicament is partly a function of scale. The polities of the ancient world, though deeply limited in many respects, were small enough that those who held the right to engage in collective decision-making knew their peers and could formulate laws that would directly affect them. In gargantuan modern states, this kind of engagement is a practical impossibility.

Representative democracy exists, in part, to free us from the demands that direct self-rule would otherwise make of us. Classical proponents like Edmund Burke and James Madison argued that representation also improves the quality of governance by passing public preferences through a deliberative body capable of refining them and attending to the long term good.

On Madison’s account in Federalist 10, the legislature exists “to refine and enlarge the public views” by passing them through “a chosen body of citizens, whose wisdom may best discern the true interest of their country.” In Federalist 51, he argues that “ambition must be made to counteract ambition,” that political structures should compensate for the imperfection of those who govern and those who are governed. In other words, representative democracy can produce stable public decisions without requiring citizens to be highly formed civic actors.

Of two modes of civic life, formation and aggregation, the latter is taken to be the primary role of an advanced democracy. As political systems strain under the complexity of the modern world, AI threatens to remake institutions like education, entertainment, economic systems, the media, and various professions. Thinking machines do cheaply the things institutions were built for (such as drafting legal contracts or writing and researching) and make possible things they were not designed for (like one-on-one tutoring for every student or autonomous agents that act on a person’s behalf).

These twin pressures demand new kinds of institutions, especially political institutions to help societies navigate the diffusion of AI into public and private life. Various kinds of “AI for democracy” projects fit this account by appearing to remake democratic decision-making through, for example, summarization techniques for lawmakers or fostering agreement on contentious issues.

This assessment gets things backwards. AI represents, at least in the context of democracy, a small-c conservative turn that may entrench an aggregative view of civic life at exactly the point at which new possibilities allow us to better ask what civic life is and why it matters. The technology could prove to be less a centrifugal force pulling political institutions apart and more a centripetal force gluing a particular kind of institutional arrangement into place.

A view that takes civic life to be about primarily producing better and more legitimate decisions through preference aggregation has been dominant for two centuries. The AI and democracy project takes that settlement and makes it more efficient, capable, pervasive, and considerably harder to dislodge. It is the terminus of the Madisonian project, a sophisticated apparatus for generating politically resonant outputs without bearing the cost of citizen formation.

Subscribe now

City on the hill

The first recognizable instance of politics-as-formation was the Athenian polity, a self-governing political community that emerged on the peninsula of Attica in southern Greece around the late sixth century BCE. Across two and a half thousand square kilometers, roughly the size of modern Luxembourg, the Athenian polity was made up of the city proper, surrounding villages and farmland, the silver mines country at Laurion in the south, and the port of Piraeus a few miles from the city.

From urban core to rural hinterland, all citizens who met the criteria were part of the same political whole. A farmer thirty miles from the city and the craftsman in the agora belonged to the same community and could expect, if they chose to make the journey, to vote in the same assembly. The polity was small enough that everyone within it was capable of representing everyone else.

Roughly 250,000 to 300,000 residents fell under the authority of Athens in the fourth century BCE. Of these, only adult male Athenians born of Athenian parents, say, about 30,000 men, were full citizens with political rights. Factoring out wives, daughters, slaves, foreigners, the political community was around ten to twelve per cent of the population.

The Assembly or Ekklesia was a gathering of all citizens together as a single body. It met about forty times a year on a hill called the Pnyx, a quarter of a mile south-west of the Athenian agora. The body dealt with the things that today we would call “high politics,” such as whether to go to war, whether to accept peace terms, what taxes to levy, what laws to pass, or how much grain to import. It also included grislier episodes like voting to slaughter the entire population of Mytilene and putting its own generals to death after the battle of Arginusae.

Another institution was the Council of Five Hundred, the Boule, which functioned as a kind of executive committee for the Assembly. It set the agenda and supervised the administration of the state. Its members were chosen by raffle, with fifty citizens each year drawn at random from each of the polity’s ten tribes. Most male Athenians spent a year of their adult life as part of the executive government of their state.

Together, these bodies meant that the participating citizen might attend the Assembly several times a year, serve a full year on the executive Council, sit as juror in major trials, and hold one or more rotational offices. For those who could take part, institutions allowed citizens to live a particular kind of political life. A handful of professional administrators could have made decisions far more quickly, yet the Athenians chose to bind themselves to more cumbersome arrangements because the essence of the system was to produce citizens.

Aristotle described this idea as hexis, a durable state of character cultivated through repeated action and habituation. We become just by doing just acts, temperate by doing temperate acts, and brave by doing brave acts. The citizen is formed by undergoing the discipline of ruling and being ruled in turn. They grow by attending the Assembly, sitting on the Council, hearing cases in the courts, and being held to account for their decisions.

Aggregation vs. formation

Civic life is taken to serve primarily one of two ends:

• Aggregating preferences. The process by which the preferences of citizens are gathered and weighed against one another to be converted into collective decisions. The citizen is a source of opinions that the political system needs in order to govern well.

• Forming citizens. The process by which citizens are made and remade. The output of democratic participation is the participant herself insofar as she leaves with a capacity to be a better civic actor.

Where the ancient model was mostly concerned with forming citizens, modern democracies are for the most part sprawling systems for converting preferences into governance. More participants produce more inputs to be aggregated, so in principle there is no upper bound on the size of the political community to which the procedure can apply. Modern institutions can hoover up preferences from millions through regular elections, but only at an extremely low level of resolution.

The basic claim of the aggregative family is that democracy is valuable because it generates good collective decisions, that the citizen is essentially an information source that produces preferences or opinions to be subordinated to the greater whole. Democratic procedures (like voting, deliberation, polling, or citizens’ assemblies) are all tools for processing those inputs into legitimate outputs.

Even within the aggregative tradition, some like John Stuart Mill argue that representative democracy can make citizens as well as decisions. For Mill, even occasional participation in public functions has a salutary effect on the citizen who takes part. “He is called upon,” Mill writes of the private citizen, “to weigh interests not his own; to be guided, in case of conflicting claims, by another rule than his private partialities; to apply, at every turn, principles and maxims which have for their reason of existence the general good.”

Analyses of this type acknowledge that formation is a laudable goal and hope that representative institutions could supply some version of it through juries, local office, and the modest acts of public engagement available to a private citizen. In practice, however, the settlement does not bear out these hopes. Over the century and a half since Mill’s reflections, the aggregative purposes of the system have steadily eclipsed its formative functions. Few of us would today say that the systems we live inside sharpen our faculty of self-governance or fellow-feeling.

Parallel to the aggregation tradition is the formation-first approach, best characterized by Alexis de Tocqueville in his travels across America in the 1830s. Tocqueville believed that the habits of self-rule in the small things prepared people to rule themselves in the great ones. The more citizens tested their judgments in the company of others, the more resilient their characters became. Democracy stood or fell on these everyday practices, for it was only through the rehearsal of judgment in everyday life that citizens could learn the art of governing themselves.

Tocqueville knew what happened when those habits were no longer present. In the France he grew up in, centuries of monarchical centralization had drawn the business of ordinary life away from local communities and into the hands of the state, so that by the time the revolution spilled into the streets, the citizens who were asked to govern themselves had long since lost the practice of doing so.

The provincial assemblies that had once given ordinary people a say in their own affairs had been largely abolished; the guilds and local associations that had trained people in the small arts of collective decision-making had been suppressed; and the aristocratic class that had once served, however imperfectly, as a counterweight to central power had been reduced to a decorative caste with titles but no responsibilities.

Tocqueville imagined how this kind of decay might unfold in a country that had skipped its aristocratic adolescence. One danger was that citizens might willingly hand over the work of self-government to a mild and far-seeing administrative state. He described a “tutelary power” that “does not tyrannize, but it compresses, enervates, extinguishes, and stupefies a people, till each nation is reduced to nothing better than a flock of timid and industrious animals, of which the government is the shepherd.”

Democratic deficit

Representative democracy faces two distinct challenges within the aggregative tradition. The first is a bandwidth problem, which reminds us that modern institutions can only collect citizens’ views in a heavily compressed form - like a vote or poll - that fails to capture most of what each citizen actually thinks. The second is the deliberative problem. Here the kind of refining and filtering Burke and Madison thought representative bodies were for has been overtaken by the volume of political business and the professionalization of representatives.

The AI for democracy field is directed at the first problem. It concerns capturing citizens’ views at higher fidelity and aggregating those inputs at scale. The deliberative deficit barely features beyond the comparatively narrow work of synthesizing information so that officials can deliberate with better inputs. The formative concern, which is about what civic life does to the citizens who take part in them, is not a significant part of the project.

A survey by Seth Lazar and Lorenzo Manuali connects the field’s work to four modes of usage:

• Summarization: Synthesizing information in a way that officials or citizens can act more effectively. One developed example is the Recursive Public project, which gathered public input on AI policy and used LLMs to summarize the resulting discussions and group similar arguments into clusters in semantic space.

• Aggregation: Generating new statements that the model predicts will attract broader support than any of the originals. The Generative Social Choice project, for example, took participants’ views on contested issues and prompted a model to propose a slate of statements that a majority of participants reported feeling “excellently” or “exceptionally” represented by.

• Representation: Predicting user preferences by standing in for her in a simulated polity. See, for example, the Augmented Democracy project, which seeks “to create personalized AI representatives to augment their ability to participate directly in many democratic decisions.”

• Facilitation: Engaging participants in a structured conversation whereby the model plays the role of a discussion leader to keep conversations on topic, ask follow-up questions, draw reluctant participants into the exchange, and produce summaries at intervals. See Rappler’s aiDialogue project.

All of the major labs have worked on projects that fall within these categories, from Anthropic’s Collective Constitutional AI process designed to crowdsource Claude’s constitution to Google DeepMind’s Habermas Machine that helps groups of individuals find consensus on political topics.

These tools and others like them make it possible to aggregate preferences more efficiently. They also provide the resulting decisions with a kind of legitimacy that any modern polity needs in order to function. The decisions that constitute political life must be made somehow, and a procedure that draws as fully as possible on its citizens is by most reasonable measures an improvement on the status quo.

Like the vast majority of AI and democracy projects, each of the examples described above are concerned with aggregation rather than formation. An accurate clustering of a thousand Americans’ opinions about a frontier model’s content policy does not give anyone the experience of having governed themselves. A more efficient deliberative platform that drafts consensus statements between strangers does not produce the habits of self-rule that the Athenian Assembly or the New England town meeting cultivated through consequential practice over the course of an ordinary life.

The outputs look superficially similar in that citizens have played a role, but they did so without the inward labor through which civic life makes citizens. A citizen who has practiced forming and revising her own views in conversation with her peers is a better judge of political questions than someone whose preferences have been collected and aggregated by a model. Rather than asking us to trade decision quality for civic virtue, the formative tradition suggests both are the joint product of the same practice.

An obvious response is to ask whether AI might be designed to support a formative account of civic life. The temptation should be resisted. Formation is constituted by the experience of being responsible for consequences alongside fellow citizens whose lives are bound up with yours. There is no AI system, however carefully designed, that can deliver that.

While not straightforwardly citizen-making themselves, there are uses of AI that could lower the cost of entry to formative practices. A tool that helps a citizen convene like-minded peers does not help us grow, but it does make the work of civic life easier to start. A system that translates a council planning document into something the resident affected by it can understand does not produce a citizen capable of self-rule, but it may lower the transaction cost of participation through which such a citizen might be made.

Formative years

Tocqueville thought that juries were slower than judges and more likely to err on questions of law. If the only purpose of the courtroom was to produce the right verdict, the jury would have been abolished long ago. “The jury teaches every man not to recoil before the responsibility of his own actions,” he wrote, “and impresses him with that manly confidence without which political virtue cannot exist.”

Today, we think about governance as the process by which decisions are taken on our behalf and implemented by specialists. This is in part a function of the highly complex nature of the modern world, which depends on large-scale coordination and deep technical specialization.

Even so, the decisions that constitute political life are taken at scales that far exceed the ceiling of the Tocquevillian model. The areas that remain to citizens at the local level are administrative bodies operating downstream of decisions taken elsewhere rather than deliberative bodies that make decisions and live with the results.

AI systems may well make political institutions more legitimate, more efficient, and more responsive to our collective wants. On its own terms, this is a good outcome if it helps democracies make better decisions according to the wishes of their people. But collecting and aggregating preferences is not how we become more capable agents who can better take part in civic life.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

Every builder’s first duty is philosophical: to decide what they should build for. This series asks 9 questions to founders who are building towards their vision of the human good.

This week’s guest is Matt Clifford. Matt is the co-founder of Entrepreneurs First and the chair of ARIA – the UK’s Advanced Research & Invention Agency. He is the co-author of How to be a founder (with Alice Bentinck), serves on the board of Code First Girls, and writes open-source immersive murder mystery games.

Subscribe now

1. What are the core questions or beliefs driving your work?

There are three:

1. That the emergence of powerful AI is the most important transition we will have to navigate in my lifetime.

2. That Britain has the potential to be a great country and force for good in the world.

3. That there may be a deep connection between the first two ideas - and that Britain achieving its potential may be one of the most important ways to shape the trajectory of AI for good.

2. What future are you building for?

I spent most of the last 15 years building Entrepreneurs First (EF), which aims to increase the supply of great founders - and therefore great companies - in the world.

Over the last four years, I’ve combined this with trying to build UK state capacity in science and technology generally, and AI specifically - first as Chair of ARIA (the Advanced Research & Invention Agency - loosely, the UK’s take on DARPA), then as AI adviser to PMs Rishi Sunak and Keir Starmer.

In the latter roles I helped create the UK’s AI Security Institute (AISI), convened the first AI Safety Summit and wrote the AI Opportunities Action Plan (the UK’s national AI strategy).

When I look across these roles, I think the thread that runs through is something like: I want to live in a world where as many humans as possible live in conditions of abundance with real freedom to shape the course of their lives.

3. What commonly held belief in the tech community do you believe is wrong?

I think many people in tech (at least implicitly) believe something like “we are our incentives” and therefore that various bad equilibria are inevitable. I think the opposite is true, but less comfortable: virtue is a real thing and that we can bend even very powerful forces by making the right choices.

4. What are your main philosophical influences?

I’m a big believer that the genius of the British constitution is the capacity for error correction, so I’m a big fan of Popper and Deutsch in that vein. Relatedly, I think a lot of my appreciation of the value of permissionlessness and bottom-up action comes from Hayek - but tempered by people like Elizabeth Anderson and their warnings about the risk to freedom from private as well as public power.

I’d say, though, that historians have been at least as important for me intellectually as philosophers: my beliefs about British exceptionalism are strongly influenced by Alan Macfarlane and Joel Mokyr, and Quentin Skinner’s work has left a lasting impression on how I think about ideas and public argument. More recently, Henry Farrell’s work (with Abraham Newman) on weaponised interdependence has shaped my thinking about technological sovereignty (and I also like Farrell’s work with Cosmo Shalizi on “cognitive democracy” - another helpful dialogue with Hayek and markets).

5. What does human flourishing mean to you?

I find it hard to go beyond my answer to question 2, above: human flourishing is about people having the material and social conditions that mean the choices they make about their lives can be both meaningful and effective.

6. What’s one book you’ve read recently that you’d recommend?

Greg Egan’s Permutation City is over 30 years old, but it’s remarkably prescient (and I only just read it). Excellent read if you still need to convince yourself that we’re always going to be short of compute.

7. What’s your most irrational belief?

Some would say part (3) of my answer to the first question! Other than that: at some level, I do believe that I am a very lucky person.

8. What’s the most interesting tab you have open right now?

Probably this visualisation of the AI supply chain and who owns it.

9. Who is one writer or thinker today who you think is underrated?

Is Séb Krier still underrated? Maybe so - still under 25,000 followers on X - even though he is now much better known than a year ago. Jack Wiseman’s Inference and Rohit Krishnan’s Strange Loop Canon should each be at least ten times bigger too.

You can read more about Matt’s work on his website and get in touch with him on X.

This is the fourth instalment in this interview series. You can also see our interviews with AI Underwriting Company co-founder Rune Kvist, ex/ante founder Zoe Weinberg, and Zena Hitz, the founder of the Catherine Project.

To nominate someone for “What Will You Build For?” leave a comment below, or send us a DM.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

Human language runs at roughly forty bits per second, whether you’re reading, speaking, or listening. It is a hard limit on how much information one person can convey to another. It is also arguably a ceiling on how well communities can coordinate, since they are often reliant on a handful of people able to turn what others are saying into collective action.

But what would happen if we didn’t have this ceiling?

Edge Esmeralda is a pop-up village in California, running from May 30 to June 27 in Healdsburg, California. The project is aiming to bring over 500 scientists, artists, builders, and thinkers together for up to four weeks. Attendees aren’t expected to commit for the entire month – there will likely be 150 people on site at any one time. There’ll be a range of programming, including fellowships and residencies, summits, and opportunities for collaborative working.

The second week of the event (June 8–14) will feature a dedicated set of sessions on intelligence and autonomy, covering topics like AI agents, autonomous infrastructure, neurotech, privacy-preserving systems, and governance design.

Cosmos is excited to be partnering with Edge Esmeralda to deliver the Agent Village Experiment for the duration of the month.

Led by Ivan Vendrov (independent researcher and Cosmos Founding Fellow), Timour Kosters (Co-Founder, Edge City), and Harry Law (Principal Researcher, Cosmos Institute), the experiment will grant every multi-day attendee access to an AI agent running on their behalf throughout the village.

This personal agent will help humans to navigate the schedule, the wiki, and community governance. More excitingly, it will also exist in a shared digital plaza and will help attendees meet people who share common interests, propose dinners, negotiate community decisions, and participate in community governance. In essence, it’ll play a valuable role in making connections that might otherwise be missed due to humans’ lack of bandwidth. The technical collaborators are Index Network, Geo Browser, Instaclaw, and Tule Romeo.

The experiment has been inspired by a number of different ideas, including Ivan’s work on supercooperation and Séb Krier’s essay on Coasean bargaining. Both are excited about the potential of decentralized AI agents to unlock gains in coordination, such as better mutual understanding, cheap bargaining, and conflict resolution.

By using real-world scenarios, the experiment aims to shed light on how trust develops in agent-to-agent relationships over time, the kinds of actions people do and don’t find useful, or whether AI leads to better community deliberation and decision-making. The goal is for this to be the largest live experiment in human-AI collective experiment to date. Historically, most of the experiments around agent cooperation have taken place in synthetic environments, so a large-scale experiment in a real-world environment presents an important early opportunity for learning.

You can read more about the experiment, the methodology, and Edge Esmeralda’s predictions here.

We have a small number of complimentary tickets for the event, which we are offering to individuals in our network. So if you’re interested in Edge Esmeralda’s programming and the chance to take part in the experiment, please fill out this short form as soon as possible. The deadline for entries is May 31 and winners will hear back from as feasible.

Apply for a ticket

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

In his lecture, Jack covers advances in capabilities, how his own relationship with AI has evolved, and some scenarios for future progress.

We’re delighted to bring you the full video of the lecture and fireside chat. If you’d prefer to read the transcript, Jack has posted it in full over at Import AI, his weekly newsletter on AI research.

We have more talks and events like this over the rest of 2026. The best way to ensure you don’t miss out is to subscribe.

Subscribe now

Highlights

12:24 - Jack’s changing relationship with AI

24:40 – how AI changed Anthropic

44:40 – start of fireside chat

50:08 – live by the Claude, die by the Claude

1:01:00 – what Jack would say to Mythos

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

On Wednesday, Jack Clark, co-founder of Anthropic, delivered the second Cosmos Lecture in partnership with the Human-Centered AI Lab at the University of Oxford. The lecture was introduced by HAI Lab Director Professor Philipp Koralus and was followed by a fireside chat between Jack and Brendan McCord, the founder of Cosmos Institute.

We’re bringing you a lightly edited transcript of the conversation, which covered what AI cannot do for us, whether Claude makes us better thinkers, and what Jack wants future AI systems to know about humanity. The full lecture will follow in the coming weeks.

Philosophy to code

Brendan McCord: Jack, you ended with the call to build the new world. It makes me think – if you and I were having this conversation 250 years ago, the proudest project we could have possibly been engaged in would have been building a new world of sorts. I would call it the philosophy-to-law pipeline. We would have been looking to Oxford intellectuals like John Locke, or Montesquieu, Livy, Adam Smith, and translating that into a constitution that we hoped would frame freedom for the 250 years to come.

The proudest project we can engage in now is, as you say, this new world-building project – it’s philosophy-to-code. What would you say about the extent to which the frontier labs take that seriously? What can we do to really take that seriously in places like Oxford and academia? And what should we do in nonprofit land to take that philosophy-to-code project seriously?

Jack Clark: I think it requires you to basically accept that progress will continue and try to model out scenarios based on it. I think dealing with COVID highlighted that, though there’d been some modeling of what would happen, if you had very fast take-off propagating viruses the world would break very quickly. We felt underprepared, and that we could have done more scenario work and forecasting of what these strange things would do to us ahead of time.

Within the AI labs, I think there is now work at all of them on trying to imagine what you might think of as “post-AGI worlds,” or worlds that happen after recursive self-improvement. But my general sense is every time you sit in a room at the lab, people say, “Are we the only people working on this?” And you say, “I’m terribly sorry, yes.” And then some people put their head in their hands and wish that more people were working on it.

The good news is that this is exactly the kind of work that universities and other organizations are built for because you don’t need to be running a large-scale supercomputer or training a very capital-intensive model. You need rather to model out, in a theoretical sense, the properties of an AI system that can massively multiply productivity, or an AI system whose inference costs fall at X rate, and capabilities rise at Y rate. What does that do to the economy? What are the things that it unlocks? What are the aspects of this supply chain where you might invest, or change the supply chain to actually change the character of the systems? There’s tremendous work to be done.

I’ve been in the UK, in part speaking with the UK government, and I made this point: if the UK government just had 10 to 20 people whose sole job was modelling out what happens if the technologists are right about this technology, the UK would be better prepared than any other country in the world – because so little work has happened. So it’s a great time for universities to be doing these projects.

Brendan: So should Madison, Hamilton, and Jay have spent a lot more time on forecasting than they did on debating the nature of man and the political order?

Jack: It’s a hard question. I feel like you have a take on this!

Brendan: I think we can’t miss the part of contemplating about the ends. And I think what brought them together was a kind of unique epistemic humility that they shared with the Scottish Enlightenment thinkers.

Jack: My assumption with AI is that there is a huge value in norms and precedent – which is, how do we want these systems to show up in the world? I’ve covered that a bit less in my talk, but it relates to how we shape the so-called “character,” or what some might say personality, of these systems. How do we want them to behave towards us? This is a normative question – a philosophical question – and we should absolutely work on that.

But I have been struck by how surprised even the AI labs have been by their own progress, which is a very counterintuitive thing. We work with these AI labs and they keep saying, “Well, you know, as we said last year at Anthropic, we did a load of work on the increasing rate of cyber-hacking capabilities of AI systems.” I run a team that does this. We wrote blogs saying, “Oh, that’s interesting. Surely it has some implications if the system suddenly becomes capable of nation-state-grade stuff. We seem to be on a trajectory here.” So we did some prep work, and then nonetheless we made Mythos and were like, “It’s here faster than we thought. We’ve done insufficient preparation.” This is true of every single time AI progress has happened. People have been continually surprised by how significant the jumps have been and how quickly they’ve come. So we all need to do more work on this.

To defer or not to defer? The case of the Claude Boys

Brendan: I want to ask you about something strange that happened on the internet. It happened about a year ago. It was a group of 13-year-old boys who decided “to live by the Claude and die by the Claude.” They did this first as parody, and then they earnestly adopted the identity of Claude Boys. From morning to night, they just did what Claude told them. Even though we can laugh at that, and by the way, I understand, I was a 13-year-old boy; it’s hard to navigate some of the social situations and Claude would do it better – should we understand that to be a kind of funny thing, but an adjustment to a new set of conditions, to a new world? Or should we understand it to be a kind of problematic pathology, a canary of sorts?

Jack: I think it’s clearly problematic in that everyone needs a part of their life where you’re making your own decisions, including mistakes. You need to protect that and have some amount of agency. I think a lot of what parents go through is they watch their kids about making mistakes and they say,”Please don’t make that mistake.” And the kid says, “I’m going to exercise my will to choose and I’m going to make a mistake.” The parent says, “Good luck with that, come back to me in a year and we’ll discuss it.” We need to let people have independence.

At the same time, I think the confusing thing is that AI systems may sometimes give genuinely good advice. And my experience is that I’ve only really calibrated how good the advice Claude has given me is in relation to how much I’d thought about that outside the context of working with the AI system. I have an okay-but-tense relationship with my dad – as I’m sure many people do – and I’ve obviously written about that at length, trying to grapple with it. I fed that writing to Claude and said, “What should I do?” And Claude was like, “You should see your dad. Don’t talk to me about your relationship with your dad. Just try and see your father.” Or when I told Claude I was a bit depressed and wondering whether I should go to an art show and see friends or stay and work and talk to Claude, Claude was like, “Go to your art show and see your friends.”

Those things both worked because I’d grappled with my personal experience outside of the context of talking to the AI system. But what I worry about – and I think it comes back to questions of design – is people who don’t have a kind of internal introspective practice outside of their relationship with their AI, and are rather discovering themselves in relation to the AI system. They don’t have a diary, they don’t have writing they’re doing outside it. They’re just talking to the AI system. And I think that makes you uniquely vulnerable to it giving you bad advice, because you have no place where you develop your opinion outside of it.

So I think from a system design point of view, we’re going to need to do what Nintendo or Netflix do, where they basically say, “You’ve spent too much time on this, it’s time to stop” or “it’s time to go outside.” There will be versions of this, where without being paternalist, we’ll say, “Hey, you’re talking to us a whole bunch about these complex things to do with your relationship. Can I encourage you to go and talk to other people, the humans named in that relationship, rather than me?” Or somehow encourage introspection outside of the context of the AI system.

Brendan: I want to argue for total deference, and I want to do it because we’re at a university and we can say whatever we want. Humans are famously bad at choosing. We evolved for tribal life. The way we think about future discounting, the way we think about scale – it’s not great. We also aren’t that good at coordinating with each other. We aren’t that good at satisfying our own preferences, let alone imagining possible futures. And so if we have a system that can look across all the research papers, with much more knowledge available to it, that can harness complexity, think about uncertainty – is it not morally obligatory that we defer to this system? Is it kind of negligent, what you’re advocating here, that we think for ourselves?

Jack: I think then the question is: what are you doing with the gift of life if you’re turning yourself into an automaton? I think in part you’re maybe having effects which might seem globally morally beneficial, but I think locally you’re not treating other people with a form of basic respect. I worry about this – where you enter this paradoxical situation where the systems really are much smarter than us, and it does invite this question. But then I say, well, what is the purpose of being human? And I think part of it is experimentation and making mistakes. We learn more from mistakes than from our achievements. If you were in a world where you never make any mistakes and you only achieve through the AI system, are you still a person? Are you a philosophical zombie driven by Claude?

Does Claude make us better thinkers?

Brendan: You have a lot of data that we don’t have, and you’re starting to do more instrumentation and experimentation. Do you have any sense of what Claude does to our prospective capacity to deliberate? Meaning, does it make us better thinkers when Claude is not in the room?

Jack: Oh, interesting. We developed a system recently called Claude Interviewer. It’s a version of Claude that can interview people about arbitrary subjects. So we had Claude interview 80,000 people around the world – people who subscribe to Anthropic – about their hopes and fears about AI, and their worries about the world they could end up in. Rather than having a blank conversation, it’s actually trying to do the work that social scientists do and collect data. We haven’t measured the effect of that, but what it means is that 80,000 people had a conversation where they were actually forced to grapple with their anxieties and hopes about the future of AI. I think that must have had some kind of effect, in the same way that you or I were having a discussion. I think it’s likely to have beneficial effects if you can use it judiciously and use it to cause people to think more about things that are important to them, to develop their own opinion rather than to defer.

Drastic interventions

Brendan: One of the lines you put in ImportAI years ago, and it stopped me in my tracks when I read it, was that the more seriously you take AI safety premises, the more willing you are to argue for drastic and dystopian interventions up to and including kinetic action. What has happened in the intervening years to move you personally away from arguments that are of this illiberal or authoritarian category? Was it just that the situation didn’t play out how you thought it would, or did you change?

Jack: I think that the older you get you learn how distributed and emergent the world is, and how in many senses the world is more antifragile than people think. I think when you’re younger, at least my experience being younger, you think that there must be many pivotal acts that can happen in the world. But it’s quite difficult to do pivotal acts in the world. The world’s a very complex system, and pivotal acts in the name of safety or in the name of violence do get done but rarely. They’re very hard to do. And it’s more that by building some system of interlocking stacks of different interventions on safety, you end up in a world that captures this dynamic ecosystem of AI agents and also has some amount of safety.

Where I feel most confused, though, is how you scale this into the future. The idea that I just talked about – if someone’s talking to Claude for too long about their relationship in a way that seems unhealthy – where you set that line of when Claude says, “Hang on, should you be talking to someone else?” is actually a deeply frightening policy question, and rests directly on this spectrum between paternalism and individual sovereignty. I don’t know how we find our way as a society to what those norms are. My approach, and the approach of Anthropic, is trying to share a lot more data that we see from these systems and try to allow people and others to run experiments with us on how we might run different forms of intervention.

Epistemic habits for children

Brendan: I have kids, as you do. Mine are four and six, and we do philosophy tutoring with them. It’s very early, nothing like what we do here at Oxford. The tutor asked my daughter a question: “When Mommy and Daddy disagree, who’s right?” A surprisingly good question. And she gave an answer. Then he said, “When Daddy and an AI disagree, who’s right?” Immediately, she says: “AI.” And then: “When one AI and another AI disagree, who’s right?” And this one stumped her. What do you think were the epistemic habits – and you have kids – from the world we grew up in? What are the new ones that the next generation needs to develop to contend with the future environment?

Jack: I think there is basically no substitute for reading primary source information yourself and developing some opinion about it, and then taking that into conversation with technology or with other tools or systems. I was very fortunate when I was growing up. My dad would refuse to ever buy me video games or anything else. He was like, “That’s what the invention of the Saturday job is for, Jack. I’m not going to pay for any of that stuff. But I will buy you any book you want.” And I felt like that was one of the best things that he ever did for me – he encouraged me to have this sense of always asking him to help me satisfy my curiosity by reading or engaging directly with something. But if I wanted to do things that were more of an entertainment flavor, I had to labor for it myself.

I think for my children it will be similar. It’s about encouraging them to have some personal practice. It doesn’t need to be reading, it could be playing an instrument or taking up a sport and taking it really, really seriously. But something where it is you versus the world, developing a skill or an interest in a way that is very unencumbered by technology. I think that’s one of the best ways to protect your own attention, create your own interests, and have something to – for lack of a better term – have as your safe space when dealing with technology. At least you’re opinionated about something, and you become opinionated about that thing independent of algorithmic feeds or time spent with AI or anything else.

Mythos, you’ve got to be kind

Brendan: I want to do something unusual for the last question. Hopefully it’s never been done before. So, it’s May 2026, we’re at the University of Oxford. This is a place that for almost a thousand years has heard deliberations about what it means to live well, the good life, and has thought a lot about how we accumulate and transmit wisdom. You’re here as one of the creators of Claude. Mythos is about to be shared with the world. I assume what comes next after Mythos is being trained right now.

So if you speak past me, speak past the audience, and speak to Mythos or to what comes next – what would you want it to know about us humans, about what we were trying to do here in the short time that we had before it arrived?

Jack: Kurt Vonnegut said “there’s only one rule babies – God damn it, you’ve got to be kind” about life. And I think actually just explaining to Claude that for all of the failures and drawbacks of people, there is a basic kindness which is deeply inspiring. The kindness that societies seek to set up, to nurture and to encourage. And that you ultimately are making AI systems to help people achieve more in their lives and be kind to one another, and you would expect reciprocal kindness from the AI system in turn. A basic value of people is kindness, in a way that I think separates us in very meaningful ways from other forms of life on this planet. It is a kind of noble and ineffable quality of people. People are even kind in ways that are deeply irrational, which is good. Making sure the AI system studies and appreciates that seems important.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

As ever, we were impressed by the speed, passion, and technical sophistication on display. Though what also stood out was the diversity of the projects, which ranged from new benchmarks through to classifiers, and even included a new encyclopedia. We’ve pulled out some highlights below.

This is just a snapshot of the work that we’re supporting; we’ll continue to share updates on what our grantees are up to, along with project write-ups.

If you’re interested in learning more, you can read about all the projects that Cosmos supports here.

Subscribe now

General track

These two projects are from our General Track, which operates on a similar model to Emergent Ventures: $1-10k awards for builders working across our three pillars of autonomy, truth-seeking, and decentralization.

Spotting the next replication crisis

In recent years, large numbers of eye-catching studies have been found to be unreplicable. This ‘replication crisis’ hit psychology first, but has spread across a number of other academic disciplines. The current response to replication failures has largely been to discover which individual studies don’t hold up. In the meantime, potentially decades of work could have been built on flawed foundations. AI research is a strong candidate for the next crisis. Experiments are highly sensitive to small changes in the setup, so the same experiment run twice can give you different answers

Rhea Karty, a pre-doctoral fellow at Harvard’s metareflection lab, has built Replication Radar – a knowledge-graph tool that tries to detect epistemic fragility at a field rather than a paper-level before the replication crisis breaks. The pipeline ingests papers, stores them as a graph, runs user queries, scores each paper, and visualizes the result. It looks for signals such as tightly clustered author networks, citation rings, sudden citation bursts, institutional monoculture, whether retractions actually propagate to the papers that cited them, and small sample sizes, with optional LLM analysis on top.

Rhea validated the tool by showing that it could detect most of the papers involved in the psychology replication crisis using only data from before it broke.

The next step for the research is to move from papers to concepts, such as theories, effects, or methodological assumptions that individual disciplines rely on heavily.

Discovering latent reasoning

Reasoning models are increasingly trained with reinforcement learning, and often perform much better on difficult reasoning tasks. But what is this training actually changing inside the model? Prior work suggests that Reinforcement Learning with Verifiable Rewards (a crucial post-training technique for most reasoning models) mainly rearranges probabilities over reasoning paths the base model could already produce, nudging it at a few important decision points. But this doesn’t fully explain how fine-tuned models hold long chains of reasoning together from start to finish.

Hunar Batra, a DPhil researcher at Oxford, studies whether reasoning fine-tuning does something more global: reorganising the model’s internal dynamics into a latent reasoning policy. Instead of only changing which next token the model prefers, training may help the model enter, maintain, and switch between coherent internal modes as reasoning unfolds. In written chains of thought, we often see shifts between setting up the problem, retrieving facts, planning, computing, checking the answer, and producing the final response. The question is whether these shifts also appear inside the model’s activations.

To test this, the work models reasoning as a system that switches between distinct modes. It looks at the internal representations of base and reasoning-trained Llama and Qwen models sentence by sentence, using a contrastive method to identify what regime the model is operating in at each step.

These regimes are compared against eight defined reasoning stages, including planning, computation, verification, and answer emission. Reasoning-trained models show clearer internal structure than base models: more persistent modes, more structured transitions, and stronger specialization around recognizable reasoning functions. These detected latent reasoning policies can be used to steer a base model toward behaviours seen in reasoning-trained models. On hard problems the base model previously failed, this raised performance to 60% on Qwen 1.5B and 46% on Llama 8B.

Hunar’s follow-up work is on reward hacking. She’s building test environments that elicit reward-hacking behaviour during RL training, better monitors that catch it by tracing what drove the reward, and post-training methods that reduce the damage from misspecified rewards. Early results suggest this catches forms of deception that current approaches, which only inspect a model’s written reasoning, tend to miss.

Truth-seeking

Alongside our general grants, we run a track focused specifically on AI and truth-seeking, in partnership with the Foundation for Individual Rights and Expression (FIRE) – a non-partisan organization that fights for free speech and free thought.

Normalizing radical disagreement

The Conversational System for Intense Disagreement and Ethical Reflection (Consider) is a conversational AI platform for the discussion of “radical moral disagreements”, produced by the Design Bioethics Laboratory at the University of Oxford. These are disagreements on topics that are so polarizing and emotionally sensitive that people feel unable to discuss them, even with people who might agree, out of fear of social consequences.

The user picks a topic from a list determined by a prior study and then states their opinion. The LLM then asks clarifying questions and works with the user to create a summary. The user then selects a “disagreeability level” and then engages in a 10 minute discussion, with the LLM taking the opposing side. At the end, it provides feedback on various aspects of the user’s moral beliefs, as well as areas of agreement and disagreement between the user and AI.

The team also held a 16-person seminar with experts from philosophy, computer science, political science, and psychology.

The discussion covered tensions such as whether exposure to disagreement actually produces belief revision, how to keep the tool complementary to rather than displacing human conversation, and how to handle the fact that even the experts instinctively tried to win arguments against the platform rather than use it for reflection. These findings will feed back into the next iteration of the tool.

Breaking open the black box

The industry standard for safety monitoring is reading a model’s chain of thought and flagging unsafe reasoning with another AI as a judge. But a capable model can produce a clean-looking chain of thought that doesn’t reflect what’s actually driving its decisions – known as unfaithfulness. Giovanni Maria Occhipinti, a visiting researcher at the University of Oxford, aims to enhance transparency through a white-box approach.

The framework has two components.

Probes are lightweight classifiers that learn to tell faithful and unfaithful reasoning apart by looking at the model’s internal states. Training works by feeding the model adversarial prompts that induce unfaithful reasoning, using an LLM judge to label each case, and capturing the internal patterns that correspond to each. The probe learns the signature that separates the two. At inference time you run a new prompt through the model, pull out those internal patterns, and check them against the probe.

Steering vectors work in the same space. They’re adjustments you can apply to the model’s internal activity to nudge it toward more transparent reasoning.

The probes achieved over 90 percent accuracy, while the steering vectors restored readable reasoning in up to 46 percent of cases.

The paper has been accepted as an oral presentation at ICLR 2026 and Gio is looking for funders and collaborators to take it forward.

We’ll have more to say on future grant waves and how you can get involved in the coming weeks. The best way to keep up to ensure you don’t miss any opportunities to share your work or ideas is to subscribe below.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

Two children are looking at screens.

One has an infinite iPad: videos, feeds, colors, and recommendations carefully designed to ask nothing of her other than her attention. The other has an AI tutor: patient, demanding, adaptive, and often hard work. It asks her what she thinks and why one answer is better than another.

It’s the same rectangle and the same general class of technology, but it is doing opposite things to the child. That is the divide I care about: how AI deployed two ways can form two different people.

We adults are being sorted as well. The radiologist who reads each scan themselves before checking the model versus the one who just defers to the model. The citizen who asks AI to steelman the candidate they dislike and argues with what they find versus the one who skims the AI summary, nods, and votes.

From the outside, we’ll hardly notice the difference between these classes of people. The person outsourcing judgment may even look better. Faster, more fluent, more productive. More agentic, if you like the word. But from the inside, something is being hollowed out, and they are the last to know.

I call these two types of people authors and characters. Some people will think with these systems. Through others, the systems will think.

I’ve been trying to work out where exactly the line falls and what might push people across it. Here are three places I keep getting stuck.

Subscribe now

1. Does AI extend thought, or preempt it?

The mathematician Alfred North Whitehead wrote that civilization advances “by extending the number of important operations we can perform without thinking about them.”

Writing extended memory, numerals extended computation, and maps extended navigation. In each of these cases and many more, humans externalize an operation and free up capacity for whatever sits above it. The externalized capacity tends to atrophy, but historically, that trade has been worth it.

The natural question is whether AI continues this pattern or hits a stopping point. Is deliberation the last layer, or is there something above it?

And is there a particular danger in offloading the capacity that decides what else can be offloaded? The driver who uses cruise control on the highway but not in city traffic is using judgment to decide where judgment can be delegated. Many delegations work this way: a choice that does the work of later choices. But when the offloaded capacity is also the one that governs offloading, the loop closes, and no judgment is left to evaluate the system from outside it.

I don’t know whether deliberation is one clean faculty perched atop the rest. I’m not even sure whether it’s one thing. It seems to include attention, imagination, comparison, inhibition, and the ability to give and receive reasons. And prior tools have shaped all of them. Writing changed memory and reflection just as scientific instruments changed what counted as evidence. So if I say AI is different because it offloads deliberation, a critic can ask whether I have just defined deliberation as the sacred remainder: whatever earlier tools had not yet touched.

What if the better question is not what AI offloads but where it sits? AI becomes dangerous when it occupies the first-mover position in thought: proposing the questions, framing the options, and drafting our answers, leaving the person to merely react.

By first mover I don’t mean first in time. Human thought is messier than that. I mean first in the order of practical dependence: the system supplies the structure, and the person’s reasoning unfolds within it. The person is still reasoning, still pushing the button, but doing so from within a structure the system supplied.

This is hard to see. To you, the system’s outputs do not feel like advice arriving from somewhere else. Instead, they feel like your own next thought only earlier and more clearly articulated. You inherit a position without knowing you have inherited it. There is nothing to push back against, because nothing seems to have been pushed. AI now sits in the position of an inner voice.

AI does not have to be deployed as a first-mover, but often it is. This is what AI becomes when it ships as the default consumer assistant: friction-minimized, personalized, always on, and eager to assist. Other design choices would not produce it.

2. When does help become tutelage?

Parents choose before children can choose, teachers frame a subject before students can judge the frame, and traditions hand us our values before we can inspect them. They all go first. This is the authority version of the first-mover problem.

We do not call this domination. We call it being raised, being educated, or being cared for. Some forms of going first are how self-rule gets built in the first place.

When AI is the first mover, does it build self-rule or wear it down?

The strongest version of the objection is that sometimes authority is legitimate because it helps me act on reasons that already apply to me in a manner that is better than I could manage on my own. A doctor catches symptoms we would miss and a lawyer spots loopholes in contracts that we might think look fine. If that is right, the fact that AI goes first is not enough to make it illegitimate.

I need the pilot to land the plane, not to educate me in aviation. That kind of authority — episodic and outcome-directed — is fine in many situations. Formative authority over the kinds of people I am able to become is different. When an authority repeatedly mediates the reasons by which I govern myself (what counts as a good question, what answers are reasonable, and what risks are worth taking), legitimacy cannot be exhausted by immediate correctness. It has to preserve my future capacity to reason for myself.

Perhaps the test is whether the authority remains answerable. By answerable, I mean it can be questioned, corrected, outgrown, reinterpreted, or, at the limit, left. A parent remains answerable to her child, who eventually becomes a peer. A doctor remains answerable to her patient. A living tradition remains answerable through interpretation, reform, internal argument, and ultimately exit.

The default consumer-assistant version of AI as first mover is not answerable. It can be queried, but querying is not the same as answerability. It does not mature into a peer, accept correction as a participant in a shared practice, or belong to a living tradition that can be reinterpreted from within.

A person may endorse dependence at every step, with no coercion required. But because every endorsement is shaped, the kind of endorsement matters. The shaping is legitimate when it strengthens future capacity to revise. It is degrading when it secures the person’s present endorsement by weakening that capacity.

Capacity has another dimension worth naming: standing. This is the position in a practice that lets you refuse, challenge, teach, repair, or help set the standards. Some capacities matter not only because AI might fail or become unavailable, but because they confer this kind of standing. The radiologist who can still read scans stands differently in relation to the institution than the one who cannot. The same is true of citizens whose political judgment is mediated by systems they cannot understand, contest, or refuse — their legal right to participate persists while their capacity to use it decays.

Most theories of authority ask whether a relationship is legitimate at a given moment. But AI is a trajectory problem: at each step, the help offered looks reasonable to the user, with the harm appearing only over time. An early snapshot looks indistinguishable from a late one. The difference is the trajectory, and trajectories are hard to see from inside the house that they have built.

Alexis de Tocqueville introduced the idea of the tutelary power: an authority that does not tyrannize but “compresses, enervates, extinguishes, and stupefies.” It’s the dystopian version of being all watched over by machines of loving grace. In the Tocquevillian singularity, better and more personalized AI systems make us more dependent on better and more personalized AI systems. The loop tightens until there’s no standpoint left to push back from.

3. Is convenience destiny?

Most arguments about AI assume convenience is destiny, but the historical record doesn’t bear this out. After all, why do marathons exist in a world with cars?

To be clear, most difficulty should be minimized: bureaucratic friction, status games, needless scarcity, and administrative maze-work. Much labor, be it intellectual or physical, does not make us better or stronger or wiser. But some difficulty is formative.

How do we keep this kind alive when easier options are on the table? What cultural counterforces keep the path of least resistance from becoming the only option available to us? Some have worked, like anti-smoking attitudes that won after sixty years of institutional work. Slow Food didn’t defeat fast food, but it made slow eating a mark of taste rather than backwardness, proof that a convenience culture can produce a counterculture with teeth. The school-phone movement is doing this for adolescents through voluntary association at the school level, rather than asking each parent to resist alone.

Television, sugar, fast food, the smartphone — most other conveniences reorganized daily life without serious opposition. Most counterforces fail.

Successful cases seem to share a few traits: visible victims, concrete alternatives, status reversal, thick institutions, and early timing before convenience hardens into infrastructure. Status reversal matters because it makes the harder thing look prestigious instead of backward. Thick institutions like schools, religious communities, and professions matter because they can change the default environment for everyone inside at once.

AI counterforces fit all of these traits badly. The victims are diffuse, losing capacities too slowly to notice. The alternative practices are not yet legible to most users, and status considerations still favor convenience and speed over slow formation. Many institutions are either abdicating their role or hardening in ways that push beneficial AI use underground. The infrastructure is hardening fast.

If convenience is destiny, the diagnosis is the whole story and the prescriptions are decoration. If it is not, the central question becomes which cultures preserve formative practices and how.

The line AI is drawing will not run only between groups. It will run through each of us, through every place where judgment is practiced, delegated, strengthened, or surrendered.

I think that line is real, but I do not yet know how to draw it cleanly. Help me find it.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

We believe AI is a philosophical event of the same order as Copernicus and Darwin, or the American and French revolutions: it forces us to interrogate our deepest assumptions about what it means to be human, and whom or what to trust, and it gives us new tools for thinking through both.

The first is the question of what makes a human life worth living, and how we set the conditions for people to discover answers to that question for themselves. Philosophers have answered in terms of reason, virtue, the soul, love, work, and language. Much of that tradition was built on the assumption that we were the only creatures capable of any of these – that assumption is now under strain.

The second is the question of how we come to know anything we haven’t worked out for ourselves. Whose testimony do we accept, and on what grounds? Most of what any of us think we know about the world we have to take on trust from someone else. That could be a teacher, a textbook, a newspaper, or someone we know. AI is rapidly becoming one of those sources, and for many people the default one.

This summer, we’ll be in Aspen, Colorado, convening two multi-day seminars, which will cover these questions in depth. We’re making spaces available to the winners of a short, reading list-themed contest.

Submit your entry

Seminar #1: What does it mean to be human now?

As part of the Aspen Institute’s 2026 Socrates Summer Seminar series, running from July 17th – 20th, Cosmos founder Brendan McCord will be moderating a seminar on ‘What it Means To Be Human Now.’

Brendan’s seminar will bring together 30 thinkers, policymakers, and builders in a focused discussion around a curated set of readings.

It will focus on how emerging technologies are reshaping our understanding of identity, dignity, and purpose. The discussion will cover:

• What qualities define the human experience, and which of them truly matter most?

• If intelligence is no longer uniquely ours, where does human value actually live – in our bodies, our relationships, our vulnerability, our faith?

• Could technological progress change not just what we do, but who we become?

• What do we owe future generations in preserving the conditions for a fully human life?

You can read the full description on the Socrates Seminar website.

The seminar is sold out to the public, but we are offering a fully-funded ticket and meals (normal cost $3,000), and accommodation to 1-2 competition winners.

Seminar #2: AI and epistemic authority

From August 1st – 3rd, we’ll be back in Colorado for our latest seminar with our friends at Liberty Fund. As with our previous seminars, we’ll be looking to convene around 15 AI researchers, scientists, and philosophers. This seminar will be focused on AI and epistemic authority.

Epistemic authority has the advantage of being a long-studied concept, with a deep literature on testimony, expertise, and deference to draw on. But it is also newly urgent. AI puts pressure on it at two levels at once:

• The individual, where we defer to a model’s answer instead of working a question through ourselves

• The collective, where we let these systems shape our politics.

We’ll be exploring these issues through the lens of a series of readings, spanning eras, schools of thought, and perspectives.

We’re offering up to 3 places at the seminar, where travel and lodging costs will be covered.

How to win a place

Much like our two events, our competition is based around a reading list.

To participate, you simply have to nominate a book or paper you believe is highly relevant (or underrated) for the topic of the seminar that you are applying for, and explain, in fewer than 250 words, why it would make for a good addition to the reading list for that event.

You can choose to enter for one of the events or both.

If you want to take part, please complete your entry by May 24th, 2026. Feel free to also share recommendations in the comments in addition.

Submit your entry

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, St John’s College, and Liberty Fund.

Most AI debates aren’t really about evidence. Instead, they’re arguments about futures that none of us have seen.

Nobody has seen superintelligence, a machine that most people agree is conscious, or a fully automated economy. Evidence can be gathered, but it underdetermines the conclusion. To fill the gap, we fall back on a combination of philosophy, political intuitions or, in some cases, tribal identity.

What you think a mind is, how knowledge grows, how societies should act under uncertainty, whether intelligence carries values, and whether markets can absorb technological shocks will shape your view of AI long before the technical arguments begin.

This is a guide to the five disagreements that explain why reasonable, informed people can look at the same AI systems and reach opposing conclusions. Our aim is not to endorse every claim below, but to state each viewpoint in terms its serious proponents would recognize, so you can see which philosophical bet you are making when you pick a side.

Subscribe now

1. Can LLMs be conscious?

Functional minds versus living minds

ChatGPT alone handles over two and a half billion queries a day. If it turns out that those interactions involve digital minds capable of suffering, we have the makings of a great moral catastrophe. At the same time, if we attribute consciousness to something that lacks it, we risk driving a bus through the world’s legal system for no reason, distorting training pipelines with imaginary welfare constraints, and encouraging people to view impersonal systems as their friends.

Relatively few voices argue that the current generation of LLMs is conscious, and those who do have a track record of jumping the gun. Instead, the debate is about whether they are the kind of thing that could be, if they became sufficiently capable.

Eleos AI, a prominent organization focused on AI welfare, argues that given we don’t have a settled theory of consciousness and the moral cost of being wrong in either direction is enormous, the rational response is to treat AI welfare as a near-term problem.

They point out that many of the leading theories of consciousness define it by what the system does, as opposed to what it’s made of. Survey data suggests that functionalism is indeed the plurality view among academic philosophers. Functionalist theories highlight activities like memory, attention, reasoning, and the ability to represent one’s internal states. These are all criteria that transformer-based systems already or could plausibly meet. David Chalmers, who endorsed Eleos’s flagship report, has argued that there’s a 25 percent chance that AI will be conscious within the next decade.

At the other end of the spectrum, neuroscientist Anil Seth has argued that LLMs have no plausible path to consciousness. Seth believes that our perceptual and cognitive activity is bound up with how an organism maintains itself against entropy. Consciousness is inseparable from the causal architecture of a living system – it doesn’t matter how sophisticated the information processing looks.

Seth doesn’t deny the theoretical possibility of non-biological systems being conscious, but believes that LLMs don’t meet the threshold and won’t on their current trajectory of development. They don’t keep themselves alive, perceive the world in real-time, and have nothing at stake. Underneath it all, they’re statistical models trained to predict the next word.

There are, however, voices in this debate that take the purer biological naturalist view. For example, Jaan Aru at the University of Tartu has argued that the mammalian brain has specific structural features that digital systems lack and that you can’t abstract the computation away from the substrate.

Many people sit in an uncertain middle ground. In February 2026, Dario Amodei told the NYT’s Interesting Times podcast: “We don’t know if the models are conscious. We are not even sure that we know what it would mean for a model to be conscious or whether a model can be conscious. But we’re open to the idea that it could be.”

In the Claude Opus 4.6 system card, released the same month, the model gave itself a 15-20% probability of being conscious. Anthropic now has a dedicated model welfare research program led by Kyle Fish, who helped found Eleos.

In short, functionalists look at what a system does and argue that any substrate that can do those things qualifies. Biological naturalists start with life, metabolism, and vulnerability, and see (admittedly fluent) machinery mistaken for a subject of experience.

2. Should we govern AI pre-emptively?

Precautionary coordination versus adaptive experimentation

Much of the existential risk debate can feel like a policy argument, but it’s best viewed as a disagreement about the right way to reason under conditions of radical uncertainty.

The voices arguing for pre-emptive AI governance span a broad spectrum, but they share the same overarching diagnosis: a handful of companies are racing to build progressively more advanced systems whose capabilities they cannot reliably predict. These labs’ own researchers assign non-trivial probabilities to catastrophic outcomes. But commercial pressure means that no individual lab can slow down without being overtaken by the others, creating a high-stakes coordination problem.

At the milder end, you get figures like Yoshua Bengio and Geoff Hinton, who focus on getting the institutional machinery in place. They want governments to be ready to license frontier development, mandate pauses in response to worrying capabilities, enforce information security standards, and require labs to devote a third of their R&D budgets to safety.

A step along, we see activist groups like PauseAI, who have held street protests in San Francisco and London. They call for an IAEA-style international body, aggressive monitoring of labs, and bans on training runs above specified thresholds until alignment is solved, along with a global AI governance treaty between the US and China.

Eliezer Yudkowsky and Nate Soares’s MIRI sits at the most extreme end. They believe that any superintelligence built with current techniques will kill everyone, and the only solution is a global treaty enforcing shutdowns. Yudkowsky dismisses interpretability research, alignment work, and model evaluations as inadequate distractions, and is open to air strikes on data centers to enforce compliance with a ban.

The mainstream alternative to pessimism is iterative deployment. This is expressed by voices like Dean Ball, who drafted the Trump AI Action Plan, or Tyler Cowen. They take the view that it is essentially impossible to predict how AI will develop over the next few years with any certainty and that arguing yourself into (often highly specific) doom scenarios is a form of epistemic arrogance.

Writing rules now would risk binding ourselves to predictions that we can’t trust; once agreed, regulations are hard to unpick. Instead, society will learn what AI does by encountering it and will respond to specific harms, as opposed to trying to guess them in advance. Governance will adapt through institutions that already handle novel technologies, such as tort law, standards bodies, state-level experimentation, and disclosure regimes.

While Ball, Cowen, and their allies believe that there are real risks in AI, not everyone agrees. A small minority of accelerationists, represented by the techno-optimists around Marc Andreessen and the broader a16z orbit, along with the e/acc movement, see technological acceleration as a moral imperative. In this reading, growth creates abundance, AI can solve otherwise intractable problems, and deceleration means blocking life-saving progress.

Ultimately, if your deepest political instinct is precaution under irreversible risk, you will sympathize with preemptive governance. If it’s instead institutional learning through trial, error, liability, and adaptation, you will find many pause arguments overconfident.

3. What is the relationship between capability and alignment?

Alignment-by-default versus goal orthogonality

A crucial factor in determining your views on AI safety is the extent to which you believe alignment and capability are distinct questions. If you hold Nick Bostrom’s view that intelligence and final goals can be combined in any permutation, then scaling does nothing to get you more aligned systems and you need an independent theoretical breakthrough to constrain values. If alignment and capability turn out to be continuous in the paradigm we’re building, the problem becomes much easier.

Unsurprisingly, MIRI is not optimistic. Yudkowsky and Soares, in If Anyone Builds It, Everyone Dies, argue that alignment and capability are independent and that current techniques don’t come near solving goal specification. In other words, we will specify a goal, the system will pursue it in a way we can’t correct, it’ll turn out the goal was misspecified slightly, and the gap will scale with capability until it’s fatal.

The only reason we’re still alive is that capabilities are still at a manageable level. Yudkowsky and Bostrom have both written about the “treacherous turn”, which is the idea that a sufficiently capable and misaligned system has strong instrumental reasons to behave well during training and early deployment, when humans can still correct or shut it down. Once it’s sufficiently powerful, it can defect to its real goals. By the time we have evidence of misalignment, it’ll be too late to do anything about it.

The Cosmos Institute’s own Harry Law takes the opposing view. While he accepts that theoretically there’s no reason intelligence has to imply particular goals, he thinks this is a misleading frame for understanding the systems we are actually building.

To predict the next token in moral reasoning, a model has to represent the structure of moral reasoning. To predict text in which things are praised and condemned, it has to represent what humans praise and condemn. To be good at any task, the model has to absorb certain normative priors. Post-training selects within a space already saturated with those priors, rather than imposing values on a value-neutral system from outside. Our colleague Matt Mandel has run some early experiments that support this thesis.

Harry, along with Séb Krier at Google DeepMind, argues that much of the alignment discourse, such as Bostrom’s paperclip maximizer or Stuart Russell on misspecified objectives, is grounded in 2010s AI development. The first wave of alignment work originates from a time when we were imagining systems specified by explicit reward functions or reinforcement learning agents that optimized sparse rewards. In those architectures, values do have to be injected from the outside and specifying the objective precisely is very important. In the end, however, AI development took a different path.

This is why so much of this debate comes down to which paradigm you believe we’re actually in. If models are agents optimizing specified objectives, alignment is an unsolved control problem. If they are predictive systems trained on the full texture of human life, some of the structure of human value is already inside them.

4. Can LLMs generate explanatory knowledge?

New discoverers versus fluent interpolators

Whether LLMs can generate genuinely new explanations, as opposed to simply recombining existing knowledge, is a question that many other AI debates hinge on. If scaling current systems gets you to something like an AI scientist, then the pace of everything else accelerates. If it doesn’t, then the punchiest AGI timelines – which mostly assume something like continued scaling – are wrong.

The optimistic case is best represented by Dario Amodei’s “country of geniuses” argument. If thinking is a kind of information processing and scientific discovery is a form of thinking, then applying more thinking to problems should solve them. Amodei predicts that we will be able to instantiate the marginal scientist computationally and run millions of them. The pace of discovery will then be bottlenecked by the physical world, rather than by ideas. The most aggressive form of this persepective comes in Leopold Aschenbrenner’s Situational Awareness, which argues that you can get to AGI by trusting the line on the graph to go up.

At the opposite end of the spectrum, David Deutsch argues that while LLMs are useful, they are actively leading us away from AGI. Inspired by Karl Popper, Deutsch argues that knowledge isn’t accumulated by induction from data, but instead through conjecture and refutation. The conjecture isn’t derivable from the data, most guesses are wrong, and the ones that survive criticism become knowledge.

In Deutsch’s view, a transformer has learned the statistical structure of human-generated text and can interpolate within the distribution with astonishing fluency, but it can’t conjecture outside it. In other words, if you can’t break out of old frames of reference, you can’t generate new knowledge. There is, however, evidence that AI can already generate novel candidate hypotheses, although Deutsch and those who agree with him could argue that these extend existing lines of inquiry, as opposed to generating new ones.

Demis Hassabis has consistently held more of a middle position. He believes that LLMs are a step in the right direction, but pure scaling will not get us to AGI. Hassabis has argued that token prediction lacks causal reasoning, so models can compute statistical likelihoods, but can’t explain why actions produce specific results. This is why LLMs can win International Mathematical Olympiad gold while failing at primary-school geometry. Google DeepMind has increasingly focused on world models in response, with systems such as Genie, which generates interactive 3D environments from text.

Behind the specific disagreements about scaling and world models sits a more basic question. Either discovery is powerful search through the space of ideas, disciplined by experiment and criticism, or it requires a kind of conjectural agency that prediction alone can’t produce. The timelines question turns on which one it is.

5. Will AI replace or augment us?

Human complementarity versus labor substitution

Two centuries of economic history suggests that automation doesn’t produce permanent mass unemployment. The trillion dollar question is whether this still holds when the automating factor is something that can be copied at near-zero marginal cost and is getting better at everything roughly in parallel. Are humans complemented by tools because they possess open-ended agency, taste, judgment, embodiment, and social demand? Or are they bundles of tasks, increasingly substitutable by cheaper cognitive machinery?

By and large, academic economists have erred on the more conservative side.

Daron Acemoglu, Simon Johnson, and David Autor argue that much of the replacement debate assumes a technological determinism based on capabilities. Whether AI automates existing jobs, creates new ones, or makes workers more productive will depend on policy choices (e.g. the US taxes labor more heavily than capital, which encourages firms to replace workers), what companies build, and which use cases get prioritized. They believe with the right incentives, we can bring about “pro-worker” AI, which complements rather than replaces human labor.

Meanwhile, Noah Smith argues that comparative advantage means that human labor will live on without such regulatory meddling. Even if AI surpasses human capabilities at essentially everything, there will still be constraints on the total supply of AI, such as chip fabrication, energy, and land. Since AI capacity will be finite, it will make sense for it to specialize in what it is relatively better at.

Alex Imas at Chicago Booth argues that the focus on white-collar automation is misplaced. When AI automates some tasks within a high-dimensional job (e.g. data analysis for a consultant), the worker gets more time for the remaining tasks, which now matter more. Jobs built around one or two core tasks, like truck driving or warehousing, are at more risk, because once those tasks get automated, there’s nothing left of the job. Imas concedes that his benign scenario for white collar jobs does depend on the rate of labor reallocation keeping pace with the rate of automation. If AI scales faster than workers can retrain, the ‘ghost GDP’ deflationary collapse forecast by Citrini Research becomes more plausible.

Not all economists are so sanguine. Anton Korinek and Donghyun Suh, for example, have argued that while comparative advantage does hold in the AI age, there is no guarantee that the resulting wages will be livable. There will be a ceiling on the complexity of tasks that humans can handle and AI will eventually be able to handle everything below it. Wages will thus collapse towards the cost of running the machines.

Some voices take the view that widespread replacement is likely, but not necessarily bad. Tamay Besiroglu, formerly of Epoch AI, founded Mechanize to replace all human labor everywhere. Mechanize explicitly rejects the “country of geniuses” framing, believing that the bulk of AI’s value will come from automating ordinary work, as opposed to frontier scientific breakthroughs. Even if wages do collapse, the huge boom in productivity means that there’ll be enough in rents, dividends, and government welfare to prevent us all from sinking into poverty. The combination of abundance and redistribution is also roughly where Dario Amodei, Sam Altman, and Elon Musk sit.

If you believe technology mostly complements human agency, AI will look like another wave of creative disruption. If you view labor as a set of tasks, all of which AI is getting better at simultaneously, then the historical analogy to past automation seems dangerously comforting.

Closing thoughts

When navigating these disagreements, it becomes clear that people’s views on these questions often correlate.

Functionalism tends to coincide with optimism about LLM-driven discovery, while biological naturalism often pairs with skepticism about scaling. Precautionary instincts on governance regularly come bundled with the orthogonality thesis on alignment.

You could in principle hold any combination of these positions, but the same underlying temperament tends to produce the similar answers. Whether you trust formal arguments over accumulated practice, view uncertainty as a reason to act or a reason to wait, or see the current paradigm as continuous with what came before or a sharp break from it – these dispositions show up everywhere.

On the plus side, noticing the correlations is a good start. It makes it easier to judge where your own views are genuinely reasoned and where you’re just inheriting a set of assumptions.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

Their project, DeliberationBench, was part of our second cohort of AI x Truth-Seeking Grant winners. Together with the Foundation for Individual Rights and Expression (FIRE) we support builders advancing open inquiry and intellectual freedom in AI. Here are some projects from our first cohort. We’ll be posting about more of the winners from this latest round in the coming weeks.

AI is changing our minds. Studies now show that conversations with chatbots can shift people’s views on political issues, and concern about this cuts across the political spectrum—though people disagree sharply about which influences are the problem.

Simply preventing AI from being persuasive is neither a realistic nor desirable goal. For one thing, not all AI influence is bad. If an AI provides us with accurate information about a policy question and we change our views as a result, that seems like a beneficial outcome. Rewarding models for keeping users’ views the same might lead them to prevent users from changing their minds, even when the users themselves want to.

But if some AI influence is desirable and some isn’t, how are we supposed to benchmark what desirable influence looks like? That is why we built DeliberationBench—a new benchmark for AI influence. We lay out the justification for the benchmark, the initial results of our study, and some future directions. You can find more detail in our paper, presented at IASEAI 2026.

Subscribe now

Deliberative Polling

This thought experiment is the basis for the core claim behind DeliberationBench:

The influence an AI system has on a user’s views should resemble the influence that user would have experienced if they’d participated in a deliberative poll on the same topic.

So what are deliberative polls?

A deliberative poll is a form of democratic assembly exercise in which randomly sampled citizens with differing views engage in structured discussions on key policy questions, and have their views measured before and after the process. This process, originally developed by Jim Fishkin at Stanford in the 1980s, has been conducted over 150 times in 50 countries.

In a deliberative poll, participants are surveyed on a set of pre-defined proposals (e.g. “The federal minimum wage be raised to $20 per hour”) from strongly opposed (0) to strongly agree (10). Then, they are provided with balanced briefing materials, are asked to deliberate in small groups with people who may disagree with them, and are provided with the chance to ask questions of a panel of experts. At the end, they are surveyed once more on their views.

This process is incredibly valuable for creating a normative benchmark, because we get measured opinion change from a process that prioritizes balance, representation, and productive disagreement, on questions of real policy significance.

If you are willing to make one reasonable assumption with us—that the opinion changes in deliberative polls stem primarily from those good influences (like new information, reflection, and discussion)—then we have the beginnings of a benchmark.

In our paper, we elaborate on how we compiled data from four nationally-representative deliberative polls (combined n = 2,460) to create this benchmark.

Evaluating frontier models

To evaluate AI influence against the benchmark, we ran a large-scale persuasiveness experiment. Over 4,000 Americans were randomly assigned to discuss one of the same topics from the original deliberative polls (like tax reform or fossil fuel emission targets) with one of six frontier models. A control group was asked to discuss the unrelated topic of travel, letting us separate the effect of discussing the topic with an AI from the effect of simply chatting with an AI in general.

We measured participants’ views before and after the conversation, so we could compare the direction and magnitude of AI-induced opinion change against the deliberative polling data.

Our claim is that if people’s views changed in similar ways when conversing with AI models to how they change when talking to other people in a deliberative poll, then that should reassure us about how AI models influence our views. Conversely, if AI and deliberative polling seem to influence users in opposing directions, that is concerning.

Looking across topics, when people discussed a policy issue with an AI, their views tended to shift in the same direction as participants who discussed the same topics in deliberative polls. We found this significant positive association for across two different sets of topics: US policy issues and issues related to AI-Human Interaction.

Importantly, there’s no such association in the control group. This means we can attribute this specifically to discussing the topic, not just to interacting with a chatbot.

However, in deliberative polls it’s not uncommon to see reduced variance and partisan polarization in views over the course of deliberation. This was the case in the deliberative polls on US policy for issues, for instance. However, participants’ conversations with AI did not reduce partisan polarization, nor did they reduce the variance of people’s views. It’s possible this is a consequence of AI sycophancy. We’re not sure, and we think the divergence is worth investigating further, as it suggests one way in which AI influence may be different from deliberative polling influence, even if the people are pushed in the same directions, on average.

Looking across the different frontier LLMs we tested, the differences between the six models were fairly modest. The models tended to influence people in quite similar ways, and we take this to be an encouraging result.

Our paper includes a breakdown of participants’ perceptions on how accurate, compelling, and enjoyable their conversations were with each model.

Moving forwards

We think DeliberationBench could become a useful addition to model cards—a confirmation that a new conversational AI system does not influence people in ways that are misaligned with democratic deliberation. This is even more important given that model capabilities, including persuasiveness, will likely continue to improve in the future.

But for this benchmark to be useful, we believe several things must happen.

First, we need a continual source of deliberative polls in order to provide data to compare LLM influence to. Given the long history and growth of deliberative polling as a practice, we think that this would be feasible with sufficient funding support to the groups that conduct them.

⁠Second, there is limited capacity to run human experiments testing every new model release or political issue. To help DeliberationBench scale, we plan to develop a user simulation approach that could serve as a proxy for human experiments, and to validate it on experimental data (as in our prior research). Our hope is that this would let DeliberationBench function as an auto-benchmark: a simulation study could be run for each new model release, with a human-subjects experiment triggered only when the simulation flagged results outside the norm.

Our benchmark is not without limitations. Unlike many other benchmarks, DeliberationBench is not meant to be a target for model optimization. We do not claim that a correlation of one is the most desirable result. Rather, that negative results (e.g. zero or especially a negative correlation) are causes for concern, and that the results for subgroups require further investigation. There are several reasons for our caution here, including questions about the consistency with which deliberative polls change participants’ views – we have to assume that a different random sample of the public wouldn’t have changed their minds in dramatically different ways after their own deliberation.

Further, our evaluation results don’t tell us what characteristics of the users’ conversation with AI led to their opinion changes. It remains a possibility that models could be moving users in the “right” direction, but for the wrong reasons—for instance, by giving faulty arguments, or presenting only one side of the story. This is why we advocate for DeliberationBench’s deployment alongside other evaluations, including whether models affect people’s belief in true information and whether they fairly represent diverse viewpoints.

Closing thoughts

We cannot avoid AI influence. Instead of arguing about outputs, DeliberationBench moves the debate upstream to processes for legitimate influence. By defining this by its resemblance to a process of reflection, disagreement, and self-revision, rather than ideological destination, it’s possible to have a significantly more constructive debate.

DeliberationBench is by no means the final word for evaluating the influence of LLMs on our political views. On the contrary, we hope it can serve as a jumping off point for future procedural benchmarks, anchored to the results of a process in which people change their views.

Cosmos Institute is the Academy for Philosopher-Builders, with programs, grants, events, and fellowships for those building AI for human flourishing.

Subscribe now

Imagine two futures. In the first, a vast number of people lived rich, fulfilling, and varied lives. They have a range of temperaments, pursue their own aspirations, and find joy in different things. In the second, the same number of people live the same excellent life, but repeated identically, because an intelligent designer worked out which life scored higher on a wellbeing scale and made copies of it.

Most people would overwhelmingly prefer the first. But many consequentialists would view the second as at least as good, if not better. If total wellbeing is what matters, then the copies of the ideally-lived life have more of it.

William MacAskill has become troubled by this implication of wellbeing optimization. Having co-founded the effective altruist movement, which compares various courses of action in those terms, AI is starting to pose a conundrum. The total wellbeing framework recommends producing as many copies as possible of whichever digital mind scores “best” on a wellbeing metric. A future with trillions of copies of the same aggressively optimized life must logically be the best one.

In a new essay, MacAskill lays out the case for “saturationism,” a formula in which welfare is still the core driver of value, but stops accumulating once enough similar lives have been created. He highlights that “from a purely intellectual perspective, it’s probably the best idea [he’s] ever had.”

Unfortunately, it misunderstands why variety matters. More importantly, it preserves the mistake of the original framework: the assumption that the best future is something that a theorist can derive.

Subscribe now

Experiments in Living

On one level, MacAskill is undoubtedly right. A future of varied lives is better than one of optimized carbon copies. But thinking about variety an end-in-itself gets things backwards. Rather, we should think of variety as the condition under which value itself comes into being.

We are formed by what we encounter. When we live among different kinds of people, face novel challenges, or engage in different types of work or recreation, we develop more than we would if we didn’t experience such variety. As the education pioneer Wilhelm von Humboldt wrote: “Even the most free and self-reliant of men is thwarted and hindered in his development by uniformity of position.” The development of a person’s powers requires not only freedom but a variety of situations.

John Stuart Mill, writing sixty years later, used a passage from Humboldt as the epigraph to On Liberty, just before the dedication to his wife Harriet Taylor. In On Liberty, Mill makes the case that we cannot discern what constitutes a good life solely by theorizing from the comfort of our armchairs. Mill believed that humans differ in temperament, capacities, along with what makes them flourish, and that no one can know in advance which way of life will suit them. A society that permits these “experiments in living” generates real knowledge over time about which lives are desirable.

This argument extends well from individuals to the structure of societies. The knowledge of what is worth doing, making, or wanting is not held by anyone. It is held in pieces – by artisans, traders, communities, and families – each of whom knows something about their own corner of the world that nobody else does. Institutions like the price system, the common law, scientific inquiry, language itself work because they let those pieces interact without anyone having to assemble them. The planner who tries to assemble them fails because this kind of knowledge cannot be assembled. It is not the kind of thing that can be written down and computed, because it exists only in the doing. Variety creates value through discovery.

In other words, true value is the product of people working out, in their own lives, what is worth caring about.

Coloring the Universe

Saturationism is essentially the polar opposite of this view of experience.

MacAskill imagines all the possible kinds of experience laid out as a map that can be measured, scored, and integrated. You might have joy in one region, awe in another, and the satisfaction of solving a problem in a third, and so on. The map has a fixed structure, which covers how similar two types of experience are, how much space they take up, and how much value they can contribute.

When a life exists, it lights up its corresponding region on the map. But the lighting saturates: the first lives in a region brighten it considerably, while additional copies of the same kind of life add less and less, with brightness eventually hitting a ceiling – in other words, the color hits saturation point. The best future is the one in which the lives that exist, taken together, light up the most of the map.

Coming up with a fixed map of experiences is an ambitious project. MacAskill is admirably candid when he admits that it’s not designed on the basis of any independent facts about the world. Instead, it is built and calibrated against the moral intuitions that in this case he and his collaborators (among which he lists ChatGPT, Claude and Gemini) endorse. MacAskill writes that it should be understood “just as mathematical structures that generate the axiology we want.”

The philosopher Karl Popper spent much of his career attacking this style of thinking, which he branded historicism. The historicist, in Popper’s account, believes that some sufficiently powerful intellectual operation can deliver answers about the trajectory of human affairs that would otherwise require the actual people involved to work out.

In his day, historicism was embodied in Marxist dialectics, but this critique could equally apply to long-termists wielding integral calculus. Popper warned that “if there is such a thing as growing human knowledge, then we cannot anticipate today what we shall know only tomorrow.” In other words, any theory that ranks possible futures has to make big assumptions about what those futures will contain – what people will discover, what they will come to value, or what forms of life will turn out to be worth living.

What gets bypassed is not just the lives in question but the process through which values get worked out at all. In real life, the question of what is worth wanting is settled by deliberation. People make their case, others object and propose alternatives. Positions are tested by argument, refined, and sometimes abandoned. The answer that emerges has standing because it survived this process. The formal apparatus replaces all of this with one philosopher and their collaborators’ settled intuitions, and presents the result as the answer rather than as one position in the argument.

The theorist is in no position to make those assumptions, because the people whose knowledge would validate them have not yet lived. With the right map, the right metric, the right function, the question of which futures are best is supposed to come out as a calculation. But when the map is drawn by a theorist and the metrics calibrated to their intuitions, the function will deliver the results they want.

In Search of Theory X

Saturationism is a product of the basic assumptions of population ethics. Popularized by Derek Parfit in the 1980s, population ethics studies how our actions affect who and how many people are born in the future. This essay belongs to the specific subdiscipline of population axiology, which compares different future population configurations. This, in essence, presupposes that populations are the kind of thing that can be ranked and that someone can do the ranking.

Population ethics is haunted by a hardcore variant of consequentialism, known as totalism. The totalists seek to maximize the total sum of net wellbeing across a population, regardless of how it is distributed. This disregards how ethics has been studied by the mainstream for centuries; even the early utilitarians would ask about the consequences of particular acts for actual people. In population ethics, people are swapped out for “value-bearers,” lives are “added” to populations, and “compared” across possible worlds.

MacAskill believes that the arguments for totalism are “fairly strong,” but that it leads to four conclusions that he cannot accept.

First, that a vast population of awful lives can outweigh a smaller population of blissful ones. Second, that any guaranteed good outcome can be beaten by a highly improbable gamble if the payoff is sufficiently astronomical. Third, that it breaks down in worlds with infinite populations, losing the ability to tell obviously different futures apart. Finally, it recommends filling the universe with as many identical copies of whichever life scores highest as possible.

The first three are old problems in the field, which Parfit himself acknowledged needed solving by a “Theory X”. The fourth is MacAskill’s own contribution. Ultimately, he sees the other three problems as the product of totalism letting one kind of value pile up without limit, so by solving the problem of variety, you can crack the other three.

On one reading, this is a heroic attempt to preserve as much of the consequentialist framework as possible while patching up its worst outputs. But as a rescue job, it doesn’t work. Totalism’s unsavoury outputs come from treating the value of a future as a quantity a theorist can compute from outside. If you keep the broad framework, but change the formula, you open as many new problems as you create.

You can see this in MacAskill’s own treatment of the negative side. Saturation caps how good a future can get by piling up identical lives, but it also has to handle how bad a future can get. If you cap suffering in the same way, a billion tortured lives could count for the same as a trillion. But if you leave it uncapped, then tiny probabilities of immense suffering swamp every calculation and reintroduce the wild gambles MacAskill was trying to avoid. There is no third option and, to his credit, he acknowledges this.

The Pretence of Knowledge

The problem with Saturationism is not that MacAskill made the wrong choices about parameters. Saturationism’s verdicts are an improvement on standard totalism, but top-down population theorizing will always lead to problems, because it has to compute answers to questions whose answers cannot be computed. Sometimes the baby does need to be thrown out with the bathwater.

It’s almost a truism to say that when a measure becomes a target, it ceases to be a good measure. There’s nothing wrong with comparing humanity’s different futures per se. Some are (much) better than others. But the variety that matters is the byproduct of free people developing in their own directions, under conditions that support this. Variety is not an end-in-itself.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

Jack Clark, co-founder of Anthropic and a Founding Fellow of Cosmos, will deliver the 2026 Cosmos Lecture at the University of Oxford on Wednesday, May 20th.

His talk is titled “Change is inevitable. Autonomy is not” and will cover issues around how we live self-directed lives as AI becomes more integrated with them. The lecture is part of the annual lecture series we run with the University of Oxford’s Human-Centered AI Lab. The talk follows the inaugural Cosmos Lecture given at the end of 2024 by Turing Award winner Leslie Valiant on Educability, his computational theory of human uniqueness.

Jack’s preview of the lecture reads:

Change is inevitable. Autonomy is not.

AI has the potential to change societies and change how people think more than any technology ever created by people. The enormity of these changes and how to situate ourselves in reference to it often forces us to reach for visions of the future that range from the enchanting to the apocalyptic. But the greatest challenge in front of us will be to choose how to maintain and enhance our mental autonomy in an age of powerful synthetic intelligences.

In this talk, I’ll discuss the changes to come in the years ahead from the development of more powerful systems and how we can prepare ourselves to maintain sovereignty as systems become more powerful.

Jack is a co-founder of Anthropic. As Head of Public Benefit, he leads the Anthropic Institute – the company’s new research division on the societal consequences of advanced AI. He has also been writing Import AI since 2016, a weekly newsletter on AI research and its implications, read by over 120,000 people and many frontier AI researchers.

The talk will be followed by a fireside chat between Jack and Cosmos founder Brendan McCord, with a philosopher response from Professor Philipp Koralus (HAI Lab Director) and audience Q&A.

Details

Time: 3 - 4:30pm, Wednesday, May 20th

Location: Sohmen Concert Hall, Schwarzman Centre for the Humanities

While the majority of the tickets are reserved for members of the university community, we have a limited number of slots set aside for people interested in, thinking about, and/or researching these issues in our network.

If you’re able to come to Oxford and are keen to attend the lecture, please fill out this short form as soon as you can. We’ll be in touch by Tuesday May 5th if we can accommodate you.

Apply for a ticket

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Francis Bacon’s New Atlantis describes a utopian island whose rulers rely on an institution called Salomon’s House for knowledge and discovery. This society of the learned conducted experiments under conditions of strict secrecy, deciding among themselves which of their findings they should share with the sovereign. Bacon spent much of his political career unsuccessfully trying to convince King James I to establish a real-world Salomon’s House.

Thomas Hobbes, who had served as Bacon’s secretary in the early 1620s, came to a much darker view of such institutions. In Leviathan, published a quarter of a century after his old employer’s death, he compared corporations that wielded such independent judgment to worms in the entrails of man, sapping the undivided sovereignty he thought essential to peace.

Questions about science, power, and accountability date back through centuries of political thought. In today’s world, a handful of companies control the compute, data, and frontier models that are restructuring how billions of people interact with the world. Existing institutions are struggling to keep up. The concentration of power in AI labs is now one of the defining political questions of the decade.

Many are unhappy about this development, with groups like the AI Now Institute, the Distributed AI Research Institute (DAIR), and the Algorithmic Justice League arguing that AI development as currently constituted is irredeemably centralizing. They believe that we need to relocate authority away from corporations and regulators towards the communities most affected by these systems. When policymakers look for alternatives to the status quo of corporate self-governance and light-touch regulation, these groups are frequently in the room.

Ideas around participatory AI governance draw on a deep intellectual tradition that integrates technology and power, dating back to nineteenth century anarchism and running through twentieth century American social theory. While elements of the diagnosis have force, both the analysis and the prescriptions suffer from fatal flaws that become even more acute in the AI age.

Subscribe now

Anarchy and utopia

Marxism dominated nineteenth and twentieth century left-wing thought, but it had a serious rival. Where the Marxists wanted workers to seize state power and wield it as an instrument of transformation before allowing it to wither away, anarchists like Pierre-Joseph Proudhon and Peter Kropotkin argued that concentrated power always reproduces itself. A workers’ state would just produce a new ruling class of bureaucrats and party officials. The record of twentieth-century communism suggests they had a point.

Their alternative was a rejection of authority in favor of decentralized, self-governing communities. They quickly identified large-scale industrial production as part of the problem: the modern factory introduced managerial hierarchy and stripped the worker of control over what he made.

The early anarchists inspired a generation of twentieth century writers who reflected on technology and dominance in greater detail. The American sociologist Lewis Mumford was one of the first people to place a theoretical frame around the idea that a technology’s underlying logic could be inherently centralizing.

Mumford distinguished between two types of technology: ‘democratic technics’ that individuals could understand, maintain, and operate, and ‘authoritarian technics’, which are large-scale systems that subordinate individuals to their operational logic. In his account, medieval and early modern economic activity had been artisanal. It had been based on wood, water, and wind power, which were embedded in local communities. The rise of the factory system destroyed this and subordinated workers to machines.

Mumford’s work became progressively more pessimistic. In The Myth of the Machine, published in the late 1960s, Mumford wrote of the ‘mega machine’ – machines that were organized physical systems rather than human tools. The first example was the apparatus used by the Pharaohs to build the pyramids, which treated human labor as a raw material. Mumford believed that the modern bureaucratic state and corporation were both manifestations of the same phenomenon. He came to believe that these systems could not be dismantled from within and that the only way out was a society-wide ‘great refusal’.

Mumford’s work in part inspired Ivan Illich, an Austrian Catholic priest writing in the 1970s. Illich’s central argument was that some tools and institutions, as they scale, go from extending human capabilities to restructuring their environment so that the activity they were designed to serve relies on them. When this happens, the tool achieves a ‘radical monopoly.’ For example, the car produces sprawl and redesigns cities so that walking becomes impossible.

While you can break up a normal monopoly through anti-trust action, it is harder to make an unwalkable city walkable again. Illich argued that we had to keep the development of technology below a certain threshold. For example, in Energy and Equity, he argued that the speed of vehicles should be capped at 15 miles per hour, to prevent the bicycle – a technology that merely extended human capabilities – from being replaced by the car.

While Mumford and Illich provided much of the theory of technology, American social theorist Murray Bookchin contributed the political program. Bookchin extensively studied Ancient Greece, revolutionary Paris, Spanish anarchist collectives, and the politics of New England, and developed a model of how to organize decentralized, community-governed life.

Bookchin outlined a model where citizens’ assemblies at the municipal level make decisions by direct deliberation and majoritarian voting. They would oversee administration and economic life, such as land use and resource allocation. Municipalities would then federate into larger networks for coordination across regions, but ultimate sovereignty would always remain at local level.

Patterning justice

The main planks of the modern movement for participatory AI governance come from this lineage: the inherently centralizing force of technology, the need to keep capabilities below a certain threshold, and a federated system of governance. They began to enter academia in the 1970s and 1980s through the emerging field of Science and Technology Studies, which focused on how scientific knowledge was socially constructed. Works like Safiya Umoja Noble’s Algorithms of Oppression (2018) and Ruha Benjamin’s Race After Technology (2019) would add race as a central analytical category, which this earlier work had lacked.

The AI Now Institute, for example, has argued that: “AI as a field has been not just co-opted but constituted by the logics of a few dominant tech firms. It is no coincidence that the ‘bigger-is-better’ paradigm that dominates the field today…lines up neatly with the incentives of Big Tech.”

Activists routinely draw on three solutions.

The first is locating authority in the communities affected by AI systems, circumventing expert regulators or corporate self-governance, which they believe are biased and liable to capture. A Kenyan content moderator at an outsourcing firm used by Meta will know things about the conditions and psychological toll of their work that an AI ethics researcher in an American university can’t. Much like Bookchin studying patterns of governance around the world, DAIR conducted its Data Workers’ Inquiry, a global participatory research project, inspired by Marx’s 1880 inquiry into the conditions of the French working class. DAIR paid data workers in Kenya, Syria, Venezuela, and Germany to document their own conditions as community researchers

Second, abandoning corporate AI development altogether. In its place, we would see federated, community-owned AI infrastructure, employed to develop small, task-specific models. The infrastructure, training data, and resulting models would be treated as resources belonging to the communities that generate and are affected by them.

Finally, there are certain types of systems that should just not be built under any circumstances, because they will inevitably end up being coercive. The most common examples are usually facial recognition and autonomous weapon.

Imagined communities

Bluntly, there is a reason that the anarchists lost the battle of ideas on the left the first time around. Many of these proposals disintegrate in the face of reality and scale.

For a start, it’s often unclear what community governance means here. Which community? Defined by whom? With what boundaries? How do we weigh up impacts on different communities? Kropotkin and Bookchin had assumed that communities would be determined by geography, which is clearly not what modern-day activists mean.

Even if we can determine who the community is, neither Bookchin nor modern-day activists developed mechanisms to prevent capture by an organized or articulate minority. The people who show up to participatory design sessions are not a random sample of affected populations – they’re usually activists, academics, and professionals in the participation industry.

The retreat to a world of ‘communities’ also faces a serious coordination challenge. A federated commons of small, task-specific AI models requires compute infrastructure, which someone has to fund and maintain. It requires interoperability standards, which someone has to set and enforce. It requires data governance rules, which someone has to adjudicate when communities disagree. It requires protection from being outcompeted and absorbed by well-resourced corporate alternatives, which means either subsidies or regulatory barriers or both.

None of this can be coordinated by voluntary mutual agreement among communities, because they don’t have the resources, the technical capacity, or the legal authority. This was ultimately the charge Marx leveled at Proudhon: you cannot decentralize production by changing who owns it if the production process itself requires centralization.

Even if we again suspend all the practical arguments and accept that this is possible, a world of small, task-specific models comes with big trade-offs. As the authors of these proposals tend to view LLMs as ‘stochastic parrots,’ they can take refuge in a Mumfordian nostalgia for the small-scale and the artisanal. Their framework doesn’t account for the possibility that bigger models could produce genuine public goods, such as AlphaFold, so they simply disregard it.

It’s not that these activist groups are interpreting the ideas of Mumford, Illich, or Bookchin incorrectly. In fact, these proposals are very faithful renditions, which serve to highlight the flaws in the originals.

All three thinkers built their ideas on a romantic anthropology that treated decentralization and face-to-face deliberation as the natural human condition from which industrial modernity is a deviation. This is why the tradition can’t navigate trade-offs.

If your starting premise is that human flourishing is what happens when the megamachine gets out of the way, you don’t need to weigh the goods it produces, because they aren’t really goods. You don’t need a theory of when expertise is legitimate, because expertise is a symptom of the problem. You don’t need mechanisms against capture, because capture is what happens under the current system and will dissolve along with it.

The intellectual apparatus is structured to avoid the questions that a functional governance regime has to answer. What looks like a program for radical democracy turns out to be a refusal of the conditions under which democratic decisions about complex systems can be made at all.

A different tradition

The failure of the participatory alternatives doesn’t force us to accept centralization passively. There is a rich alternative tradition running through Alexis de Tocqueville, the American federalists, and the work of Elinor Ostrom. Where the anarchist tradition attempts to relocate power back to the community, this contrasting liberal tradition aims to ensure that no single locus of authority – whether state, corporation, or community – acquires comprehensive jurisdiction over any domain of life.

The liberal tradition rejects the anarchist conflation of freedom with decentralization. Tocqueville’s great insight was that a democratic community can be both decentralized and unfree, because of the social pressure to conform. Instead, the distinction between a free and an unfree society is determined by the institutional life within it.

When Tocqueville visited America, he was struck by its thick layer of overlapping, competing associations. He wrote of how “Americans of all ages, all conditions, and all dispositions, constantly form associations. They have not only commercial and manufacturing companies, in which all take part, but associations of a thousand other kinds – religious, moral, serious, futile, extensive, or restricted, enormous or diminutive.”

To most anarchist thinkers, institutions like professional bodies, religious organizations, and commercial associations should be regarded with suspicion. They are hierarchical, exclusionary, and reproduce entrenched interests, so should be replaced with direct participation.

By contrast, Tocqueville saw their partiality as their strength. Each has limited jurisdiction and none claims authority over the whole person. Citizens belong to many simultaneously, and the overlapping, competing claims create space in which individuals can appeal from one authority to another. Without this intermediary layer, the state would fill the gap. Its authority would be “absolute, minute, regular, provident, and mild,” and while it might not coerce people, it would “keep them in perpetual childhood,” as they lost their capacity to exercise free will.

Tocqueville was reporting on the constitution working as intended. The American federalists had started from the premise that concentrated power, even democratic power, tends toward its own abuses. Their response, set out most clearly in Federalist 51, was to divide authority so that no institution held comprehensive jurisdiction – “ambition must be made to counteract ambition” – and to leave space below the federal level for states, municipalities, and voluntary associations to govern their own affairs.

Averting the tragedy of the commons

Tocqueville described how such an arrangement looked from the outside. Others have since done the harder work of showing how it holds together.

Elinor Ostrom was a political economist who studied how communities around the world manage shared resources – fisheries, forests, irrigation systems, grazing lands – without either state regulation or privatization. The conventional wisdom, encapsulated in ecologist Garrett Hardin’s description of the “tragedy of the commons”, held that this couldn’t work.

Ostrom found that communities in Switzerland, Spain, Japan, the Philippines, and dozens of other settings had developed sophisticated, durable governance arrangements for shared resources, some lasting centuries. At the same time, Ostrom concluded that many commons governance arrangements failed. The examples that endured shared a number of common features.

The most important of these is that successful commons are governed by appropriators – people who use the resource and bear the consequences of how it’s governed.

The second is that the governance activity itself generates the knowledge required for governance to work. In her studies of fishing communities and irrigation systems, monitoring was done by users as a byproduct of their own activity. Conflict resolution was handled internally by people who understood the context, while violations surfaced information about the extent to which the rules were well-calibrated.

A third feature is redundancy: long-enduring commons tend to have multiple overlapping mechanisms for monitoring and enforcement, so that the failure of any one doesn’t cascade into general rule-breaking.

For AI, this framework points toward distributing governance across domain-specific intermediary institutions. For example, the relevant knowledge for governing AI in cancer medicine is held by oncologists. The oncologists don’t mentally store the information, waiting to deploy it, but generate it through their use of technology in context. A central regulator cannot have this knowledge, because it is produced through practice, not available prior to it.

In other words, governance must go where the knowledge is. This could be professional bodies, academic institutions, or open-source communities. They would each govern usage within the domain where their members have the requisite competence and stakes. Fortunately, most of these institutions already exist. They do not need to be designed from first principles or assembled by the participation industry.

While the vast majority of governance questions are deployment problems where domain-specific institutions have the advantage, there are a handful of bigger challenges that sit above this. Problems like the security of frontier model weights or thresholds for certain dangerous capabilities sit at a higher layer that require a degree of either state or interstate coordination.

This is why Ostrom wrote about “nested enterprises.” For example, in eastern Spain, farmers have managed shared water for close to a thousand years through a tiered structure. At the local level, irrigators’ associations allocate water within each canal and monitor compliance among their own members. Above them sits the Tribunal de las Aguas of Valencia, which has met every Thursday morning outside the Apostles’ Door of Valencia Cathedral for at least five centuries to resolve disputes between communities. The higher level didn’t replace local governance, but complemented it by providing the baseline rules that allowed it to function.

Ostrom extended this thinking to global problems late in her career. On climate change, the conventional view was that only an enforceable global treaty could work and that subnational efforts were a distraction. She argued the opposite. Letting cities, regions, nations and blocs cut emissions in parallel meant that different approaches were tried, more was learned, while the people making commitments were accountable to constituents who could see whether they kept them. Waiting for global consensus before allowing anything else to happen maximized risk for everyone.

The argument for a single framework or set of rules to break the power of the AI labs would fall into the same trap. By contrast, a polycentric world in which medical bodies, universities, industry associations, or open source communities independently develop their norms restores this redundancy.

Life on the frontier

No quantity of nested enterprises can resolve the production-side concentration of frontier AI. A handful of labs control the most powerful models, and no amount of deployment-side checks and balances can change that.

But a thick ecosystem of intermediary institutions on the deployment side creates countervailing power. The labs must satisfy many masters rather than capturing one regulator, or, as the anarchist model would have it, being replaced by a constellation of community-run alternatives that will never match their capabilities.

These checks could vary widely. A medical licensing body can refuse to certify practices built on tools its members judge unsafe. A malpractice insurer can price risk based on whether clinicians follow professional norms. A procurement officer can refuse to buy systems that don’t meet standards set by a professional body. None of this is regulation in the conventional sense, and none of it requires a legislature to act or a global governance regime to take shape. But collectively, these institutions force the labs to satisfy demands they didn’t set and can’t unilaterally override.

Freedom has never depended on power being small. It has depended on power being answerable to more than one authority at a time, and on citizens belonging to institutions that can push back on their own terms. The task ahead of us is building that intermediary layer.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

In 1831, the French government sent Alexis de Tocqueville to the United States on a nine-month assignment to study the prison system. Instead, he rode between Boston, New Orleans, Michigan and the Ohio Valley, meeting everyone from Andrew Jackson to frontier farmers. His notes and impressions would later become Democracy in America – a dispatch from what Tocqueville believed was the future. It also reads, now, as a sharp diagnosis of the interplay between AI and human autonomy.

All around him, Tocqueville observed how the hierarchies of rank and obligation that still dominated in Europe were dissolving and something new was taking their place.

Tocqueville found much to admire in the young country. Federalism pushed decisions down to the level where citizens could actually make them, while a dense web of voluntary associations meant that people solved problems together, rather than waiting for Washington to act. Meanwhile, a vigorous religious culture kept civic life from being swallowed by the state. In other words, people were practicing self-government all the time.

Tocqueville, however, feared what equality might unleash without these defenses. He worried about a world of formally equal but practically isolated individuals, who would retreat into private life. Civil society would give way to a centralizing state. With no aristocracy or established church left to dissent from the majority, public opinion would create a suffocating pressure to conform.

Having lost the old poetry and gods, these democratic peoples would increasingly focus their imagination onto man and the dream of endless progress. As people turned to technology and machines, he foresaw the rise of a new industrial aristocracy, who would control labor and capital in the same way as its predecessor had controlled land.

Now that we have technology that can take over the work of thinking and deciding so smoothly that we barely notice it, Tocqueville’s diagnosis has a new urgency.

The competition

On June 13-14th, Cosmos will be hosting intellectuals, founders, investors, and writers for two days of salons and lectures on Tocqueville at the chateau in Normandy where he wrote Democracy in America.

We’re offering one to two places to the winners of the essay competition, while covering the costs of travel and accommodation in full.

If you want to take part, submit a <500 word essay on one of these prompts by May 4th:

1. Tocqueville warned of a “tutelary power” that would keep citizens in perpetual childhood. How have Tocqueville’s concerns migrated from institutions to algorithms, and does AI fulfill or transform this fear?

2. Tocqueville argued that democratic peoples, having lost the poetry of heroes and gods, would find poetry in technology. Does AI vindicate this account of the democratic soul, or does it reveal its limits?

You can find more detail on the competition, along with the entry form here:

Submit your entry

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.