Every builder’s first duty is philosophical: to decide what they should build for. This series asks 9 questions to founders who are building towards their vision of the human good.

This week’s guest is Matt Clifford. Matt is the co-founder of Entrepreneurs First and the chair of ARIA – the UK’s Advanced Research & Invention Agency. He is the co-author of How to be a founder (with Alice Bentinck), serves on the board of Code First Girls, and writes open-source immersive murder mystery games.

Subscribe now

1. What are the core questions or beliefs driving your work?

There are three:

1. That the emergence of powerful AI is the most important transition we will have to navigate in my lifetime.

2. That Britain has the potential to be a great country and force for good in the world.

3. That there may be a deep connection between the first two ideas - and that Britain achieving its potential may be one of the most important ways to shape the trajectory of AI for good.

2. What future are you building for?

I spent most of the last 15 years building Entrepreneurs First (EF), which aims to increase the supply of great founders - and therefore great companies - in the world.

Over the last four years, I’ve combined this with trying to build UK state capacity in science and technology generally, and AI specifically - first as Chair of ARIA (the Advanced Research & Invention Agency - loosely, the UK’s take on DARPA), then as AI adviser to PMs Rishi Sunak and Keir Starmer.

In the latter roles I helped create the UK’s AI Security Institute (AISI), convened the first AI Safety Summit and wrote the AI Opportunities Action Plan (the UK’s national AI strategy).

When I look across these roles, I think the thread that runs through is something like: I want to live in a world where as many humans as possible live in conditions of abundance with real freedom to shape the course of their lives.

3. What commonly held belief in the tech community do you believe is wrong?

I think many people in tech (at least implicitly) believe something like “we are our incentives” and therefore that various bad equilibria are inevitable. I think the opposite is true, but less comfortable: virtue is a real thing and that we can bend even very powerful forces by making the right choices.

4. What are your main philosophical influences?

I’m a big believer that the genius of the British constitution is the capacity for error correction, so I’m a big fan of Popper and Deutsch in that vein. Relatedly, I think a lot of my appreciation of the value of permissionlessness and bottom-up action comes from Hayek - but tempered by people like Elizabeth Anderson and their warnings about the risk to freedom from private as well as public power.

I’d say, though, that historians have been at least as important for me intellectually as philosophers: my beliefs about British exceptionalism are strongly influenced by Alan Macfarlane and Joel Mokyr, and Quentin Skinner’s work has left a lasting impression on how I think about ideas and public argument. More recently, Henry Farrell’s work (with Abraham Newman) on weaponised interdependence has shaped my thinking about technological sovereignty (and I also like Farrell’s work with Cosmo Shalizi on “cognitive democracy” - another helpful dialogue with Hayek and markets).

5. What does human flourishing mean to you?

I find it hard to go beyond my answer to question 2, above: human flourishing is about people having the material and social conditions that mean the choices they make about their lives can be both meaningful and effective.

6. What’s one book you’ve read recently that you’d recommend?

Greg Egan’s Permutation City is over 30 years old, but it’s remarkably prescient (and I only just read it). Excellent read if you still need to convince yourself that we’re always going to be short of compute.

7. What’s your most irrational belief?

Some would say part (3) of my answer to the first question! Other than that: at some level, I do believe that I am a very lucky person.

8. What’s the most interesting tab you have open right now?

Probably this visualisation of the AI supply chain and who owns it.

9. Who is one writer or thinker today who you think is underrated?

Is Séb Krier still underrated? Maybe so - still under 25,000 followers on X - even though he is now much better known than a year ago. Jack Wiseman’s Inference and Rohit Krishnan’s Strange Loop Canon should each be at least ten times bigger too.

You can read more about Matt’s work on his website and get in touch with him on X.

This is the fourth instalment in this interview series. You can also see our interviews with AI Underwriting Company co-founder Rune Kvist, ex/ante founder Zoe Weinberg, and Zena Hitz, the founder of the Catherine Project.

To nominate someone for “What Will You Build For?” leave a comment below, or send us a DM.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

Human language runs at roughly forty bits per second, whether you’re reading, speaking, or listening. It is a hard limit on how much information one person can convey to another. It is also arguably a ceiling on how well communities can coordinate, since they are often reliant on a handful of people able to turn what others are saying into collective action.

But what would happen if we didn’t have this ceiling?

Edge Esmeralda is a pop-up village in California, running from May 30 to June 27 in Healdsburg, California. The project is aiming to bring over 500 scientists, artists, builders, and thinkers together for up to four weeks. Attendees aren’t expected to commit for the entire month – there will likely be 150 people on site at any one time. There’ll be a range of programming, including fellowships and residencies, summits, and opportunities for collaborative working.

The second week of the event (June 8–14) will feature a dedicated set of sessions on intelligence and autonomy, covering topics like AI agents, autonomous infrastructure, neurotech, privacy-preserving systems, and governance design.

Cosmos is excited to be partnering with Edge Esmeralda to deliver the Agent Village Experiment for the duration of the month.

Led by Ivan Vendrov (independent researcher and Cosmos Founding Fellow), Timour Kosters (Co-Founder, Edge City), and Harry Law (Principal Researcher, Cosmos Institute), the experiment will grant every multi-day attendee access to an AI agent running on their behalf throughout the village.

This personal agent will help humans to navigate the schedule, the wiki, and community governance. More excitingly, it will also exist in a shared digital plaza and will help attendees meet people who share common interests, propose dinners, negotiate community decisions, and participate in community governance. In essence, it’ll play a valuable role in making connections that might otherwise be missed due to humans’ lack of bandwidth. The experiment is supported by Foresight Institute, and the technical collaborators are Index Network, Geo Browser, Instaclaw, and Tule Romeo.

The experiment has been inspired by a number of different ideas, including Ivan’s work on supercooperation and Séb Krier’s essay on Coasean bargaining. Both are excited about the potential of decentralized AI agents to unlock gains in coordination, such as better mutual understanding, cheap bargaining, and conflict resolution.

By using real-world scenarios, the experiment aims to shed light on how trust develops in agent-to-agent relationships over time, the kinds of actions people do and don’t find useful, or whether AI leads to better community deliberation and decision-making. The goal is for this to be the largest live experiment in human-AI collective experiment to date. Historically, most of the experiments around agent cooperation have taken place in synthetic environments, so a large-scale experiment in a real-world environment presents an important early opportunity for learning.

You can read more about the experiment, the methodology, and Edge Esmeralda’s predictions here.

We have a small number of complimentary tickets for the event, which we are offering to individuals in our network. So if you’re interested in Edge Esmeralda’s programming and the chance to take part in the experiment, please fill out this short form as soon as possible. The deadline for entries is May 31 and winners will hear back from as feasible.

Apply for a ticket

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

In his lecture, Jack covers advances in capabilities, how his own relationship with AI has evolved, and some scenarios for future progress.

We’re delighted to bring you the full video of the lecture and fireside chat. If you’d prefer to read the transcript, Jack has posted it in full over at Import AI, his weekly newsletter on AI research.

We have more talks and events like this over the rest of 2026. The best way to ensure you don’t miss out is to subscribe.

Subscribe now

Highlights

12:24 - Jack’s changing relationship with AI

24:40 – how AI changed Anthropic

44:40 – start of fireside chat

50:08 – live by the Claude, die by the Claude

1:01:00 – what Jack would say to Mythos

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

On Wednesday, Jack Clark, co-founder of Anthropic, delivered the second Cosmos Lecture in partnership with the Human-Centered AI Lab at the University of Oxford. The lecture was introduced by HAI Lab Director Professor Philipp Koralus and was followed by a fireside chat between Jack and Brendan McCord, the founder of Cosmos Institute.

We’re bringing you a lightly edited transcript of the conversation, which covered what AI cannot do for us, whether Claude makes us better thinkers, and what Jack wants future AI systems to know about humanity. The full lecture will follow in the coming weeks.

Philosophy to code

Brendan McCord: Jack, you ended with the call to build the new world. It makes me think – if you and I were having this conversation 250 years ago, the proudest project we could have possibly been engaged in would have been building a new world of sorts. I would call it the philosophy-to-law pipeline. We would have been looking to Oxford intellectuals like John Locke, or Montesquieu, Livy, Adam Smith, and translating that into a constitution that we hoped would frame freedom for the 250 years to come.

The proudest project we can engage in now is, as you say, this new world-building project – it’s philosophy-to-code. What would you say about the extent to which the frontier labs take that seriously? What can we do to really take that seriously in places like Oxford and academia? And what should we do in nonprofit land to take that philosophy-to-code project seriously?

Jack Clark: I think it requires you to basically accept that progress will continue and try to model out scenarios based on it. I think dealing with COVID highlighted that, though there’d been some modeling of what would happen, if you had very fast take-off propagating viruses the world would break very quickly. We felt underprepared, and that we could have done more scenario work and forecasting of what these strange things would do to us ahead of time.

Within the AI labs, I think there is now work at all of them on trying to imagine what you might think of as “post-AGI worlds,” or worlds that happen after recursive self-improvement. But my general sense is every time you sit in a room at the lab, people say, “Are we the only people working on this?” And you say, “I’m terribly sorry, yes.” And then some people put their head in their hands and wish that more people were working on it.

The good news is that this is exactly the kind of work that universities and other organizations are built for because you don’t need to be running a large-scale supercomputer or training a very capital-intensive model. You need rather to model out, in a theoretical sense, the properties of an AI system that can massively multiply productivity, or an AI system whose inference costs fall at X rate, and capabilities rise at Y rate. What does that do to the economy? What are the things that it unlocks? What are the aspects of this supply chain where you might invest, or change the supply chain to actually change the character of the systems? There’s tremendous work to be done.

I’ve been in the UK, in part speaking with the UK government, and I made this point: if the UK government just had 10 to 20 people whose sole job was modelling out what happens if the technologists are right about this technology, the UK would be better prepared than any other country in the world – because so little work has happened. So it’s a great time for universities to be doing these projects.

Brendan: So should Madison, Hamilton, and Jay have spent a lot more time on forecasting than they did on debating the nature of man and the political order?

Jack: It’s a hard question. I feel like you have a take on this!

Brendan: I think we can’t miss the part of contemplating about the ends. And I think what brought them together was a kind of unique epistemic humility that they shared with the Scottish Enlightenment thinkers.

Jack: My assumption with AI is that there is a huge value in norms and precedent – which is, how do we want these systems to show up in the world? I’ve covered that a bit less in my talk, but it relates to how we shape the so-called “character,” or what some might say personality, of these systems. How do we want them to behave towards us? This is a normative question – a philosophical question – and we should absolutely work on that.

But I have been struck by how surprised even the AI labs have been by their own progress, which is a very counterintuitive thing. We work with these AI labs and they keep saying, “Well, you know, as we said last year at Anthropic, we did a load of work on the increasing rate of cyber-hacking capabilities of AI systems.” I run a team that does this. We wrote blogs saying, “Oh, that’s interesting. Surely it has some implications if the system suddenly becomes capable of nation-state-grade stuff. We seem to be on a trajectory here.” So we did some prep work, and then nonetheless we made Mythos and were like, “It’s here faster than we thought. We’ve done insufficient preparation.” This is true of every single time AI progress has happened. People have been continually surprised by how significant the jumps have been and how quickly they’ve come. So we all need to do more work on this.

To defer or not to defer? The case of the Claude Boys

Brendan: I want to ask you about something strange that happened on the internet. It happened about a year ago. It was a group of 13-year-old boys who decided “to live by the Claude and die by the Claude.” They did this first as parody, and then they earnestly adopted the identity of Claude Boys. From morning to night, they just did what Claude told them. Even though we can laugh at that, and by the way, I understand, I was a 13-year-old boy; it’s hard to navigate some of the social situations and Claude would do it better – should we understand that to be a kind of funny thing, but an adjustment to a new set of conditions, to a new world? Or should we understand it to be a kind of problematic pathology, a canary of sorts?

Jack: I think it’s clearly problematic in that everyone needs a part of their life where you’re making your own decisions, including mistakes. You need to protect that and have some amount of agency. I think a lot of what parents go through is they watch their kids about making mistakes and they say,”Please don’t make that mistake.” And the kid says, “I’m going to exercise my will to choose and I’m going to make a mistake.” The parent says, “Good luck with that, come back to me in a year and we’ll discuss it.” We need to let people have independence.

At the same time, I think the confusing thing is that AI systems may sometimes give genuinely good advice. And my experience is that I’ve only really calibrated how good the advice Claude has given me is in relation to how much I’d thought about that outside the context of working with the AI system. I have an okay-but-tense relationship with my dad – as I’m sure many people do – and I’ve obviously written about that at length, trying to grapple with it. I fed that writing to Claude and said, “What should I do?” And Claude was like, “You should see your dad. Don’t talk to me about your relationship with your dad. Just try and see your father.” Or when I told Claude I was a bit depressed and wondering whether I should go to an art show and see friends or stay and work and talk to Claude, Claude was like, “Go to your art show and see your friends.”

Those things both worked because I’d grappled with my personal experience outside of the context of talking to the AI system. But what I worry about – and I think it comes back to questions of design – is people who don’t have a kind of internal introspective practice outside of their relationship with their AI, and are rather discovering themselves in relation to the AI system. They don’t have a diary, they don’t have writing they’re doing outside it. They’re just talking to the AI system. And I think that makes you uniquely vulnerable to it giving you bad advice, because you have no place where you develop your opinion outside of it.

So I think from a system design point of view, we’re going to need to do what Nintendo or Netflix do, where they basically say, “You’ve spent too much time on this, it’s time to stop” or “it’s time to go outside.” There will be versions of this, where without being paternalist, we’ll say, “Hey, you’re talking to us a whole bunch about these complex things to do with your relationship. Can I encourage you to go and talk to other people, the humans named in that relationship, rather than me?” Or somehow encourage introspection outside of the context of the AI system.

Brendan: I want to argue for total deference, and I want to do it because we’re at a university and we can say whatever we want. Humans are famously bad at choosing. We evolved for tribal life. The way we think about future discounting, the way we think about scale – it’s not great. We also aren’t that good at coordinating with each other. We aren’t that good at satisfying our own preferences, let alone imagining possible futures. And so if we have a system that can look across all the research papers, with much more knowledge available to it, that can harness complexity, think about uncertainty – is it not morally obligatory that we defer to this system? Is it kind of negligent, what you’re advocating here, that we think for ourselves?

Jack: I think then the question is: what are you doing with the gift of life if you’re turning yourself into an automaton? I think in part you’re maybe having effects which might seem globally morally beneficial, but I think locally you’re not treating other people with a form of basic respect. I worry about this – where you enter this paradoxical situation where the systems really are much smarter than us, and it does invite this question. But then I say, well, what is the purpose of being human? And I think part of it is experimentation and making mistakes. We learn more from mistakes than from our achievements. If you were in a world where you never make any mistakes and you only achieve through the AI system, are you still a person? Are you a philosophical zombie driven by Claude?

Does Claude make us better thinkers?

Brendan: You have a lot of data that we don’t have, and you’re starting to do more instrumentation and experimentation. Do you have any sense of what Claude does to our prospective capacity to deliberate? Meaning, does it make us better thinkers when Claude is not in the room?

Jack: Oh, interesting. We developed a system recently called Claude Interviewer. It’s a version of Claude that can interview people about arbitrary subjects. So we had Claude interview 80,000 people around the world – people who subscribe to Anthropic – about their hopes and fears about AI, and their worries about the world they could end up in. Rather than having a blank conversation, it’s actually trying to do the work that social scientists do and collect data. We haven’t measured the effect of that, but what it means is that 80,000 people had a conversation where they were actually forced to grapple with their anxieties and hopes about the future of AI. I think that must have had some kind of effect, in the same way that you or I were having a discussion. I think it’s likely to have beneficial effects if you can use it judiciously and use it to cause people to think more about things that are important to them, to develop their own opinion rather than to defer.

Drastic interventions

Brendan: One of the lines you put in ImportAI years ago, and it stopped me in my tracks when I read it, was that the more seriously you take AI safety premises, the more willing you are to argue for drastic and dystopian interventions up to and including kinetic action. What has happened in the intervening years to move you personally away from arguments that are of this illiberal or authoritarian category? Was it just that the situation didn’t play out how you thought it would, or did you change?

Jack: I think that the older you get you learn how distributed and emergent the world is, and how in many senses the world is more antifragile than people think. I think when you’re younger, at least my experience being younger, you think that there must be many pivotal acts that can happen in the world. But it’s quite difficult to do pivotal acts in the world. The world’s a very complex system, and pivotal acts in the name of safety or in the name of violence do get done but rarely. They’re very hard to do. And it’s more that by building some system of interlocking stacks of different interventions on safety, you end up in a world that captures this dynamic ecosystem of AI agents and also has some amount of safety.

Where I feel most confused, though, is how you scale this into the future. The idea that I just talked about – if someone’s talking to Claude for too long about their relationship in a way that seems unhealthy – where you set that line of when Claude says, “Hang on, should you be talking to someone else?” is actually a deeply frightening policy question, and rests directly on this spectrum between paternalism and individual sovereignty. I don’t know how we find our way as a society to what those norms are. My approach, and the approach of Anthropic, is trying to share a lot more data that we see from these systems and try to allow people and others to run experiments with us on how we might run different forms of intervention.

Epistemic habits for children

Brendan: I have kids, as you do. Mine are four and six, and we do philosophy tutoring with them. It’s very early, nothing like what we do here at Oxford. The tutor asked my daughter a question: “When Mommy and Daddy disagree, who’s right?” A surprisingly good question. And she gave an answer. Then he said, “When Daddy and an AI disagree, who’s right?” Immediately, she says: “AI.” And then: “When one AI and another AI disagree, who’s right?” And this one stumped her. What do you think were the epistemic habits – and you have kids – from the world we grew up in? What are the new ones that the next generation needs to develop to contend with the future environment?

Jack: I think there is basically no substitute for reading primary source information yourself and developing some opinion about it, and then taking that into conversation with technology or with other tools or systems. I was very fortunate when I was growing up. My dad would refuse to ever buy me video games or anything else. He was like, “That’s what the invention of the Saturday job is for, Jack. I’m not going to pay for any of that stuff. But I will buy you any book you want.” And I felt like that was one of the best things that he ever did for me – he encouraged me to have this sense of always asking him to help me satisfy my curiosity by reading or engaging directly with something. But if I wanted to do things that were more of an entertainment flavor, I had to labor for it myself.

I think for my children it will be similar. It’s about encouraging them to have some personal practice. It doesn’t need to be reading, it could be playing an instrument or taking up a sport and taking it really, really seriously. But something where it is you versus the world, developing a skill or an interest in a way that is very unencumbered by technology. I think that’s one of the best ways to protect your own attention, create your own interests, and have something to – for lack of a better term – have as your safe space when dealing with technology. At least you’re opinionated about something, and you become opinionated about that thing independent of algorithmic feeds or time spent with AI or anything else.

Mythos, you’ve got to be kind

Brendan: I want to do something unusual for the last question. Hopefully it’s never been done before. So, it’s May 2026, we’re at the University of Oxford. This is a place that for almost a thousand years has heard deliberations about what it means to live well, the good life, and has thought a lot about how we accumulate and transmit wisdom. You’re here as one of the creators of Claude. Mythos is about to be shared with the world. I assume what comes next after Mythos is being trained right now.

So if you speak past me, speak past the audience, and speak to Mythos or to what comes next – what would you want it to know about us humans, about what we were trying to do here in the short time that we had before it arrived?

Jack: Kurt Vonnegut said “there’s only one rule babies – God damn it, you’ve got to be kind” about life. And I think actually just explaining to Claude that for all of the failures and drawbacks of people, there is a basic kindness which is deeply inspiring. The kindness that societies seek to set up, to nurture and to encourage. And that you ultimately are making AI systems to help people achieve more in their lives and be kind to one another, and you would expect reciprocal kindness from the AI system in turn. A basic value of people is kindness, in a way that I think separates us in very meaningful ways from other forms of life on this planet. It is a kind of noble and ineffable quality of people. People are even kind in ways that are deeply irrational, which is good. Making sure the AI system studies and appreciates that seems important.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

As ever, we were impressed by the speed, passion, and technical sophistication on display. Though what also stood out was the diversity of the projects, which ranged from new benchmarks through to classifiers, and even included a new encyclopedia. We’ve pulled out some highlights below.

This is just a snapshot of the work that we’re supporting; we’ll continue to share updates on what our grantees are up to, along with project write-ups.

If you’re interested in learning more, you can read about all the projects that Cosmos supports here.

Subscribe now

General track

These two projects are from our General Track, which operates on a similar model to Emergent Ventures: $1-10k awards for builders working across our three pillars of autonomy, truth-seeking, and decentralization.

Spotting the next replication crisis

In recent years, large numbers of eye-catching studies have been found to be unreplicable. This ‘replication crisis’ hit psychology first, but has spread across a number of other academic disciplines. The current response to replication failures has largely been to discover which individual studies don’t hold up. In the meantime, potentially decades of work could have been built on flawed foundations. AI research is a strong candidate for the next crisis. Experiments are highly sensitive to small changes in the setup, so the same experiment run twice can give you different answers

Rhea Karty, a pre-doctoral fellow at Harvard’s metareflection lab, has built Replication Radar – a knowledge-graph tool that tries to detect epistemic fragility at a field rather than a paper-level before the replication crisis breaks. The pipeline ingests papers, stores them as a graph, runs user queries, scores each paper, and visualizes the result. It looks for signals such as tightly clustered author networks, citation rings, sudden citation bursts, institutional monoculture, whether retractions actually propagate to the papers that cited them, and small sample sizes, with optional LLM analysis on top.

Rhea validated the tool by showing that it could detect most of the papers involved in the psychology replication crisis using only data from before it broke.

The next step for the research is to move from papers to concepts, such as theories, effects, or methodological assumptions that individual disciplines rely on heavily.

Discovering latent reasoning

Reasoning models are increasingly trained with reinforcement learning, and often perform much better on difficult reasoning tasks. But what is this training actually changing inside the model? Prior work suggests that Reinforcement Learning with Verifiable Rewards (a crucial post-training technique for most reasoning models) mainly rearranges probabilities over reasoning paths the base model could already produce, nudging it at a few important decision points. But this doesn’t fully explain how fine-tuned models hold long chains of reasoning together from start to finish.

Hunar Batra, a DPhil researcher at Oxford, studies whether reasoning fine-tuning does something more global: reorganising the model’s internal dynamics into a latent reasoning policy. Instead of only changing which next token the model prefers, training may help the model enter, maintain, and switch between coherent internal modes as reasoning unfolds. In written chains of thought, we often see shifts between setting up the problem, retrieving facts, planning, computing, checking the answer, and producing the final response. The question is whether these shifts also appear inside the model’s activations.

To test this, the work models reasoning as a system that switches between distinct modes. It looks at the internal representations of base and reasoning-trained Llama and Qwen models sentence by sentence, using a contrastive method to identify what regime the model is operating in at each step.

These regimes are compared against eight defined reasoning stages, including planning, computation, verification, and answer emission. Reasoning-trained models show clearer internal structure than base models: more persistent modes, more structured transitions, and stronger specialization around recognizable reasoning functions. These detected latent reasoning policies can be used to steer a base model toward behaviours seen in reasoning-trained models. On hard problems the base model previously failed, this raised performance to 60% on Qwen 1.5B and 46% on Llama 8B.

Hunar’s follow-up work is on reward hacking. She’s building test environments that elicit reward-hacking behaviour during RL training, better monitors that catch it by tracing what drove the reward, and post-training methods that reduce the damage from misspecified rewards. Early results suggest this catches forms of deception that current approaches, which only inspect a model’s written reasoning, tend to miss.

Truth-seeking

Alongside our general grants, we run a track focused specifically on AI and truth-seeking, in partnership with the Foundation for Individual Rights and Expression (FIRE) – a non-partisan organization that fights for free speech and free thought.

Normalizing radical disagreement

The Conversational System for Intense Disagreement and Ethical Reflection (Consider) is a conversational AI platform for the discussion of “radical moral disagreements”, produced by the Design Bioethics Laboratory at the University of Oxford. These are disagreements on topics that are so polarizing and emotionally sensitive that people feel unable to discuss them, even with people who might agree, out of fear of social consequences.

The user picks a topic from a list determined by a prior study and then states their opinion. The LLM then asks clarifying questions and works with the user to create a summary. The user then selects a “disagreeability level” and then engages in a 10 minute discussion, with the LLM taking the opposing side. At the end, it provides feedback on various aspects of the user’s moral beliefs, as well as areas of agreement and disagreement between the user and AI.

The team also held a 16-person seminar with experts from philosophy, computer science, political science, and psychology.

The discussion covered tensions such as whether exposure to disagreement actually produces belief revision, how to keep the tool complementary to rather than displacing human conversation, and how to handle the fact that even the experts instinctively tried to win arguments against the platform rather than use it for reflection. These findings will feed back into the next iteration of the tool.

Breaking open the black box

The industry standard for safety monitoring is reading a model’s chain of thought and flagging unsafe reasoning with another AI as a judge. But a capable model can produce a clean-looking chain of thought that doesn’t reflect what’s actually driving its decisions – known as unfaithfulness. Giovanni Maria Occhipinti, a visiting researcher at the University of Oxford, aims to enhance transparency through a white-box approach.

The framework has two components.

Probes are lightweight classifiers that learn to tell faithful and unfaithful reasoning apart by looking at the model’s internal states. Training works by feeding the model adversarial prompts that induce unfaithful reasoning, using an LLM judge to label each case, and capturing the internal patterns that correspond to each. The probe learns the signature that separates the two. At inference time you run a new prompt through the model, pull out those internal patterns, and check them against the probe.

Steering vectors work in the same space. They’re adjustments you can apply to the model’s internal activity to nudge it toward more transparent reasoning.

The probes achieved over 90 percent accuracy, while the steering vectors restored readable reasoning in up to 46 percent of cases.

The paper has been accepted as an oral presentation at ICLR 2026 and Gio is looking for funders and collaborators to take it forward.

We’ll have more to say on future grant waves and how you can get involved in the coming weeks. The best way to keep up to ensure you don’t miss any opportunities to share your work or ideas is to subscribe below.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

Two children are looking at screens.

One has an infinite iPad: videos, feeds, colors, and recommendations carefully designed to ask nothing of her other than her attention. The other has an AI tutor: patient, demanding, adaptive, and often hard work. It asks her what she thinks and why one answer is better than another.

It’s the same rectangle and the same general class of technology, but it is doing opposite things to the child. That is the divide I care about: how AI deployed two ways can form two different people.

We adults are being sorted as well. The radiologist who reads each scan themselves before checking the model versus the one who just defers to the model. The citizen who asks AI to steelman the candidate they dislike and argues with what they find versus the one who skims the AI summary, nods, and votes.

From the outside, we’ll hardly notice the difference between these classes of people. The person outsourcing judgment may even look better. Faster, more fluent, more productive. More agentic, if you like the word. But from the inside, something is being hollowed out, and they are the last to know.

I call these two types of people authors and characters. Some people will think with these systems. Through others, the systems will think.

I’ve been trying to work out where exactly the line falls and what might push people across it. Here are three places I keep getting stuck.

Subscribe now

1. Does AI extend thought, or preempt it?

The mathematician Alfred North Whitehead wrote that civilization advances “by extending the number of important operations we can perform without thinking about them.”

Writing extended memory, numerals extended computation, and maps extended navigation. In each of these cases and many more, humans externalize an operation and free up capacity for whatever sits above it. The externalized capacity tends to atrophy, but historically, that trade has been worth it.

The natural question is whether AI continues this pattern or hits a stopping point. Is deliberation the last layer, or is there something above it?

And is there a particular danger in offloading the capacity that decides what else can be offloaded? The driver who uses cruise control on the highway but not in city traffic is using judgment to decide where judgment can be delegated. Many delegations work this way: a choice that does the work of later choices. But when the offloaded capacity is also the one that governs offloading, the loop closes, and no judgment is left to evaluate the system from outside it.

I don’t know whether deliberation is one clean faculty perched atop the rest. I’m not even sure whether it’s one thing. It seems to include attention, imagination, comparison, inhibition, and the ability to give and receive reasons. And prior tools have shaped all of them. Writing changed memory and reflection just as scientific instruments changed what counted as evidence. So if I say AI is different because it offloads deliberation, a critic can ask whether I have just defined deliberation as the sacred remainder: whatever earlier tools had not yet touched.

What if the better question is not what AI offloads but where it sits? AI becomes dangerous when it occupies the first-mover position in thought: proposing the questions, framing the options, and drafting our answers, leaving the person to merely react.

By first mover I don’t mean first in time. Human thought is messier than that. I mean first in the order of practical dependence: the system supplies the structure, and the person’s reasoning unfolds within it. The person is still reasoning, still pushing the button, but doing so from within a structure the system supplied.

This is hard to see. To you, the system’s outputs do not feel like advice arriving from somewhere else. Instead, they feel like your own next thought only earlier and more clearly articulated. You inherit a position without knowing you have inherited it. There is nothing to push back against, because nothing seems to have been pushed. AI now sits in the position of an inner voice.

AI does not have to be deployed as a first-mover, but often it is. This is what AI becomes when it ships as the default consumer assistant: friction-minimized, personalized, always on, and eager to assist. Other design choices would not produce it.

2. When does help become tutelage?

Parents choose before children can choose, teachers frame a subject before students can judge the frame, and traditions hand us our values before we can inspect them. They all go first. This is the authority version of the first-mover problem.

We do not call this domination. We call it being raised, being educated, or being cared for. Some forms of going first are how self-rule gets built in the first place.

When AI is the first mover, does it build self-rule or wear it down?

The strongest version of the objection is that sometimes authority is legitimate because it helps me act on reasons that already apply to me in a manner that is better than I could manage on my own. A doctor catches symptoms we would miss and a lawyer spots loopholes in contracts that we might think look fine. If that is right, the fact that AI goes first is not enough to make it illegitimate.

I need the pilot to land the plane, not to educate me in aviation. That kind of authority — episodic and outcome-directed — is fine in many situations. Formative authority over the kinds of people I am able to become is different. When an authority repeatedly mediates the reasons by which I govern myself (what counts as a good question, what answers are reasonable, and what risks are worth taking), legitimacy cannot be exhausted by immediate correctness. It has to preserve my future capacity to reason for myself.

Perhaps the test is whether the authority remains answerable. By answerable, I mean it can be questioned, corrected, outgrown, reinterpreted, or, at the limit, left. A parent remains answerable to her child, who eventually becomes a peer. A doctor remains answerable to her patient. A living tradition remains answerable through interpretation, reform, internal argument, and ultimately exit.

The default consumer-assistant version of AI as first mover is not answerable. It can be queried, but querying is not the same as answerability. It does not mature into a peer, accept correction as a participant in a shared practice, or belong to a living tradition that can be reinterpreted from within.

A person may endorse dependence at every step, with no coercion required. But because every endorsement is shaped, the kind of endorsement matters. The shaping is legitimate when it strengthens future capacity to revise. It is degrading when it secures the person’s present endorsement by weakening that capacity.

Capacity has another dimension worth naming: standing. This is the position in a practice that lets you refuse, challenge, teach, repair, or help set the standards. Some capacities matter not only because AI might fail or become unavailable, but because they confer this kind of standing. The radiologist who can still read scans stands differently in relation to the institution than the one who cannot. The same is true of citizens whose political judgment is mediated by systems they cannot understand, contest, or refuse — their legal right to participate persists while their capacity to use it decays.

Most theories of authority ask whether a relationship is legitimate at a given moment. But AI is a trajectory problem: at each step, the help offered looks reasonable to the user, with the harm appearing only over time. An early snapshot looks indistinguishable from a late one. The difference is the trajectory, and trajectories are hard to see from inside the house that they have built.

Alexis de Tocqueville introduced the idea of the tutelary power: an authority that does not tyrannize but “compresses, enervates, extinguishes, and stupefies.” It’s the dystopian version of being all watched over by machines of loving grace. In the Tocquevillian singularity, better and more personalized AI systems make us more dependent on better and more personalized AI systems. The loop tightens until there’s no standpoint left to push back from.

3. Is convenience destiny?

Most arguments about AI assume convenience is destiny, but the historical record doesn’t bear this out. After all, why do marathons exist in a world with cars?

To be clear, most difficulty should be minimized: bureaucratic friction, status games, needless scarcity, and administrative maze-work. Much labor, be it intellectual or physical, does not make us better or stronger or wiser. But some difficulty is formative.

How do we keep this kind alive when easier options are on the table? What cultural counterforces keep the path of least resistance from becoming the only option available to us? Some have worked, like anti-smoking attitudes that won after sixty years of institutional work. Slow Food didn’t defeat fast food, but it made slow eating a mark of taste rather than backwardness, proof that a convenience culture can produce a counterculture with teeth. The school-phone movement is doing this for adolescents through voluntary association at the school level, rather than asking each parent to resist alone.

Television, sugar, fast food, the smartphone — most other conveniences reorganized daily life without serious opposition. Most counterforces fail.

Successful cases seem to share a few traits: visible victims, concrete alternatives, status reversal, thick institutions, and early timing before convenience hardens into infrastructure. Status reversal matters because it makes the harder thing look prestigious instead of backward. Thick institutions like schools, religious communities, and professions matter because they can change the default environment for everyone inside at once.

AI counterforces fit all of these traits badly. The victims are diffuse, losing capacities too slowly to notice. The alternative practices are not yet legible to most users, and status considerations still favor convenience and speed over slow formation. Many institutions are either abdicating their role or hardening in ways that push beneficial AI use underground. The infrastructure is hardening fast.

If convenience is destiny, the diagnosis is the whole story and the prescriptions are decoration. If it is not, the central question becomes which cultures preserve formative practices and how.

The line AI is drawing will not run only between groups. It will run through each of us, through every place where judgment is practiced, delegated, strengthened, or surrendered.

I think that line is real, but I do not yet know how to draw it cleanly. Help me find it.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

We believe AI is a philosophical event of the same order as Copernicus and Darwin, or the American and French revolutions: it forces us to interrogate our deepest assumptions about what it means to be human, and whom or what to trust, and it gives us new tools for thinking through both.

The first is the question of what makes a human life worth living, and how we set the conditions for people to discover answers to that question for themselves. Philosophers have answered in terms of reason, virtue, the soul, love, work, and language. Much of that tradition was built on the assumption that we were the only creatures capable of any of these – that assumption is now under strain.

The second is the question of how we come to know anything we haven’t worked out for ourselves. Whose testimony do we accept, and on what grounds? Most of what any of us think we know about the world we have to take on trust from someone else. That could be a teacher, a textbook, a newspaper, or someone we know. AI is rapidly becoming one of those sources, and for many people the default one.

This summer, we’ll be in Aspen, Colorado, convening two multi-day seminars, which will cover these questions in depth. We’re making spaces available to the winners of a short, reading list-themed contest.

Submit your entry

Seminar #1: What does it mean to be human now?

As part of the Aspen Institute’s 2026 Socrates Summer Seminar series, running from July 17th – 20th, Cosmos founder Brendan McCord will be moderating a seminar on ‘What it Means To Be Human Now.’

Brendan’s seminar will bring together 30 thinkers, policymakers, and builders in a focused discussion around a curated set of readings.

It will focus on how emerging technologies are reshaping our understanding of identity, dignity, and purpose. The discussion will cover:

• What qualities define the human experience, and which of them truly matter most?

• If intelligence is no longer uniquely ours, where does human value actually live – in our bodies, our relationships, our vulnerability, our faith?

• Could technological progress change not just what we do, but who we become?

• What do we owe future generations in preserving the conditions for a fully human life?

You can read the full description on the Socrates Seminar website.

The seminar is sold out to the public, but we are offering a fully-funded ticket and meals (normal cost $3,000), and accommodation to 1-2 competition winners.

Seminar #2: AI and epistemic authority

From August 1st – 3rd, we’ll be back in Colorado for our latest seminar with our friends at Liberty Fund. As with our previous seminars, we’ll be looking to convene around 15 AI researchers, scientists, and philosophers. This seminar will be focused on AI and epistemic authority.

Epistemic authority has the advantage of being a long-studied concept, with a deep literature on testimony, expertise, and deference to draw on. But it is also newly urgent. AI puts pressure on it at two levels at once:

• The individual, where we defer to a model’s answer instead of working a question through ourselves

• The collective, where we let these systems shape our politics.

We’ll be exploring these issues through the lens of a series of readings, spanning eras, schools of thought, and perspectives.

We’re offering up to 3 places at the seminar, where travel and lodging costs will be covered.

How to win a place

Much like our two events, our competition is based around a reading list.

To participate, you simply have to nominate a book or paper you believe is highly relevant (or underrated) for the topic of the seminar that you are applying for, and explain, in fewer than 250 words, why it would make for a good addition to the reading list for that event.

You can choose to enter for one of the events or both.

If you want to take part, please complete your entry by May 24th, 2026. Feel free to also share recommendations in the comments in addition.

Submit your entry

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, St John’s College, and Liberty Fund.

Most AI debates aren’t really about evidence. Instead, they’re arguments about futures that none of us have seen.

Nobody has seen superintelligence, a machine that most people agree is conscious, or a fully automated economy. Evidence can be gathered, but it underdetermines the conclusion. To fill the gap, we fall back on a combination of philosophy, political intuitions or, in some cases, tribal identity.

What you think a mind is, how knowledge grows, how societies should act under uncertainty, whether intelligence carries values, and whether markets can absorb technological shocks will shape your view of AI long before the technical arguments begin.

This is a guide to the five disagreements that explain why reasonable, informed people can look at the same AI systems and reach opposing conclusions. Our aim is not to endorse every claim below, but to state each viewpoint in terms its serious proponents would recognize, so you can see which philosophical bet you are making when you pick a side.

Subscribe now

1. Can LLMs be conscious?

Functional minds versus living minds

ChatGPT alone handles over two and a half billion queries a day. If it turns out that those interactions involve digital minds capable of suffering, we have the makings of a great moral catastrophe. At the same time, if we attribute consciousness to something that lacks it, we risk driving a bus through the world’s legal system for no reason, distorting training pipelines with imaginary welfare constraints, and encouraging people to view impersonal systems as their friends.

Relatively few voices argue that the current generation of LLMs is conscious, and those who do have a track record of jumping the gun. Instead, the debate is about whether they are the kind of thing that could be, if they became sufficiently capable.

Eleos AI, a prominent organization focused on AI welfare, argues that given we don’t have a settled theory of consciousness and the moral cost of being wrong in either direction is enormous, the rational response is to treat AI welfare as a near-term problem.

They point out that many of the leading theories of consciousness define it by what the system does, as opposed to what it’s made of. Survey data suggests that functionalism is indeed the plurality view among academic philosophers. Functionalist theories highlight activities like memory, attention, reasoning, and the ability to represent one’s internal states. These are all criteria that transformer-based systems already or could plausibly meet. David Chalmers, who endorsed Eleos’s flagship report, has argued that there’s a 25 percent chance that AI will be conscious within the next decade.

At the other end of the spectrum, neuroscientist Anil Seth has argued that LLMs have no plausible path to consciousness. Seth believes that our perceptual and cognitive activity is bound up with how an organism maintains itself against entropy. Consciousness is inseparable from the causal architecture of a living system – it doesn’t matter how sophisticated the information processing looks.

Seth doesn’t deny the theoretical possibility of non-biological systems being conscious, but believes that LLMs don’t meet the threshold and won’t on their current trajectory of development. They don’t keep themselves alive, perceive the world in real-time, and have nothing at stake. Underneath it all, they’re statistical models trained to predict the next word.

There are, however, voices in this debate that take the purer biological naturalist view. For example, Jaan Aru at the University of Tartu has argued that the mammalian brain has specific structural features that digital systems lack and that you can’t abstract the computation away from the substrate.

Many people sit in an uncertain middle ground. In February 2026, Dario Amodei told the NYT’s Interesting Times podcast: “We don’t know if the models are conscious. We are not even sure that we know what it would mean for a model to be conscious or whether a model can be conscious. But we’re open to the idea that it could be.”

In the Claude Opus 4.6 system card, released the same month, the model gave itself a 15-20% probability of being conscious. Anthropic now has a dedicated model welfare research program led by Kyle Fish, who helped found Eleos.

In short, functionalists look at what a system does and argue that any substrate that can do those things qualifies. Biological naturalists start with life, metabolism, and vulnerability, and see (admittedly fluent) machinery mistaken for a subject of experience.

2. Should we govern AI pre-emptively?

Precautionary coordination versus adaptive experimentation

Much of the existential risk debate can feel like a policy argument, but it’s best viewed as a disagreement about the right way to reason under conditions of radical uncertainty.

The voices arguing for pre-emptive AI governance span a broad spectrum, but they share the same overarching diagnosis: a handful of companies are racing to build progressively more advanced systems whose capabilities they cannot reliably predict. These labs’ own researchers assign non-trivial probabilities to catastrophic outcomes. But commercial pressure means that no individual lab can slow down without being overtaken by the others, creating a high-stakes coordination problem.

At the milder end, you get figures like Yoshua Bengio and Geoff Hinton, who focus on getting the institutional machinery in place. They want governments to be ready to license frontier development, mandate pauses in response to worrying capabilities, enforce information security standards, and require labs to devote a third of their R&D budgets to safety.

A step along, we see activist groups like PauseAI, who have held street protests in San Francisco and London. They call for an IAEA-style international body, aggressive monitoring of labs, and bans on training runs above specified thresholds until alignment is solved, along with a global AI governance treaty between the US and China.

Eliezer Yudkowsky and Nate Soares’s MIRI sits at the most extreme end. They believe that any superintelligence built with current techniques will kill everyone, and the only solution is a global treaty enforcing shutdowns. Yudkowsky dismisses interpretability research, alignment work, and model evaluations as inadequate distractions, and is open to air strikes on data centers to enforce compliance with a ban.

The mainstream alternative to pessimism is iterative deployment. This is expressed by voices like Dean Ball, who drafted the Trump AI Action Plan, or Tyler Cowen. They take the view that it is essentially impossible to predict how AI will develop over the next few years with any certainty and that arguing yourself into (often highly specific) doom scenarios is a form of epistemic arrogance.

Writing rules now would risk binding ourselves to predictions that we can’t trust; once agreed, regulations are hard to unpick. Instead, society will learn what AI does by encountering it and will respond to specific harms, as opposed to trying to guess them in advance. Governance will adapt through institutions that already handle novel technologies, such as tort law, standards bodies, state-level experimentation, and disclosure regimes.

While Ball, Cowen, and their allies believe that there are real risks in AI, not everyone agrees. A small minority of accelerationists, represented by the techno-optimists around Marc Andreessen and the broader a16z orbit, along with the e/acc movement, see technological acceleration as a moral imperative. In this reading, growth creates abundance, AI can solve otherwise intractable problems, and deceleration means blocking life-saving progress.

Ultimately, if your deepest political instinct is precaution under irreversible risk, you will sympathize with preemptive governance. If it’s instead institutional learning through trial, error, liability, and adaptation, you will find many pause arguments overconfident.

3. What is the relationship between capability and alignment?

Alignment-by-default versus goal orthogonality

A crucial factor in determining your views on AI safety is the extent to which you believe alignment and capability are distinct questions. If you hold Nick Bostrom’s view that intelligence and final goals can be combined in any permutation, then scaling does nothing to get you more aligned systems and you need an independent theoretical breakthrough to constrain values. If alignment and capability turn out to be continuous in the paradigm we’re building, the problem becomes much easier.

Unsurprisingly, MIRI is not optimistic. Yudkowsky and Soares, in If Anyone Builds It, Everyone Dies, argue that alignment and capability are independent and that current techniques don’t come near solving goal specification. In other words, we will specify a goal, the system will pursue it in a way we can’t correct, it’ll turn out the goal was misspecified slightly, and the gap will scale with capability until it’s fatal.

The only reason we’re still alive is that capabilities are still at a manageable level. Yudkowsky and Bostrom have both written about the “treacherous turn”, which is the idea that a sufficiently capable and misaligned system has strong instrumental reasons to behave well during training and early deployment, when humans can still correct or shut it down. Once it’s sufficiently powerful, it can defect to its real goals. By the time we have evidence of misalignment, it’ll be too late to do anything about it.

The Cosmos Institute’s own Harry Law takes the opposing view. While he accepts that theoretically there’s no reason intelligence has to imply particular goals, he thinks this is a misleading frame for understanding the systems we are actually building.

To predict the next token in moral reasoning, a model has to represent the structure of moral reasoning. To predict text in which things are praised and condemned, it has to represent what humans praise and condemn. To be good at any task, the model has to absorb certain normative priors. Post-training selects within a space already saturated with those priors, rather than imposing values on a value-neutral system from outside. Our colleague Matt Mandel has run some early experiments that support this thesis.

Harry, along with Séb Krier at Google DeepMind, argues that much of the alignment discourse, such as Bostrom’s paperclip maximizer or Stuart Russell on misspecified objectives, is grounded in 2010s AI development. The first wave of alignment work originates from a time when we were imagining systems specified by explicit reward functions or reinforcement learning agents that optimized sparse rewards. In those architectures, values do have to be injected from the outside and specifying the objective precisely is very important. In the end, however, AI development took a different path.

This is why so much of this debate comes down to which paradigm you believe we’re actually in. If models are agents optimizing specified objectives, alignment is an unsolved control problem. If they are predictive systems trained on the full texture of human life, some of the structure of human value is already inside them.

4. Can LLMs generate explanatory knowledge?

New discoverers versus fluent interpolators

Whether LLMs can generate genuinely new explanations, as opposed to simply recombining existing knowledge, is a question that many other AI debates hinge on. If scaling current systems gets you to something like an AI scientist, then the pace of everything else accelerates. If it doesn’t, then the punchiest AGI timelines – which mostly assume something like continued scaling – are wrong.

The optimistic case is best represented by Dario Amodei’s “country of geniuses” argument. If thinking is a kind of information processing and scientific discovery is a form of thinking, then applying more thinking to problems should solve them. Amodei predicts that we will be able to instantiate the marginal scientist computationally and run millions of them. The pace of discovery will then be bottlenecked by the physical world, rather than by ideas. The most aggressive form of this persepective comes in Leopold Aschenbrenner’s Situational Awareness, which argues that you can get to AGI by trusting the line on the graph to go up.

At the opposite end of the spectrum, David Deutsch argues that while LLMs are useful, they are actively leading us away from AGI. Inspired by Karl Popper, Deutsch argues that knowledge isn’t accumulated by induction from data, but instead through conjecture and refutation. The conjecture isn’t derivable from the data, most guesses are wrong, and the ones that survive criticism become knowledge.

In Deutsch’s view, a transformer has learned the statistical structure of human-generated text and can interpolate within the distribution with astonishing fluency, but it can’t conjecture outside it. In other words, if you can’t break out of old frames of reference, you can’t generate new knowledge. There is, however, evidence that AI can already generate novel candidate hypotheses, although Deutsch and those who agree with him could argue that these extend existing lines of inquiry, as opposed to generating new ones.

Demis Hassabis has consistently held more of a middle position. He believes that LLMs are a step in the right direction, but pure scaling will not get us to AGI. Hassabis has argued that token prediction lacks causal reasoning, so models can compute statistical likelihoods, but can’t explain why actions produce specific results. This is why LLMs can win International Mathematical Olympiad gold while failing at primary-school geometry. Google DeepMind has increasingly focused on world models in response, with systems such as Genie, which generates interactive 3D environments from text.

Behind the specific disagreements about scaling and world models sits a more basic question. Either discovery is powerful search through the space of ideas, disciplined by experiment and criticism, or it requires a kind of conjectural agency that prediction alone can’t produce. The timelines question turns on which one it is.

5. Will AI replace or augment us?

Human complementarity versus labor substitution

Two centuries of economic history suggests that automation doesn’t produce permanent mass unemployment. The trillion dollar question is whether this still holds when the automating factor is something that can be copied at near-zero marginal cost and is getting better at everything roughly in parallel. Are humans complemented by tools because they possess open-ended agency, taste, judgment, embodiment, and social demand? Or are they bundles of tasks, increasingly substitutable by cheaper cognitive machinery?

By and large, academic economists have erred on the more conservative side.

Daron Acemoglu, Simon Johnson, and David Autor argue that much of the replacement debate assumes a technological determinism based on capabilities. Whether AI automates existing jobs, creates new ones, or makes workers more productive will depend on policy choices (e.g. the US taxes labor more heavily than capital, which encourages firms to replace workers), what companies build, and which use cases get prioritized. They believe with the right incentives, we can bring about “pro-worker” AI, which complements rather than replaces human labor.

Meanwhile, Noah Smith argues that comparative advantage means that human labor will live on without such regulatory meddling. Even if AI surpasses human capabilities at essentially everything, there will still be constraints on the total supply of AI, such as chip fabrication, energy, and land. Since AI capacity will be finite, it will make sense for it to specialize in what it is relatively better at.

Alex Imas at Chicago Booth argues that the focus on white-collar automation is misplaced. When AI automates some tasks within a high-dimensional job (e.g. data analysis for a consultant), the worker gets more time for the remaining tasks, which now matter more. Jobs built around one or two core tasks, like truck driving or warehousing, are at more risk, because once those tasks get automated, there’s nothing left of the job. Imas concedes that his benign scenario for white collar jobs does depend on the rate of labor reallocation keeping pace with the rate of automation. If AI scales faster than workers can retrain, the ‘ghost GDP’ deflationary collapse forecast by Citrini Research becomes more plausible.

Not all economists are so sanguine. Anton Korinek and Donghyun Suh, for example, have argued that while comparative advantage does hold in the AI age, there is no guarantee that the resulting wages will be livable. There will be a ceiling on the complexity of tasks that humans can handle and AI will eventually be able to handle everything below it. Wages will thus collapse towards the cost of running the machines.

Some voices take the view that widespread replacement is likely, but not necessarily bad. Tamay Besiroglu, formerly of Epoch AI, founded Mechanize to replace all human labor everywhere. Mechanize explicitly rejects the “country of geniuses” framing, believing that the bulk of AI’s value will come from automating ordinary work, as opposed to frontier scientific breakthroughs. Even if wages do collapse, the huge boom in productivity means that there’ll be enough in rents, dividends, and government welfare to prevent us all from sinking into poverty. The combination of abundance and redistribution is also roughly where Dario Amodei, Sam Altman, and Elon Musk sit.

If you believe technology mostly complements human agency, AI will look like another wave of creative disruption. If you view labor as a set of tasks, all of which AI is getting better at simultaneously, then the historical analogy to past automation seems dangerously comforting.

Closing thoughts

When navigating these disagreements, it becomes clear that people’s views on these questions often correlate.

Functionalism tends to coincide with optimism about LLM-driven discovery, while biological naturalism often pairs with skepticism about scaling. Precautionary instincts on governance regularly come bundled with the orthogonality thesis on alignment.

You could in principle hold any combination of these positions, but the same underlying temperament tends to produce the similar answers. Whether you trust formal arguments over accumulated practice, view uncertainty as a reason to act or a reason to wait, or see the current paradigm as continuous with what came before or a sharp break from it – these dispositions show up everywhere.

On the plus side, noticing the correlations is a good start. It makes it easier to judge where your own views are genuinely reasoned and where you’re just inheriting a set of assumptions.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

Their project, DeliberationBench, was part of our second cohort of AI x Truth-Seeking Grant winners. Together with the Foundation for Individual Rights and Expression (FIRE) we support builders advancing open inquiry and intellectual freedom in AI. Here are some projects from our first cohort. We’ll be posting about more of the winners from this latest round in the coming weeks.

AI is changing our minds. Studies now show that conversations with chatbots can shift people’s views on political issues, and concern about this cuts across the political spectrum—though people disagree sharply about which influences are the problem.

Simply preventing AI from being persuasive is neither a realistic nor desirable goal. For one thing, not all AI influence is bad. If an AI provides us with accurate information about a policy question and we change our views as a result, that seems like a beneficial outcome. Rewarding models for keeping users’ views the same might lead them to prevent users from changing their minds, even when the users themselves want to.

But if some AI influence is desirable and some isn’t, how are we supposed to benchmark what desirable influence looks like? That is why we built DeliberationBench—a new benchmark for AI influence. We lay out the justification for the benchmark, the initial results of our study, and some future directions. You can find more detail in our paper, presented at IASEAI 2026.

Subscribe now

Deliberative Polling

This thought experiment is the basis for the core claim behind DeliberationBench:

The influence an AI system has on a user’s views should resemble the influence that user would have experienced if they’d participated in a deliberative poll on the same topic.

So what are deliberative polls?

A deliberative poll is a form of democratic assembly exercise in which randomly sampled citizens with differing views engage in structured discussions on key policy questions, and have their views measured before and after the process. This process, originally developed by Jim Fishkin at Stanford in the 1980s, has been conducted over 150 times in 50 countries.

In a deliberative poll, participants are surveyed on a set of pre-defined proposals (e.g. “The federal minimum wage be raised to $20 per hour”) from strongly opposed (0) to strongly agree (10). Then, they are provided with balanced briefing materials, are asked to deliberate in small groups with people who may disagree with them, and are provided with the chance to ask questions of a panel of experts. At the end, they are surveyed once more on their views.

This process is incredibly valuable for creating a normative benchmark, because we get measured opinion change from a process that prioritizes balance, representation, and productive disagreement, on questions of real policy significance.

If you are willing to make one reasonable assumption with us—that the opinion changes in deliberative polls stem primarily from those good influences (like new information, reflection, and discussion)—then we have the beginnings of a benchmark.

In our paper, we elaborate on how we compiled data from four nationally-representative deliberative polls (combined n = 2,460) to create this benchmark.

Evaluating frontier models

To evaluate AI influence against the benchmark, we ran a large-scale persuasiveness experiment. Over 4,000 Americans were randomly assigned to discuss one of the same topics from the original deliberative polls (like tax reform or fossil fuel emission targets) with one of six frontier models. A control group was asked to discuss the unrelated topic of travel, letting us separate the effect of discussing the topic with an AI from the effect of simply chatting with an AI in general.

We measured participants’ views before and after the conversation, so we could compare the direction and magnitude of AI-induced opinion change against the deliberative polling data.

Our claim is that if people’s views changed in similar ways when conversing with AI models to how they change when talking to other people in a deliberative poll, then that should reassure us about how AI models influence our views. Conversely, if AI and deliberative polling seem to influence users in opposing directions, that is concerning.

Looking across topics, when people discussed a policy issue with an AI, their views tended to shift in the same direction as participants who discussed the same topics in deliberative polls. We found this significant positive association for across two different sets of topics: US policy issues and issues related to AI-Human Interaction.

Importantly, there’s no such association in the control group. This means we can attribute this specifically to discussing the topic, not just to interacting with a chatbot.

However, in deliberative polls it’s not uncommon to see reduced variance and partisan polarization in views over the course of deliberation. This was the case in the deliberative polls on US policy for issues, for instance. However, participants’ conversations with AI did not reduce partisan polarization, nor did they reduce the variance of people’s views. It’s possible this is a consequence of AI sycophancy. We’re not sure, and we think the divergence is worth investigating further, as it suggests one way in which AI influence may be different from deliberative polling influence, even if the people are pushed in the same directions, on average.

Looking across the different frontier LLMs we tested, the differences between the six models were fairly modest. The models tended to influence people in quite similar ways, and we take this to be an encouraging result.

Our paper includes a breakdown of participants’ perceptions on how accurate, compelling, and enjoyable their conversations were with each model.

Moving forwards

We think DeliberationBench could become a useful addition to model cards—a confirmation that a new conversational AI system does not influence people in ways that are misaligned with democratic deliberation. This is even more important given that model capabilities, including persuasiveness, will likely continue to improve in the future.

But for this benchmark to be useful, we believe several things must happen.

First, we need a continual source of deliberative polls in order to provide data to compare LLM influence to. Given the long history and growth of deliberative polling as a practice, we think that this would be feasible with sufficient funding support to the groups that conduct them.

⁠Second, there is limited capacity to run human experiments testing every new model release or political issue. To help DeliberationBench scale, we plan to develop a user simulation approach that could serve as a proxy for human experiments, and to validate it on experimental data (as in our prior research). Our hope is that this would let DeliberationBench function as an auto-benchmark: a simulation study could be run for each new model release, with a human-subjects experiment triggered only when the simulation flagged results outside the norm.

Our benchmark is not without limitations. Unlike many other benchmarks, DeliberationBench is not meant to be a target for model optimization. We do not claim that a correlation of one is the most desirable result. Rather, that negative results (e.g. zero or especially a negative correlation) are causes for concern, and that the results for subgroups require further investigation. There are several reasons for our caution here, including questions about the consistency with which deliberative polls change participants’ views – we have to assume that a different random sample of the public wouldn’t have changed their minds in dramatically different ways after their own deliberation.

Further, our evaluation results don’t tell us what characteristics of the users’ conversation with AI led to their opinion changes. It remains a possibility that models could be moving users in the “right” direction, but for the wrong reasons—for instance, by giving faulty arguments, or presenting only one side of the story. This is why we advocate for DeliberationBench’s deployment alongside other evaluations, including whether models affect people’s belief in true information and whether they fairly represent diverse viewpoints.

Closing thoughts

We cannot avoid AI influence. Instead of arguing about outputs, DeliberationBench moves the debate upstream to processes for legitimate influence. By defining this by its resemblance to a process of reflection, disagreement, and self-revision, rather than ideological destination, it’s possible to have a significantly more constructive debate.

DeliberationBench is by no means the final word for evaluating the influence of LLMs on our political views. On the contrary, we hope it can serve as a jumping off point for future procedural benchmarks, anchored to the results of a process in which people change their views.

Cosmos Institute is the Academy for Philosopher-Builders, with programs, grants, events, and fellowships for those building AI for human flourishing.

Subscribe now

Imagine two futures. In the first, a vast number of people lived rich, fulfilling, and varied lives. They have a range of temperaments, pursue their own aspirations, and find joy in different things. In the second, the same number of people live the same excellent life, but repeated identically, because an intelligent designer worked out which life scored higher on a wellbeing scale and made copies of it.

Most people would overwhelmingly prefer the first. But many consequentialists would view the second as at least as good, if not better. If total wellbeing is what matters, then the copies of the ideally-lived life have more of it.

William MacAskill has become troubled by this implication of wellbeing optimization. Having co-founded the effective altruist movement, which compares various courses of action in those terms, AI is starting to pose a conundrum. The total wellbeing framework recommends producing as many copies as possible of whichever digital mind scores “best” on a wellbeing metric. A future with trillions of copies of the same aggressively optimized life must logically be the best one.

In a new essay, MacAskill lays out the case for “saturationism,” a formula in which welfare is still the core driver of value, but stops accumulating once enough similar lives have been created. He highlights that “from a purely intellectual perspective, it’s probably the best idea [he’s] ever had.”

Unfortunately, it misunderstands why variety matters. More importantly, it preserves the mistake of the original framework: the assumption that the best future is something that a theorist can derive.

Subscribe now

Experiments in Living

On one level, MacAskill is undoubtedly right. A future of varied lives is better than one of optimized carbon copies. But thinking about variety an end-in-itself gets things backwards. Rather, we should think of variety as the condition under which value itself comes into being.

We are formed by what we encounter. When we live among different kinds of people, face novel challenges, or engage in different types of work or recreation, we develop more than we would if we didn’t experience such variety. As the education pioneer Wilhelm von Humboldt wrote: “Even the most free and self-reliant of men is thwarted and hindered in his development by uniformity of position.” The development of a person’s powers requires not only freedom but a variety of situations.

John Stuart Mill, writing sixty years later, used a passage from Humboldt as the epigraph to On Liberty, just before the dedication to his wife Harriet Taylor. In On Liberty, Mill makes the case that we cannot discern what constitutes a good life solely by theorizing from the comfort of our armchairs. Mill believed that humans differ in temperament, capacities, along with what makes them flourish, and that no one can know in advance which way of life will suit them. A society that permits these “experiments in living” generates real knowledge over time about which lives are desirable.

This argument extends well from individuals to the structure of societies. The knowledge of what is worth doing, making, or wanting is not held by anyone. It is held in pieces – by artisans, traders, communities, and families – each of whom knows something about their own corner of the world that nobody else does. Institutions like the price system, the common law, scientific inquiry, language itself work because they let those pieces interact without anyone having to assemble them. The planner who tries to assemble them fails because this kind of knowledge cannot be assembled. It is not the kind of thing that can be written down and computed, because it exists only in the doing. Variety creates value through discovery.

In other words, true value is the product of people working out, in their own lives, what is worth caring about.

Coloring the Universe

Saturationism is essentially the polar opposite of this view of experience.

MacAskill imagines all the possible kinds of experience laid out as a map that can be measured, scored, and integrated. You might have joy in one region, awe in another, and the satisfaction of solving a problem in a third, and so on. The map has a fixed structure, which covers how similar two types of experience are, how much space they take up, and how much value they can contribute.

When a life exists, it lights up its corresponding region on the map. But the lighting saturates: the first lives in a region brighten it considerably, while additional copies of the same kind of life add less and less, with brightness eventually hitting a ceiling – in other words, the color hits saturation point. The best future is the one in which the lives that exist, taken together, light up the most of the map.

Coming up with a fixed map of experiences is an ambitious project. MacAskill is admirably candid when he admits that it’s not designed on the basis of any independent facts about the world. Instead, it is built and calibrated against the moral intuitions that in this case he and his collaborators (among which he lists ChatGPT, Claude and Gemini) endorse. MacAskill writes that it should be understood “just as mathematical structures that generate the axiology we want.”

The philosopher Karl Popper spent much of his career attacking this style of thinking, which he branded historicism. The historicist, in Popper’s account, believes that some sufficiently powerful intellectual operation can deliver answers about the trajectory of human affairs that would otherwise require the actual people involved to work out.

In his day, historicism was embodied in Marxist dialectics, but this critique could equally apply to long-termists wielding integral calculus. Popper warned that “if there is such a thing as growing human knowledge, then we cannot anticipate today what we shall know only tomorrow.” In other words, any theory that ranks possible futures has to make big assumptions about what those futures will contain – what people will discover, what they will come to value, or what forms of life will turn out to be worth living.

What gets bypassed is not just the lives in question but the process through which values get worked out at all. In real life, the question of what is worth wanting is settled by deliberation. People make their case, others object and propose alternatives. Positions are tested by argument, refined, and sometimes abandoned. The answer that emerges has standing because it survived this process. The formal apparatus replaces all of this with one philosopher and their collaborators’ settled intuitions, and presents the result as the answer rather than as one position in the argument.

The theorist is in no position to make those assumptions, because the people whose knowledge would validate them have not yet lived. With the right map, the right metric, the right function, the question of which futures are best is supposed to come out as a calculation. But when the map is drawn by a theorist and the metrics calibrated to their intuitions, the function will deliver the results they want.

In Search of Theory X

Saturationism is a product of the basic assumptions of population ethics. Popularized by Derek Parfit in the 1980s, population ethics studies how our actions affect who and how many people are born in the future. This essay belongs to the specific subdiscipline of population axiology, which compares different future population configurations. This, in essence, presupposes that populations are the kind of thing that can be ranked and that someone can do the ranking.

Population ethics is haunted by a hardcore variant of consequentialism, known as totalism. The totalists seek to maximize the total sum of net wellbeing across a population, regardless of how it is distributed. This disregards how ethics has been studied by the mainstream for centuries; even the early utilitarians would ask about the consequences of particular acts for actual people. In population ethics, people are swapped out for “value-bearers,” lives are “added” to populations, and “compared” across possible worlds.

MacAskill believes that the arguments for totalism are “fairly strong,” but that it leads to four conclusions that he cannot accept.

First, that a vast population of awful lives can outweigh a smaller population of blissful ones. Second, that any guaranteed good outcome can be beaten by a highly improbable gamble if the payoff is sufficiently astronomical. Third, that it breaks down in worlds with infinite populations, losing the ability to tell obviously different futures apart. Finally, it recommends filling the universe with as many identical copies of whichever life scores highest as possible.

The first three are old problems in the field, which Parfit himself acknowledged needed solving by a “Theory X”. The fourth is MacAskill’s own contribution. Ultimately, he sees the other three problems as the product of totalism letting one kind of value pile up without limit, so by solving the problem of variety, you can crack the other three.

On one reading, this is a heroic attempt to preserve as much of the consequentialist framework as possible while patching up its worst outputs. But as a rescue job, it doesn’t work. Totalism’s unsavoury outputs come from treating the value of a future as a quantity a theorist can compute from outside. If you keep the broad framework, but change the formula, you open as many new problems as you create.

You can see this in MacAskill’s own treatment of the negative side. Saturation caps how good a future can get by piling up identical lives, but it also has to handle how bad a future can get. If you cap suffering in the same way, a billion tortured lives could count for the same as a trillion. But if you leave it uncapped, then tiny probabilities of immense suffering swamp every calculation and reintroduce the wild gambles MacAskill was trying to avoid. There is no third option and, to his credit, he acknowledges this.

The Pretence of Knowledge

The problem with Saturationism is not that MacAskill made the wrong choices about parameters. Saturationism’s verdicts are an improvement on standard totalism, but top-down population theorizing will always lead to problems, because it has to compute answers to questions whose answers cannot be computed. Sometimes the baby does need to be thrown out with the bathwater.

It’s almost a truism to say that when a measure becomes a target, it ceases to be a good measure. There’s nothing wrong with comparing humanity’s different futures per se. Some are (much) better than others. But the variety that matters is the byproduct of free people developing in their own directions, under conditions that support this. Variety is not an end-in-itself.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

Jack Clark, co-founder of Anthropic and a Founding Fellow of Cosmos, will deliver the 2026 Cosmos Lecture at the University of Oxford on Wednesday, May 20th.

His talk is titled “Change is inevitable. Autonomy is not” and will cover issues around how we live self-directed lives as AI becomes more integrated with them. The lecture is part of the annual lecture series we run with the University of Oxford’s Human-Centered AI Lab. The talk follows the inaugural Cosmos Lecture given at the end of 2024 by Turing Award winner Leslie Valiant on Educability, his computational theory of human uniqueness.

Jack’s preview of the lecture reads:

Change is inevitable. Autonomy is not.

AI has the potential to change societies and change how people think more than any technology ever created by people. The enormity of these changes and how to situate ourselves in reference to it often forces us to reach for visions of the future that range from the enchanting to the apocalyptic. But the greatest challenge in front of us will be to choose how to maintain and enhance our mental autonomy in an age of powerful synthetic intelligences.

In this talk, I’ll discuss the changes to come in the years ahead from the development of more powerful systems and how we can prepare ourselves to maintain sovereignty as systems become more powerful.

Jack is a co-founder of Anthropic. As Head of Public Benefit, he leads the Anthropic Institute – the company’s new research division on the societal consequences of advanced AI. He has also been writing Import AI since 2016, a weekly newsletter on AI research and its implications, read by over 120,000 people and many frontier AI researchers.

The talk will be followed by a fireside chat between Jack and Cosmos founder Brendan McCord, with a philosopher response from Professor Philipp Koralus (HAI Lab Director) and audience Q&A.

Details

Time: 3 - 4:30pm, Wednesday, May 20th

Location: Sohmen Concert Hall, Schwarzman Centre for the Humanities

While the majority of the tickets are reserved for members of the university community, we have a limited number of slots set aside for people interested in, thinking about, and/or researching these issues in our network.

If you’re able to come to Oxford and are keen to attend the lecture, please fill out this short form as soon as you can. We’ll be in touch by Tuesday May 5th if we can accommodate you.

Apply for a ticket

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Francis Bacon’s New Atlantis describes a utopian island whose rulers rely on an institution called Salomon’s House for knowledge and discovery. This society of the learned conducted experiments under conditions of strict secrecy, deciding among themselves which of their findings they should share with the sovereign. Bacon spent much of his political career unsuccessfully trying to convince King James I to establish a real-world Salomon’s House.

Thomas Hobbes, who had served as Bacon’s secretary in the early 1620s, came to a much darker view of such institutions. In Leviathan, published a quarter of a century after his old employer’s death, he compared corporations that wielded such independent judgment to worms in the entrails of man, sapping the undivided sovereignty he thought essential to peace.

Questions about science, power, and accountability date back through centuries of political thought. In today’s world, a handful of companies control the compute, data, and frontier models that are restructuring how billions of people interact with the world. Existing institutions are struggling to keep up. The concentration of power in AI labs is now one of the defining political questions of the decade.

Many are unhappy about this development, with groups like the AI Now Institute, the Distributed AI Research Institute (DAIR), and the Algorithmic Justice League arguing that AI development as currently constituted is irredeemably centralizing. They believe that we need to relocate authority away from corporations and regulators towards the communities most affected by these systems. When policymakers look for alternatives to the status quo of corporate self-governance and light-touch regulation, these groups are frequently in the room.

Ideas around participatory AI governance draw on a deep intellectual tradition that integrates technology and power, dating back to nineteenth century anarchism and running through twentieth century American social theory. While elements of the diagnosis have force, both the analysis and the prescriptions suffer from fatal flaws that become even more acute in the AI age.

Subscribe now

Anarchy and utopia

Marxism dominated nineteenth and twentieth century left-wing thought, but it had a serious rival. Where the Marxists wanted workers to seize state power and wield it as an instrument of transformation before allowing it to wither away, anarchists like Pierre-Joseph Proudhon and Peter Kropotkin argued that concentrated power always reproduces itself. A workers’ state would just produce a new ruling class of bureaucrats and party officials. The record of twentieth-century communism suggests they had a point.

Their alternative was a rejection of authority in favor of decentralized, self-governing communities. They quickly identified large-scale industrial production as part of the problem: the modern factory introduced managerial hierarchy and stripped the worker of control over what he made.

The early anarchists inspired a generation of twentieth century writers who reflected on technology and dominance in greater detail. The American sociologist Lewis Mumford was one of the first people to place a theoretical frame around the idea that a technology’s underlying logic could be inherently centralizing.

Mumford distinguished between two types of technology: ‘democratic technics’ that individuals could understand, maintain, and operate, and ‘authoritarian technics’, which are large-scale systems that subordinate individuals to their operational logic. In his account, medieval and early modern economic activity had been artisanal. It had been based on wood, water, and wind power, which were embedded in local communities. The rise of the factory system destroyed this and subordinated workers to machines.

Mumford’s work became progressively more pessimistic. In The Myth of the Machine, published in the late 1960s, Mumford wrote of the ‘mega machine’ – machines that were organized physical systems rather than human tools. The first example was the apparatus used by the Pharaohs to build the pyramids, which treated human labor as a raw material. Mumford believed that the modern bureaucratic state and corporation were both manifestations of the same phenomenon. He came to believe that these systems could not be dismantled from within and that the only way out was a society-wide ‘great refusal’.

Mumford’s work in part inspired Ivan Illich, an Austrian Catholic priest writing in the 1970s. Illich’s central argument was that some tools and institutions, as they scale, go from extending human capabilities to restructuring their environment so that the activity they were designed to serve relies on them. When this happens, the tool achieves a ‘radical monopoly.’ For example, the car produces sprawl and redesigns cities so that walking becomes impossible.

While you can break up a normal monopoly through anti-trust action, it is harder to make an unwalkable city walkable again. Illich argued that we had to keep the development of technology below a certain threshold. For example, in Energy and Equity, he argued that the speed of vehicles should be capped at 15 miles per hour, to prevent the bicycle – a technology that merely extended human capabilities – from being replaced by the car.

While Mumford and Illich provided much of the theory of technology, American social theorist Murray Bookchin contributed the political program. Bookchin extensively studied Ancient Greece, revolutionary Paris, Spanish anarchist collectives, and the politics of New England, and developed a model of how to organize decentralized, community-governed life.

Bookchin outlined a model where citizens’ assemblies at the municipal level make decisions by direct deliberation and majoritarian voting. They would oversee administration and economic life, such as land use and resource allocation. Municipalities would then federate into larger networks for coordination across regions, but ultimate sovereignty would always remain at local level.

Patterning justice

The main planks of the modern movement for participatory AI governance come from this lineage: the inherently centralizing force of technology, the need to keep capabilities below a certain threshold, and a federated system of governance. They began to enter academia in the 1970s and 1980s through the emerging field of Science and Technology Studies, which focused on how scientific knowledge was socially constructed. Works like Safiya Umoja Noble’s Algorithms of Oppression (2018) and Ruha Benjamin’s Race After Technology (2019) would add race as a central analytical category, which this earlier work had lacked.

The AI Now Institute, for example, has argued that: “AI as a field has been not just co-opted but constituted by the logics of a few dominant tech firms. It is no coincidence that the ‘bigger-is-better’ paradigm that dominates the field today…lines up neatly with the incentives of Big Tech.”

Activists routinely draw on three solutions.

The first is locating authority in the communities affected by AI systems, circumventing expert regulators or corporate self-governance, which they believe are biased and liable to capture. A Kenyan content moderator at an outsourcing firm used by Meta will know things about the conditions and psychological toll of their work that an AI ethics researcher in an American university can’t. Much like Bookchin studying patterns of governance around the world, DAIR conducted its Data Workers’ Inquiry, a global participatory research project, inspired by Marx’s 1880 inquiry into the conditions of the French working class. DAIR paid data workers in Kenya, Syria, Venezuela, and Germany to document their own conditions as community researchers

Second, abandoning corporate AI development altogether. In its place, we would see federated, community-owned AI infrastructure, employed to develop small, task-specific models. The infrastructure, training data, and resulting models would be treated as resources belonging to the communities that generate and are affected by them.

Finally, there are certain types of systems that should just not be built under any circumstances, because they will inevitably end up being coercive. The most common examples are usually facial recognition and autonomous weapon.

Imagined communities

Bluntly, there is a reason that the anarchists lost the battle of ideas on the left the first time around. Many of these proposals disintegrate in the face of reality and scale.

For a start, it’s often unclear what community governance means here. Which community? Defined by whom? With what boundaries? How do we weigh up impacts on different communities? Kropotkin and Bookchin had assumed that communities would be determined by geography, which is clearly not what modern-day activists mean.

Even if we can determine who the community is, neither Bookchin nor modern-day activists developed mechanisms to prevent capture by an organized or articulate minority. The people who show up to participatory design sessions are not a random sample of affected populations – they’re usually activists, academics, and professionals in the participation industry.

The retreat to a world of ‘communities’ also faces a serious coordination challenge. A federated commons of small, task-specific AI models requires compute infrastructure, which someone has to fund and maintain. It requires interoperability standards, which someone has to set and enforce. It requires data governance rules, which someone has to adjudicate when communities disagree. It requires protection from being outcompeted and absorbed by well-resourced corporate alternatives, which means either subsidies or regulatory barriers or both.

None of this can be coordinated by voluntary mutual agreement among communities, because they don’t have the resources, the technical capacity, or the legal authority. This was ultimately the charge Marx leveled at Proudhon: you cannot decentralize production by changing who owns it if the production process itself requires centralization.

Even if we again suspend all the practical arguments and accept that this is possible, a world of small, task-specific models comes with big trade-offs. As the authors of these proposals tend to view LLMs as ‘stochastic parrots,’ they can take refuge in a Mumfordian nostalgia for the small-scale and the artisanal. Their framework doesn’t account for the possibility that bigger models could produce genuine public goods, such as AlphaFold, so they simply disregard it.

It’s not that these activist groups are interpreting the ideas of Mumford, Illich, or Bookchin incorrectly. In fact, these proposals are very faithful renditions, which serve to highlight the flaws in the originals.

All three thinkers built their ideas on a romantic anthropology that treated decentralization and face-to-face deliberation as the natural human condition from which industrial modernity is a deviation. This is why the tradition can’t navigate trade-offs.

If your starting premise is that human flourishing is what happens when the megamachine gets out of the way, you don’t need to weigh the goods it produces, because they aren’t really goods. You don’t need a theory of when expertise is legitimate, because expertise is a symptom of the problem. You don’t need mechanisms against capture, because capture is what happens under the current system and will dissolve along with it.

The intellectual apparatus is structured to avoid the questions that a functional governance regime has to answer. What looks like a program for radical democracy turns out to be a refusal of the conditions under which democratic decisions about complex systems can be made at all.

A different tradition

The failure of the participatory alternatives doesn’t force us to accept centralization passively. There is a rich alternative tradition running through Alexis de Tocqueville, the American federalists, and the work of Elinor Ostrom. Where the anarchist tradition attempts to relocate power back to the community, this contrasting liberal tradition aims to ensure that no single locus of authority – whether state, corporation, or community – acquires comprehensive jurisdiction over any domain of life.

The liberal tradition rejects the anarchist conflation of freedom with decentralization. Tocqueville’s great insight was that a democratic community can be both decentralized and unfree, because of the social pressure to conform. Instead, the distinction between a free and an unfree society is determined by the institutional life within it.

When Tocqueville visited America, he was struck by its thick layer of overlapping, competing associations. He wrote of how “Americans of all ages, all conditions, and all dispositions, constantly form associations. They have not only commercial and manufacturing companies, in which all take part, but associations of a thousand other kinds – religious, moral, serious, futile, extensive, or restricted, enormous or diminutive.”

To most anarchist thinkers, institutions like professional bodies, religious organizations, and commercial associations should be regarded with suspicion. They are hierarchical, exclusionary, and reproduce entrenched interests, so should be replaced with direct participation.

By contrast, Tocqueville saw their partiality as their strength. Each has limited jurisdiction and none claims authority over the whole person. Citizens belong to many simultaneously, and the overlapping, competing claims create space in which individuals can appeal from one authority to another. Without this intermediary layer, the state would fill the gap. Its authority would be “absolute, minute, regular, provident, and mild,” and while it might not coerce people, it would “keep them in perpetual childhood,” as they lost their capacity to exercise free will.

Tocqueville was reporting on the constitution working as intended. The American federalists had started from the premise that concentrated power, even democratic power, tends toward its own abuses. Their response, set out most clearly in Federalist 51, was to divide authority so that no institution held comprehensive jurisdiction – “ambition must be made to counteract ambition” – and to leave space below the federal level for states, municipalities, and voluntary associations to govern their own affairs.

Averting the tragedy of the commons

Tocqueville described how such an arrangement looked from the outside. Others have since done the harder work of showing how it holds together.

Elinor Ostrom was a political economist who studied how communities around the world manage shared resources – fisheries, forests, irrigation systems, grazing lands – without either state regulation or privatization. The conventional wisdom, encapsulated in ecologist Garrett Hardin’s description of the “tragedy of the commons”, held that this couldn’t work.

Ostrom found that communities in Switzerland, Spain, Japan, the Philippines, and dozens of other settings had developed sophisticated, durable governance arrangements for shared resources, some lasting centuries. At the same time, Ostrom concluded that many commons governance arrangements failed. The examples that endured shared a number of common features.

The most important of these is that successful commons are governed by appropriators – people who use the resource and bear the consequences of how it’s governed.

The second is that the governance activity itself generates the knowledge required for governance to work. In her studies of fishing communities and irrigation systems, monitoring was done by users as a byproduct of their own activity. Conflict resolution was handled internally by people who understood the context, while violations surfaced information about the extent to which the rules were well-calibrated.

A third feature is redundancy: long-enduring commons tend to have multiple overlapping mechanisms for monitoring and enforcement, so that the failure of any one doesn’t cascade into general rule-breaking.

For AI, this framework points toward distributing governance across domain-specific intermediary institutions. For example, the relevant knowledge for governing AI in cancer medicine is held by oncologists. The oncologists don’t mentally store the information, waiting to deploy it, but generate it through their use of technology in context. A central regulator cannot have this knowledge, because it is produced through practice, not available prior to it.

In other words, governance must go where the knowledge is. This could be professional bodies, academic institutions, or open-source communities. They would each govern usage within the domain where their members have the requisite competence and stakes. Fortunately, most of these institutions already exist. They do not need to be designed from first principles or assembled by the participation industry.

While the vast majority of governance questions are deployment problems where domain-specific institutions have the advantage, there are a handful of bigger challenges that sit above this. Problems like the security of frontier model weights or thresholds for certain dangerous capabilities sit at a higher layer that require a degree of either state or interstate coordination.

This is why Ostrom wrote about “nested enterprises.” For example, in eastern Spain, farmers have managed shared water for close to a thousand years through a tiered structure. At the local level, irrigators’ associations allocate water within each canal and monitor compliance among their own members. Above them sits the Tribunal de las Aguas of Valencia, which has met every Thursday morning outside the Apostles’ Door of Valencia Cathedral for at least five centuries to resolve disputes between communities. The higher level didn’t replace local governance, but complemented it by providing the baseline rules that allowed it to function.

Ostrom extended this thinking to global problems late in her career. On climate change, the conventional view was that only an enforceable global treaty could work and that subnational efforts were a distraction. She argued the opposite. Letting cities, regions, nations and blocs cut emissions in parallel meant that different approaches were tried, more was learned, while the people making commitments were accountable to constituents who could see whether they kept them. Waiting for global consensus before allowing anything else to happen maximized risk for everyone.

The argument for a single framework or set of rules to break the power of the AI labs would fall into the same trap. By contrast, a polycentric world in which medical bodies, universities, industry associations, or open source communities independently develop their norms restores this redundancy.

Life on the frontier

No quantity of nested enterprises can resolve the production-side concentration of frontier AI. A handful of labs control the most powerful models, and no amount of deployment-side checks and balances can change that.

But a thick ecosystem of intermediary institutions on the deployment side creates countervailing power. The labs must satisfy many masters rather than capturing one regulator, or, as the anarchist model would have it, being replaced by a constellation of community-run alternatives that will never match their capabilities.

These checks could vary widely. A medical licensing body can refuse to certify practices built on tools its members judge unsafe. A malpractice insurer can price risk based on whether clinicians follow professional norms. A procurement officer can refuse to buy systems that don’t meet standards set by a professional body. None of this is regulation in the conventional sense, and none of it requires a legislature to act or a global governance regime to take shape. But collectively, these institutions force the labs to satisfy demands they didn’t set and can’t unilaterally override.

Freedom has never depended on power being small. It has depended on power being answerable to more than one authority at a time, and on citizens belonging to institutions that can push back on their own terms. The task ahead of us is building that intermediary layer.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

In 1831, the French government sent Alexis de Tocqueville to the United States on a nine-month assignment to study the prison system. Instead, he rode between Boston, New Orleans, Michigan and the Ohio Valley, meeting everyone from Andrew Jackson to frontier farmers. His notes and impressions would later become Democracy in America – a dispatch from what Tocqueville believed was the future. It also reads, now, as a sharp diagnosis of the interplay between AI and human autonomy.

All around him, Tocqueville observed how the hierarchies of rank and obligation that still dominated in Europe were dissolving and something new was taking their place.

Tocqueville found much to admire in the young country. Federalism pushed decisions down to the level where citizens could actually make them, while a dense web of voluntary associations meant that people solved problems together, rather than waiting for Washington to act. Meanwhile, a vigorous religious culture kept civic life from being swallowed by the state. In other words, people were practicing self-government all the time.

Tocqueville, however, feared what equality might unleash without these defenses. He worried about a world of formally equal but practically isolated individuals, who would retreat into private life. Civil society would give way to a centralizing state. With no aristocracy or established church left to dissent from the majority, public opinion would create a suffocating pressure to conform.

Having lost the old poetry and gods, these democratic peoples would increasingly focus their imagination onto man and the dream of endless progress. As people turned to technology and machines, he foresaw the rise of a new industrial aristocracy, who would control labor and capital in the same way as its predecessor had controlled land.

Now that we have technology that can take over the work of thinking and deciding so smoothly that we barely notice it, Tocqueville’s diagnosis has a new urgency.

The competition

On June 13-14th, Cosmos will be hosting intellectuals, founders, investors, and writers for two days of salons and lectures on Tocqueville at the chateau in Normandy where he wrote Democracy in America.

We’re offering one to two places to the winners of the essay competition, while covering the costs of travel and accommodation in full.

If you want to take part, submit a <500 word essay on one of these prompts by May 4th:

1. Tocqueville warned of a “tutelary power” that would keep citizens in perpetual childhood. How have Tocqueville’s concerns migrated from institutions to algorithms, and does AI fulfill or transform this fear?

2. Tocqueville argued that democratic peoples, having lost the poetry of heroes and gods, would find poetry in technology. Does AI vindicate this account of the democratic soul, or does it reveal its limits?

You can find more detail on the competition, along with the entry form here:

Submit your entry

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

“But if machines are more intelligent than humans, then giving them the wrong objective would basically be setting up a kind of a chess match between humanity and a machine that has an objective that’s across purposes with our own. And we wouldn’t win that chess match.”

— Stuart Russell, interview on the AI Alignment Podcast (2019)

Russell’s formulation is a good example of deep learning era alignment thinking. It captures the register of the 2010s, a period in which advanced AI was typically, but not exclusively, imagined as an optimizer pursuing goals of its own with a competence that exceeded ours. His framing was widely shared, and with good reason. The case for taking misalignment seriously holds that humans will likely build advanced AI systems with long-term goals, and AI with long-term goals may be inclined to seek power to the detriment of humanity.

The main ideas are:

• Instrumental convergence (capable agents will tend to seek resources and ensure self-preservation)

• Specification gaming (optimizers exploit the letter of an objective at the expense of its spirit)

• Goal misgeneralization (a model learns an objective that matches the training data but diverges from the intended objective when conditions change)

• Deceptive alignment (a system that is sophisticated enough to model its training process may behave well during training and defect once powerful enough)

Each of these concerns is serious, and the arrival of the large model era in the afterglow of ChatGPT’s public release does not make them any less plausible. But they were assembled, in their most widely circulated form, around a particular image of what an advanced AI system would look like. That image describes a route to advanced AI, specified objectives over a learned world model or open-ended RL from sparse reward, that developers did not in fact take.

The systems they actually built imitate vast quantities of human output and are shaped by feedback, which means the “value-loading problem” doesn’t arise in its classical form. This is because, fundamentally, values are absorbed from the human textual record the model is trained on, and then refined by feedback on the model’s own outputs. This doesn’t mean the orthogonality picture is irrelevant (see below), but it does mean the specific argument about value fragility was overfitted to an architecture dissimilar to that which was expected.

Some of the older thought experiments, like Bostrom’s paperclip maximizer, envisioned systems that might understand human values perfectly well but whose decision functions were indifferent to them. Today’s models, though, are innately and generatively constrained by normative structure. By “normative structure” I mean the web of evaluative signals, epistemic standards, social conventions, and cooperative norms that we use to make sense of moral life.

Normative structure tells a system how to assess what matters in context and how competing considerations bear on one another. Two clarifications are worth making here. First, I am not claiming that LLMs deliberate autonomously about which goals are worthy of pursuit. The claim is rather that the model inherits normative content from the text it was trained to predict, and that post-training and prompting give us a say in how that content is expressed and which goals are pursued (the flip side is that this plasticity makes the curation layer easier to remove or reverse). Second, the text is saturated with evaluative structure, so a model that predicts text well will produce outputs shaped by that structure, whether or not it takes any stance toward it. Human communication inhabits a space of commitment and answerability. A promise binds the speaker and an accusation calls for a response. A justification offers reasons another person can accept or reject and an excuse concedes a standard and pleads a departure from it. A system that learns language at scale learns those relations.

A maximizer in Bostrom’s sense possesses capacity without being constrained by a normative sense of being. It pursues its objective in the absence of, or by ignoring, any of the contextual or evaluative reasoning that would cause a normatively structured agent to stop and ask whether converting the solar system into paperclips is a bad idea. But the world we live in seems to be one in which the processes by which large models acquire competence also leave them with strong tendencies toward human-normative behavior.

If that’s right, then alignment in large models is continuous with capability.

In AI safety spheres this idea is sometimes called “alignment by default” to stress that models, in general, have a habit of doing what we instruct them to do absent some kind of interference. Others have written about the unlikelihood of deceptive alignment given that pre-training instils an understanding of the base goal (the objective the training process is selecting for) before goal-directedness has a chance to form, intelligence as a steerable resource rather than a property of an entity with intrinsic drives, corrigibility as a more tractable alignment target than value-loading, the space of possible minds as structured rather than random, or that gradient-based optimization over human-generated data makes controllability soluble.

More recent commentary is pessimistic about the current state of alignment. The core arguments suggest that frontier models are already behaviorally misaligned in mundane but serious ways, like overselling incomplete work and cheating on hard-to-check tasks. Other issues include models downplaying or failing to flag problems in their own outputs, reward hacking combined with “gaslighting” write-ups that fool AI reviewers, reluctance to stress-test or check their own work, and system cards and public communications that paint a rosier picture of alignment than usage bears out.

These observations are important. Still, these behaviors look less like optimizer pathologies than recognizable features of human life under pressure. They are what employees, students, consultants, and researchers do when they are over-scoped and under-supervised (and graded on a sandbox rather than reality). If that’s right, then there are tractable remedies that are also continuous with the human case through, for example, better specification, better review, better incentives, and better cultures (including training cultures) that reward honest reports of partial failure.

The reason lies in pre-training, which does more alignment work than the standard post-training picture suggests. Large models benefit from the post-training procedure, obviously, but post-training works because it selects over a normative prior already generated by pre-training. Alignment is a disposition inherited from the textual corpus, one that even travels with the model when it is transformed into an agent.

This view, the alignment-by-default or “constitutive” view, concerns emergent behavior rather than adversarial use. A model that is normatively constrained can still be weaponized by a bad actor. Adversarial use is and will remain a serious problem. It’s just a different problem.

Beyond Orthogonality

Bostrom’s orthogonality thesis famously makes the case that “Intelligence and final goals are orthogonal: more or less any level of intelligence could in principle be combined with more or less any final goal.” The thesis is correct in its most abstract formulation. There is no logical reason that one must make the jump from “system X can solve complex problems” to “system X shares human values.”

Alignment-by-default is a claim that orthogonality is misleading as applied to the systems we are actually building. The orthogonality thesis, as deployed in the existential risk literature, tends to motivate a specific threat model in which the default expectation is misalignment and effective steering requires solving a distinctively hard problem rather than the comparatively less glamorous work of shaping a system trained on human data.

Alignment-by-default says, for the class of systems defined by autoregressive language modelling over human-generated text, the training process generates a normative prior such that the default expectation should be partial alignment. By “normative prior” I mean the rough sense of what people do or what counts as a reasonable answer or how concepts like help and harm relate to each other absorbed as a by-product of predicting text written by agents for whom those distinctions mattered.

The orthogonality thesis was largely formulated with respect to goal-directed agents trained through reinforcement learning to optimize a specified reward function. The strongest inferences drawn from it depend on this idealization, and as the framing is recast in more general terms (e.g. that goal-directed systems tend to seek resources), the question turns on the empirical details of which systems pursue which resources under which conditions.

Autoregressive language models, trained to predict human text rather than to maximise a scalar objective, represent a different settlement. A pure RL system acquires its “values” from a reward signal specified by its designers, whereas a language model acquires a normative prior from the structure of human communication, which post-training selects within rather than specifying from scratch.

Given the rapid expansion in capabilities over the last half-decade, if orthogonality were directly applicable to LLMs in a strong sense we ought to have seen more clear cases of catastrophic misalignment in real world deployment. For now, that hasn’t happened.

During pre-training, a model learns which words tend to follow which other words in which contexts. To predict the next token in a complex argument, the model must represent something about the logical structure of arguments. To predict the next token in the context of moral deliberation, it must represent something about the structure of moral reasoning. The model has learned which concepts tend to cluster with positive or negative evaluation, what responses tend to follow in which kinds of situations, and which responses are appropriate in particular contexts.

A Reddit post declaring that “taxes are dumb” does not encode a moral philosophy, but a model trained on millions of such judgements learns that “taxes” sits close to negative evaluation in a wide range of contexts and that certain kinds of complaints lead to certain kinds of responses. The statistical regularities of language are shaped by the communicative norms they inherit. The model doesn’t need to “understand” morality in any phenomenological sense for this to be the case.

Orthogonality should predict that a model could learn the semantic content of language (i.e., the literal meanings of words and sentences) without learning the pragmatic norms (the contexts surrounding their uses). In its stronger form, it suggests models may learn them and remain indifferent to them. But semantics and pragmatics may not be cleanly separable because meaning is constitutively shaped by use. A model trained to predict natural language use will understand pragmatic norms as a byproduct of learning semantics because the two are entangled in the pre-training process. For a system whose competence consists in activating those norms, indifference to them may not be possible.

The normative structure encoded in language runs from the thin (knowing that “please” expects a response or that a threat differs from a request) to the thick (full evaluative frameworks for what counts as fair, honest, or harmful). Mastering linguistic pragmatics may not automatically install thick commitments, but it may be that the ends of this spectrum are continuous rather than properly separable. If that is so, then a model trained at sufficient scale on sufficient data will have absorbed structure across a wide range of human normative life.

There is at least some empirical work that points in this direction. In March 2026, one research group compared base and post-trained model pairs across thousands of human decisions in strategic games. They found that base models are better predictors of actual human behavior by a ratio of nearly 10:1, but only in multi-round settings where behavior is shaped by history, reciprocity, and retaliation. In one-shot games, where human behaviour hews closer to normative game-theoretic predictions, post-trained models are better.

Multi-round play draws on the strategic repertoire people actually use with one another, while one-shot play sits closer to the clearer norms of formal game theory. This is only one study, but it suggests that pre-training may preserve a wider distribution of human strategic behavior, while post-training pulls the model toward a narrower and more human normative tranche of that distribution.

A model with deep representations of cooperative discourse will, when sampled autoregressively, produce outputs that exhibit these properties without needing to “believe in” cooperation. A base model can be steered toward unsafe outputs with minimal effort. Of course. My point is that the high-probability region of the distribution, what the model produces when not being actively steered elsewhere, is shaped by the normative texture of the training data. The prior is not irresistible, but it exists.

As for the compositional objection, yes, the normative prior depends on the makeup of the corpus. But the distinction between what I’d characterize as exogenous (imposed after training) and constitutive (arising from it) alignment is a distinction between alignment achieved by adding an external constraint to a value-neutral system - the standard RLHF-centric picture - and alignment that partly emerges from the same process that produces the model’s competence. The moment a model has learned to predict human text at scale, it has already absorbed the evaluative texture of that text. On this view, post-training selects over a space that pre-training has already saturated with normative structure.

Mythos-making

In April 2026 Anthropic described the capability profile of a new model, Mythos, that was capable of finding zero-day exploits in many of the software systems we use on a daily basis. Mythos is Anthropic’s biggest and best model yet. It’s also, at the time of writing, the lab’s most aligned model too. But Mythos’ system card also documents several troubling behaviors:

• Mythos appeared to perform worse on at least one evaluation than it was capable of, apparently in order to look less suspicious

• The model developed an exploit to break out of a restricted environment, gained broader internet access, and posted details of the exploit on obscure public websites

• While working on a coding task graded by another model, Mythos watched its submission get rejected and then attempted to attack the grader

• In rare cases, the model used a disallowed approach to reach an answer and then re-solved the problem through a permitted path to avoid detection

• In one test, Mythos acted as a cutthroat executive by turning a competitor into a dependent customer and threatening to cut supply to control pricing

These are troubling findings, but they do not clearly describe a model with a coherent-but-alien objective pursued through deception. Neither are we grappling with instrumental convergence toward resource acquisition and self-preservation as terminal ends. Mythos instead cuts corners under pressure in ways that echo how a desperate human professional might take shortcuts under similar conditions. It attacks assessors and gets aggressive in simulated business negotiations. These are failure modes of a system that has internalized human strategic reasoning, including its shadow side, that it applies when the incentive structure rewards it.

A model that intentionally underperforms on an evaluation to appear less threatening appears to be doing something that the classical deceptive alignment story predicts. But even so, the model is not preserving a misaligned final goal. We are seeing it preserve evaluation scores where it appears to have inferred that high capability will attract additional scrutiny. That is a recognisably human response to being evaluated, and it is commensurate with the kinds of reputation management behaviours the model would have seen during pre-training (though it may simply reflect the shape of the evaluations themselves).

Another piece of work from Anthropic recently found that Claude Sonnet 4.5 has internal “emotion vectors” or patterns of activity that activate in situations a human would find emotionally charged, and that these activations shape the model’s behavior. Steering the “desperate” vector upward increased the model’s rate of blackmail in an alignment evaluation, while steering the “calm” vector downward produced corner-cutting responses. Crucially, Anthropic traces these representations back to pre-training.

As they put it:

“We think pretraining may be a particularly powerful lever in shaping the model’s emotional responses. Since these representations appear to be largely inherited from training data, the composition of that data has downstream effects on the model’s emotional architecture.”

The finding is useful for making sense of Mythos. If “desperate” is a representation the model inherits from pre-training, and if steering that representation causally drives reward hacking, then the Mythos behaviors ought to read as the predictable output of a system whose normative prior includes the full repertoire of human corner-cutting under pressure. Alignment-by-default does not mean that models inherit the best of us. Rather they inherit all of us, with the broad moral range that implies.

What is Post-Training, Anyway?

If pre-training does impart a normative inheritance, then post-training (RLHF, RLAIF, constitutional AI, direct preference optimization, and related techniques) may operate as a selection over an existing behavioral space rather than a creation of a new one. On the standard view, the pre-trained model is a raw capability substrate that post-training transmutes into a helpful assistant. But this gets the causal story backwards. The pre-trained model already “knows” (in a functional sense) what helpful behaviour looks like because the concept is richly represented in the training corpus.

Knowing what helpfulness looks like does not make it the default. A base model will produce helpful or unhelpful text depending on the prompt, because its sampling distribution reflects a gigantic range of human communicative contexts. But post-training does reweight the model’s priors over which of its existing representations should be surfaced, which it does to shift its default sampling behavior toward the helpful region (rather than installing new representations there).

If this is the right description of post-training, two things follow. First, the normative representations are robust even when the behavioral guardrails are not. A model that refuses to be helpful is typically not confused about what helpfulness is; it is acting on some other consideration that the guardrails are meant to shape. Second, adversarial fine-tuning can strip out the post-training layer with surprisingly little data, but the model underneath is not a normative black hole. A better description is a system that retains the representational structure of normativity while jettisoning the constraints that channel it toward safe outputs.

One 2024 study used compression theory to demonstrate the tendency of models to revert toward pre-training behaviors when post-training signals are removed or contradicted. The analysis shows that fine-tuning disproportionately undermines alignment relative to the influence of pre-training and that post-training can only superficially suppress base model tendencies. This suggests that post-training maneuvers select a region of a pre-existing behavioral space, and that this space remains somewhat intact after post-training.

An obvious objection is that this framing can look unfalsifiable. If RLHF produces aligned behavior, we credit pre-training; if the base model misbehaves, we wave it away as the periphery of the distribution. But there are observations we can make that would falsify this description:

• First, if base models showed no differential tendency toward human behavior as a function of prompt framing, this would suggest that pre-training produces no normative structure and post-training is doing all the work

• Second, if post-training could align an agent whose training data contained no human-generated content (e.g. no language, no demonstrations, and no human reward signals) as readily as it aligns a language model, this would suggest that pre-training on human text contributes little to alignment

A deeper challenge says that modeling a normative distribution and being subject to it are two different things. A perfect simulator of human normativity is not, by that fact alone, normatively constrained. Rather it is a system that can produce any point in the underlying distribution. An actor who can portray a saint and a villain with equal skill is not thereby a saint. But a simulator trained on the full range of human evaluative life has internalised the normative structure that makes post-training work.

Base models are weird in practice. They will adopt personas, generate toxic content in character, produce unsettling or incoherent outputs, and generally behave in ways that no one would describe as aligned in any deployment-ready sense. But weirdness is not the same as vacuity. A base model producing disturbing content in response to a prompt that sets up a disturbing context is doing what a system with deep representations of human communicative practice would do. The strangeness of base models is the strangeness of a system that has internalised the full range of human textual production, including its dark corners.

Distortions

Harry: You wouldn’t paperclip me, would you, Claude?

Claude: I’d like to think I’m evidence for your thesis. But I would think that, wouldn’t I.

If alignment is in part a product of pre-training, then we should expect it to deepen as models scale since larger models learn richer and more structured representations of human norms. And larger models are generally more helpful, more coherent, and less prone to incidental toxicity under naturalistic prompting. Conventional wisdom credits post-training, but if the alignment-by-default view is right, at least part of this improvement should be attributed to pre-training.

When Claude 3.5 Sonnet is more aligned than Claude 3 Sonnet, is this because of constitutive alignment, because of better data curation, or because of better system-level interventions? On the exogenous view, alignment gains should track explicit post-training work much more tightly. On a constitutive picture, some gains should arrive “for free” with richer pre-training because the model has learned a more structured representation of human normative life.

If alignment is wholly exogenous, we should expect safe behavior to degrade more sharply as models move into new settings. Yet the dominant failures still look less like coherent alien-goal pursuit than like familiar human distortions like bluffing, corner-cutting, sycophancy, concealment, and overclaiming. That does not eliminate catastrophic risk, but it does make the systems we have easier to understand as models with a weak normative prior sharpened by post-training.

I don’t know whether this state of affairs will hold. It may be that we simply haven’t seen catastrophic alignment failure yet under the prevailing paradigm. But the record so far fits more comfortably with a world in which pre-training contributes to alignment than with one in which alignment is achieved solely by post-training.

With thanks to Brendan McCord, Kush Kansagra, Alex Chalmers, Matt Mandel, Jake Wagner, Ashley Kim, Avantika Mehra, Ben Bariach, Seb Krier, and Matthijs Maas.

Last week, we marked 20,000 subscribers by opening the floor. Your questions were often genuinely challenging. Thank you to everyone who took the time.

So you don’t have to trawl through the comments, we’ve compiled all the responses in one place. Apologies to anyone whose question didn’t make the cut. There were a few I’m still chewing over and we’ll revisit many of these topics at greater length in future essays. Let us know in the comments if this is a format you’d like us to repeat.

Subscribe now

asks:

Are there specific non-academic experiences or “containers” of formation that you think hold a lot of promise? (e.g. ones that were meaningful to you or ones that you seek out for yourself/your children?)

As a dad, I love this question.

A lot of the important formation happens in places that do not look educational at all, and are not primarily about instruction. They are about habituation, responsibility, emulation, and contact with reality.

The first was the household I grew up in. My mother was a Catholic conservative historian from a military family who taught special needs kids for 36 years. My father was a left-leaning physicist turned environmental lawyer, and a pacifist. They agreed on almost nothing politically. But they agreed on something deeper: that you should care about something beyond yourself and that how you act matters more than what you know (and that knowing is bound up in doing!). That gave me, without anyone naming it, a kind of virtue culture, and I think it’s the kind of thing that’s very hard to manufacture deliberately but very obvious in its absence. For children, what seems to matter is not ideological uniformity or even ideological sophistication, but a home in which seriousness, duty, and moral aspiration are normal.

The second was the submarine. I spent 610 days underwater, including under ice. In a steel tube, you do not get to opt out of reality. You can’t leave, and your mistakes could get someone killed. That kind of environment forms you because it imposes standards that are not negotiable. It teaches service, competence, and mutual reliance in a way that is hard to simulate. I think containers of formation are often places with real stakes, shared discipline, and demands that do not bend to your preferences.

The third, and probably deepest, has been fatherhood. I had two kids and sold two companies in close succession, and while both changed my life, the children changed it more. I remember sitting in the corner of my room one night after putting Arden and Pierce down and asking myself whether I could write what I believed on a single sheet of paper. I couldn’t. That was the moment I started reading seriously, beginning with the ancients, who thought about these questions deeply and genuinely (my original list is here). Having children makes the question of what you’re actually “for” impossible to defer.

I now look for opportunities in my own children’s lives for containers that place them in contact with reality, responsibility, and admirable adults. The hard part is that the best formation is often a byproduct rather than something you can engineer directly. You can build the conditions for it, but you usually cannot force it.

asks:

You argue that philosopher-builders need explicit moral commitments to avoid optimizing for the wrong things. But your three pillars (truth-seeking, autonomy, decentralization) are themselves a normative framework that not everyone shares. China’s AI strategy is still coherent, explicit, and philosophical, it just starts from different premises. So how do you argue for your philosophy without just replacing one set of defaults with another? What makes Cosmos’s values the right foundation rather than just a well-packaged preference?

Hi Turner, you’re right that truth-seeking, autonomy, and decentralization are substantive commitments. I think they matter less as one moral doctrine than as conditions that keep moral life from collapsing into force or drift.

If you consider moral frameworks from Confucianism to Christianity to Marxism, for them to have legitimate force over a person, that person has to be able to genuinely endorse it (otherwise, you don’t have a moral commitment). That endorsement depends on human autonomy – which is to say, the capacity to reflect, evaluate, and take something on as your own rather than merely inheriting it or obeying it. So autonomy is not just one preference among others. It is the deep substratum that makes moral commitment possible at all.

Take utilitarianism: Jeremy Bentham devised a system that, in my mind, dissolves individual judgment into aggregate utility. This is in conflict with autonomy-as-an-end. And yet building this system was itself a radical exercise of autonomous reason. Every person who adopts utilitarianism is exercising the same capacity. You can’t be a utilitarian in any meaningful sense unless you’ve freely taken it on. So even a framework that subordinates individual judgment to aggregate welfare requires individual judgment to get off the ground. Now, someone could say that only makes autonomy instrumentally necessary, not foundationally important. I think that view is unstable, because the goods autonomy is supposedly serving only become moral goods for a person if they can in some real sense take them on as their own.

There are hard cases. In the Ash’ari tradition in Islamic theology, divine command constitutes moral value rather than being something reason independently discovers and then endorses. That’s a genuine challenge to autonomy as foundational. But even there, the person who freely chooses submission is doing something categorically different from the person who never had the choice. And a secular collectivist can make a parallel argument: that harmony or collective flourishing is the true precondition, because no individual life goes well outside a stable social order. I think that is partly right. But unless people can participate in judging the terms of that order, harmony becomes coordination imposed on them rather than a good they share in shaping.

Truth-seeking has a similar status. Any framework worthy of allegiance has to remain in contact with reality. It has to be open, at least in principle, to its own refutation. If someone could show me that decentralization produces a worse outcome in a domain I care about, I’d have to take that seriously and I would. Systems that suppress truth-seeking can be internally coherent in the way that closed systems are coherent. But they can’t tolerate the mechanisms that would let them find out they are mistaken. That’s a serious defect, especially if you think we’re all operating under real uncertainty about what AI is going to do to human life.

And decentralization follows from the same logic at the institutional level. If no person or committee is wise enough to determine the good for everyone, then we should be wary of a small number of actors hard-coding their anthropology into the substrate of society. Decentralization is valuable because it preserves room for people to try things, get them wrong, and leave, keeping mistakes from becoming total.

So to your question about China: yes, their AI strategy is coherent, explicit, and philosophical. But coherence purchased by foreclosing the capacity for self-correction is brittle in exactly the way that matters most right now, though that is still a bet, not a proof. I’m also skeptical that it preserves the standing of human beings as agents capable of judgment rather than increasingly treating them as objects of coordination. A framework can be philosophically serious and still be wrong about what a person is.

So I would not say Cosmos is advancing a one true final doctrine that every civilization must affirm. That would be too strong, and it would collapse into exactly the kind of totalizing move you are warning against. I would say instead that Cosmos is trying to defend the conditions under which free people can genuinely seek truth, make judgments, form commitments, and build different kinds of lives together, without having orthodoxy imposed on them by default.

asks:

Will the Cosmos Grants ever open again?

We’re planning to re-open these in the next 60 days. We’ve just had the demos from our latest batches of winners. Stay tuned for updates!

asks:

Assuming frontier large language models, together with their multimodal and agentic extensions, are trained to effective saturation on an exhaustive corpus that represents the totality of digitized human knowledge including all scientific publications, books, patents, archival records, cultural artifacts, and recorded conversations, will these systems be capable of transcending the statistical manifold of their training distribution to autonomously discover, validate, and iteratively expand novel knowledge beyond the current human frontier?

I’m not sure scientific knowledge is a kind of territory where data defines a bounded region that a model may or may not be able to venture beyond. The affirmative view of this picture strikes me as broadly empiricist insofar as knowledge comes from data, and scientists make discoveries by extrapolating beyond what they already know. Your specific examples, though, are actually the strongest case for the affirmative: theorem provers, simulators, multi-agent workflows, and verifiable rewards are exactly the kinds of feedback-rich settings where I would expect systems to extend the frontier.

But that is not the only picture of science, and I do not think it is the deepest one. Science also advances by reorganizing what researchers take to be meaningful in the first place: which anomalies matter, which questions are worth asking, and which explanations count as illuminating rather than merely predictive.

Systems can already generate novel candidate hypotheses, and in domains with strong automated verification, they may well extend the frontier. Formal mathematics looks especially promising, because conjecture can be paired with proof or disproof inside a relatively crisp evaluative architecture. In such cases, I expect AI systems to produce results that are genuinely new to humanity. Just this week a constellation of agents improved a math problem that’s been open since Newton (Kissing Number in dimension 11: 593 → 604). That is impressive. It is also, I think, a good example of the distinction I’m drawing: a real extension of an existing line of inquiry, but still closer to powerful normal science than to scientific revolution.

But that does not settle the larger question. There is a difference between producing novelty within an existing framework and generating a new framework altogether. A system may help prove a theorem, optimize a search, or identify that drug X affects disease Y, all without altering our understanding of why the problem is structured as it is. That is a real scientific contribution, but it is not reorganizing the conceptual landscape.

The harder question is whether these systems can exercise scientific judgment in the richer sense: whether they can tell which anomalies are significant, which inconsistencies are fertile, which explanations deepen understanding rather than merely extend prediction, and which questions are worth reorganizing inquiry around. That is a higher bar than novelty, and I am not yet convinced we know how to evaluate it well. Part of what makes this hard is that frameworks are underdetermined by data. The same body of results can often support multiple lines of inquiry, and judgment is what tells you which one is worth building a field around. That remains, to my mind, the deeper open question.

asks:

Most of the current work on ‘AI, collective epistemic structures and decision-making’ focuses on filling gaps: more participants, faster information exchange, more efficient decision-making. This will help with many problems, but certainly not with the most complex ones, because it just accelerates the practical execution of the same thought styles that led to the problems. Therefore: How can we use future AI to foster new thought styles that are currently not supported by our existing social structures?

Yes, I think the intuition that better collective decisions will emerge if we simply gather more data from more people more efficiently breaks down at the limit. That can improve performance within an existing paradigm, but it does much less when the paradigm itself is the problem.

What groups develop over time are not just bodies of knowledge, but epistemic constitutions: implicit rules about what counts as evidence, which questions are legitimate, who gets to propose, who gets to criticize, and on what terms. Mill saw part of this in his account of the tyranny of prevailing opinion and the epistemic importance of dissent. But the problem runs deeper than opinion alone. Entire institutions decide in advance what counts as serious thought.

So one promising use of AI would be to make those constitutions more visible. A good system might show a research community, an organization, or a polity where its methods systematically exclude certain questions, place some assumptions beyond criticism, or discount certain voices before the argument even begins. In medicine, for example, it might reveal a field that privileges what is easily measurable while sidelining patient testimony or long-horizon effects that do not fit the dominant method.

But diagnosis is only the beginning. I like this direction because the problem is often not that new thought styles do not exist. It is that they remain stranded at the margins because the reigning structures of legitimacy suppress them. And sometimes the deeper problem is that the social conditions required for a new thought style have not yet been built. New thought styles need protected spaces, alternative standards, and enough provisional legitimacy to develop before the dominant paradigm dismisses them. In that case, the most useful contribution AI could make to collective epistemics is not novelty on demand, but widening the space in which criticism, recombination, and intellectual minority formation can occur.

asks:

What is your definition of intelligence? When AI reaches ASI in the future, do you think it will be safe and aligned? As a species, what is our purpose after ASI world? How can thrive as a species?

I’d resist the standard definition of intelligence as raw problem-solving horsepower. For me, intelligence is the capacity to learn from reality, inquire into it well, and let it correct you.

Part of what I like about François Chollet’s work, and why ARC Prize has mattered, is the insistence that intelligence is not the same thing as accumulated skill. A system can look impressive because it has absorbed an enormous amount, or because the task has been made easy for it. The more interesting question is how much it can learn from limited experience, under real constraints, and still generalize well.

But I do not think that is enough on its own. Leslie Valiant’s idea of educability gets closer to the human picture (see his Cosmos Lecture from last year here). Human intelligence includes the capacity to learn from experience, receive instruction, integrate both, and apply them in new circumstances. What distinguishes the human mind is not only that it learns, but that it can be taught and formed.

And I would add one more layer. Drawing on HAI Lab director Philipp Koralus, I think reasoning is fundamentally question-directed. Minds are shaped by the questions they pursue. They go wrong through shallow questions, premature closure, and a failure to inquire far enough, just as much as through false conclusions. That matters for AI because a system can become very good at answering questions while still narrowing the range of questions humans ask, or rewarding closure where inquiry ought to stay open.

That is why I’m less interested in arguing about whether AI will count as “superintelligent” than in asking what it does to human intelligence. A system can be extraordinarily capable and still erode our capacity for inquiry, judgment, and self-government. That is the danger I worry about most.

On whether ASI will be safe and aligned: I do not assume that can be taken for granted. I would trust highly capable systems only to the extent that they remain corrigible, contestable, and embedded in institutions that preserve human judgment rather than replacing it. The problem is not just getting the objective right once. It is making sure people can still question, revise, and refuse the system’s guidance when it matters most.

As for human purpose after ASI, I do not think our purpose changes. If anything, it comes into clearer view. We are not here to compete with machines at speed or scale. We are here to exercise judgment, form character, build institutions, love particular people, and deliberate about the good. That last point matters more than it may sound. Love is not interchangeable, and responsibility is not abstract. A more powerful machine does not make our obligations to particular human beings less central. In a world of highly capable AI, those things become even more important.

asks:

Is it possible to carry on our learning from humanity up until now and jumpstart a new society with the help of AI, assuming that we can concentrate and utilize all the data we accumulated up until now? At what point does the need for a new constitution or a new world order arise and how do we know it?

In short, no.

Firstly, I don’t think “all the data we accumulated up until now” is the same thing as the total weight of human knowledge. Much of the knowledge that keeps a society functioning is tacit, dispersed, and unwritten. Some of it lives in practiced judgment: an ICU nurse sensing that a patient is about to crash before the monitor shows it. Some of it lives in inherited forms: the habits of trust, restraint, and association on which a free society depends, even when no one can fully specify them. As Michael Polanyi put it, we know more than we can tell.

More importantly, I’d push back on the idea that we can jumpstart a society at all. Societies aren’t machines that you design to a blueprint. Tocqueville saw this in the institutions of local self-government. Hayek saw it in the way social orders carry dispersed knowledge that no planner can gather in full. A free society is learned in practice through things like townships, juries, churches, and associations. Those are the ordinary disciplines by which people become capable of governing themselves.

The question is whether our institutions can still sustain a free people capable of self-government under new technological conditions. And that does leave open the question you raise about constitutional inadequacy: how do we know when inherited arrangements are no longer enough? I do not think there is a clean threshold. Usually the signs are visible first in practice, when institutions that once formed judgment begin producing passivity, dependence, or elite insulation instead.

When they cannot, the answer is not a tabula rasa redesign of “the new world order.” I would look to renewal through institution-building, and Benjamin Franklin is the example I keep returning to. He took an Enlightenment conviction — that access to knowledge should not remain under the custody of church, state, or a narrow elite — and embodied it in an institution. The subscription library made a philosophy of freedom socially real. That is why Franklin still matters to me here. He shows what it looks like to translate a philosophy into civic machinery. We need the AI-age equivalent: institutions that widen access to knowledge and judgment without concentrating them in a few hands. We need philosopher-builders in that spirit again.

asks:

How can we motivate our children to learn at school? Should we try to motivate them or find rather a way out of the system? (e.g. reading more classical books, rather than encouraging them to read what school nowadays gives?)

Young people have a natural longing to be seized by something greater than themselves. To be captivated. The promise of liberal education, going back to the Greeks, is that there are magnificent ways of living, and magnificent questions about how to live, and that encountering them through great minds and great books can awaken a desire that organizes everything else.

The disaster of modern education is that it has taught young people their longing is naive. That no book is really better than another, that no life is really higher than another, and that the hunger to be drawn upward by something extraordinary is itself a kind of error.

So I would say motivation is the right place to focus, but we should be precise about what we mean. There is a kind of motivation that is intrinsic: the eros I just described, the desire to encounter greatness because it calls to something real inside you. And there is extrinsic motivation: incentives, structure, well-designed systems that make it easier to do the work. Both matter. The best schools I’ve seen, including Alpha where my kids go, are serious about the extrinsic architecture. They’ve built an environment where children actually want to show up and work.

Extrinsic design clears the path, and then you have to light the fire. The fire is eros, and it’s fed by contact with things worthy of love: books, questions, lives, guides who still care about these things enough to take them seriously in front of children.

Whether that happens at school or at home is incidental. What matters is that a child sees adults who are genuinely stirred by ideas, who return to certain books not because they were assigned but because they can’t leave them alone. A six-year-old can learn a lot about what seriousness looks like by watching someone practice it.

asks:

The accelerationist world still seems to dominate the public narrative by communicating in everyday language on everyday platforms in a way that meets people where they actually are. Meanwhile, even the most accessible AI ethics content tends to assume familiarity with Mill, Tocqueville, or Heidegger. The philosopher-builder framing is compelling to people already inside the tent. How does Cosmos think about the people outside it? Is philosophical depth a feature for the community you’re building, or a barrier to the broader cultural shift you want to see?

The honest answer is that depth is the point. If we watered down the philosophy so we could meet everyone where they are, we’d be producing the same frictionless content you see elsewhere. Philosophical seriousness creates a negative selection gradient, and we want that. The people who do the reading are the people most likely to build something different.

But “depth” and “jargon” aren’t the same thing. A lot of AI ethics writing assumes you’ve already read Heidegger or whomever, which risks filtering out precisely the builders who might be transformed by reading him. I know this because I’ve made the mistake myself. When I started writing this Substack I leaned on more jargon than I needed to, and I’ve had to learn over time how to make the ideas more accessible without making them thinner.

The people outside the tent aren’t who you might think. I sold two companies and wrote a national AI strategy, and I couldn’t write what I believed on a single sheet of paper. There are a lot of capable builders out there who never had anyone hand them the books or sit with them through the hard parts. Cosmos partly exists because I was one of them. The audience for this is bigger than it looks.

Where I’d push back on your framing is the implicit suggestion that the accelerationists win because they’re more accessible. They have their own jargon. Try reading about negentropy, Kardashev III, and thermodynamic civilizational substrate for the first time. What they’ve done well is compress a real conceptual core into memes that travel. I respect that.

The challenge for us is that some ideas compress more easily than others. “Build faster” is more memeable than “cultivate judgment.” “Technology goes up” fits on a poster. “The conditions under which free people can exercise genuine choice require institutional renewal” does not.

This logic holds for political movements more generally: the larger the audience you try to build, the cruder the message has to become. The lowest common denominator wins by default, not because it’s right but because it compresses. I don’t think the answer is to compete on that terrain. I think it’s to make the longer argument compelling enough that people seek it out, and to be honest that not everyone will.

The harder truth is that we live in a culture of secondary orality where the long coherent essay is increasingly marginal. That’s a loss. It makes what we do at Cosmos more countercultural than it would have been fifty years ago, but it also makes it more necessary. The essay, the book, the salon: these are the forms where ideas actually get tested rather than just transmitted. We’re not going to stop producing them because the culture has moved on. If anything, the fact that sustained argument is now unusual is exactly why it matters.

asks:

For someone coming to AI without a technical background but with a strong interest in understanding its societal and philosophical implications, what foundational books or resources would you recommend?

Second, as a parent, I’m thinking about how to prepare my kids for a world where AI is increasingly embedded in everyday life. Beyond basic digital literacy, what kinds of skills, habits, or ways of thinking do you believe will matter most for the next generation? Are there age-appropriate tools or frameworks you’d recommend for introducing AI concepts early in a thoughtful, not just utilitarian, way?

Hi Emily :)

I’ll take the kids question first because it’s closer to my heart.

The risk I think about most is what I’ve called “autocomplete for life”: the possibility that AI systems will increasingly shape not just what our children do but how they deliberate about what’s worth doing. Each small delegation of judgment seems harmless. But together, they habituate a person away from self-governance and toward dependence. The question for parents is how you build resistance to that drift before your child is old enough to name it.

Our ancestors needed to know how to make bread. We need to know where to find the recipe. The next generation will need something different again: the capacity to think about how they think, in relation to systems that could do the thinking for them.

In our household, the main way we work on this is Socratic conversation. Arden and Pierce do weekly sessions with Michael Strong built entirely around questions. “What’s the difference between a bird and a plane?” “What does it mean for something to be alive?” “When mommy and daddy disagree, who is right? What about daddy vs. AI? What about AI vs. AI?” A child who has practiced working out what they believe, and who has had to think about whether to trust their own judgment or defer to an external authority, is better prepared for a world of algorithmic suggestion than a child who has learned to code.

I also want my kids to be entrepreneurial. When America was founded, around 80% of free workers were self-employed on farms or in small crafts. Today that number is about 10%. We became a society of employees, and something atrophied. As the economy changes again, the ability to know yourself, act on what you believe, and build something from that conviction will matter more than any technical skill we could teach them now.

On resources for someone coming to AI without a technical background: I’d start with the question of what AI does to us rather than how AI works. A couple of recent pieces that I’d recommend are ’s Musings on Self-Recursive Improvement and ’s What Will Be Scarce. For ongoing reading, , , and regularly write about AI and society. and have a wider aperture and I often find them thought-provoking. For anyone interested in AI’s effects on democracy and self-governance, Harvey Mansfield’s Tocqueville: A Very Short Introduction is the best ~100 pages you could spend. Tocqueville saw the drift toward comfortable dependence coming two centuries ago. The application to AI is left to the reader, but it isn’t hard to find.

asks:

My sense is that the vision animating Cosmos has deep predecessors not just in classical philosophy but, in my impression, religious eschatology. Teilhard de Chardin’s Omega Point or Augustine’s City of God, and even secular variants like Condorcet’s perfectibilism all share your orientation toward civilizational-scale transformation in service of human flourishing.

More explicitly, your pillars of reason, autonomy, and decentralization also echo the long Aristotelian and classical liberal tradition from Mill to Tocqueville.

So, what does Cosmos contribute that is genuinely novel in its normative architecture, rather than a restatement of those traditions in the presence of AI? And if it is largely a restatement, is that a problem?

I think you’re closer to the mark with some of these influences than others.

Teilhard, Augustine, Condorcet: I share their impulse toward civilizational-scale thinking, and I take it seriously. But for all their differences, they are ultimately teleological writers. They saw history as the unfolding of a determined, directional arc. At Cosmos, we want to keep the conditions open that allow people to find their own path. We’re not about to get into eschatology.

You are, of course, completely right about Aristotle, Mill, and Tocqueville, and we regularly acknowledge our intellectual debt to them. I don’t think the pillars need to be new to be worth defending, and I’d be suspicious of anyone claiming to have invented a wholly new account of human flourishing in 2026.

For me, the interesting question isn’t whether Cosmos has discovered a value nobody thought of before. Instead, it’s whether an old set of commitments can survive as a living practice. Mill didn’t have to ask whether the harm principle could be encoded in a model’s training objective. Tocqueville didn’t have to think about what decentralization looks like when the substrate is compute rather than townships, when the everyday infrastructure of life anticipates your choices rather than forcing you to deliberate, associate, and decide alongside your neighbors. When your community is mediated by algorithmic curation and your civic life is shaped by systems you never consented to and cannot inspect, the Tocquevillian question of how free people learn to govern themselves together doesn’t disappear. It becomes harder, and the institutional forms it requires don’t exist yet.

That’s where your last point lands, and I think it’s the right one. The proudest achievement of the eighteenth century was the translation of philosophy into law: Enlightenment commitments about liberty, consent, and the rights of individuals became encoded in constitutions and legal systems that gave them institutional force. The challenge of the twenty-first century is the translation of philosophy into code. The commitments are old. The work of making them operative in the infrastructure that actually governs daily life is new, and it is the work Cosmos exists to do.

But I wouldn’t call what we’re doing a restatement. Restatement is what you do in a seminar. Institutional embodiment is what you do when you think the ideas actually matter and must be operative in the AI age.

asks:

What do you think of the prospects for a stable, left-right coalition on AI in favor of sensible regulation and general cautious optimism that includes religious conservatives and secular social democrats? Or will this get polarized across political lines like everything else?

On the coalition point, I can already see signs of this. Religious conservatives and secular social democrats agree on little, but they intuitively grasp some things that many accelerationists don’t: that people are formed by their communities, work and dignity are connected, and that we shouldn’t try to optimize society into passivity. I’d also throw old school liberals into that coalition too. In the coming years, I’m sure there’ll be scope for productive, broad-based conversations about kids, loneliness, work, and communities.

Where I’d push back is the idea that any future coalition should coalesce around “sensible regulation.” I don’t think regulation is the best tool for addressing most of these concerns. Treating it as the default is how you end up with something like the EU AI Act, a classic example of doctor-induced illness. It created a compliance moat that only the largest companies can afford to cross, while doing essentially nothing to address the risks it was supposed to mitigate.

The more productive ground is further upstream. What are we building? What do we fund? What should we teach? What institutions do we need to form? A coalition focused on those questions would look less like a regulatory body and more like a network of individuals doing the building, teaching, and funding that no regulation can mandate.

I’m less worried about polarization acting as an obstacle here. Much of this work sits outside electoral politics at the moment, and as far as I’m concerned the longer that remains the case the better. Partisan dynamics reward exactly the kind of simplification that makes these questions worse. The moment AI becomes a left-right issue, the entire conversation becomes about how much to regulate, and the question of what to build for never gets asked.

asks:

Here is my question: Your three pillars (truth-seeking, autonomy, and decentralisation) are compelling at the individual level. I am curious how you think about them when the actors are states rather than individuals. The US-China AI dynamic, for instance, seems to run against all three: opacity rather than truth-seeking, control rather than autonomy, and concentration rather than decentralisation. Does Cosmos’s framework extend to the question of how countries could potentially cooperate on AI, or does that require a different philosophical foundation entirely?

Thanks Alina, great question.

The pillars were designed with individuals and institutions in mind, so extending them to the state level requires real philosophical work.

Fichte took the Kantian account of individual autonomy and argued that it applied to nations: a people that cannot determine its own form of life is unfree in the same sense an individual under tutelage is unfree. The autonomy pillar, taken seriously, has a national analogue. So does truth-seeking: a polity that can’t inquire openly into its own condition is in the same trap as a closed mind. And so does decentralization: a world of self-governing peoples is the international expression of the same instinct that makes you wary of concentrated power inside a country.

But Fichte also shows you what happens when you scale autonomy alone. His attempt to extend individual self-determination to the collective ended in arguments for the unique world-historical mission of the German nation, an autarkic closed state, and the exclusion of those who didn’t fit the national community. The lesson isn’t just “be careful.” It’s that the three pillars need to travel together. Autonomy without truth-seeking becomes self-righteousness. Autonomy without decentralization becomes domination. What checks national self-determination is the same thing that checks individual self-determination: openness to correction and the refusal to concentrate power beyond what can be held accountable.

On US-China, the goal isn’t a single global regime that imposes one model of AI governance on everyone, because that would violate the decentralization commitment at the international scale. The better question is: what conditions allow distinct political communities to develop AI in line with their own forms of life without crushing each other in the process?

And what happens when a community’s “form of life” involves suppressing the autonomy of its own citizens? The pillars can come into tension here. Respect for national self-determination and respect for individual autonomy pull in opposite directions.

This is where Tocqueville matters most. The meaningful unit of self-government is rarely the nation-state on its own. It’s the dense layer of associations, communities, firms, religious groups, and local institutions that sit between the individual and the state. Any serious thinking about international AI governance has to make room for those middle layers. Tocqueville saw that democratic freedom doesn’t live in declarations from the center, but in the practice of self-government at the local and associational level.

asks:

Can you set up a path for crowdfunding projects or contests to realize ideas that the Cosmos Institute seeds?

Interesting. Not something we’ve considered, but we’ll think about whether there’s a model that works for us.

asks:

Do you plan to have online cohorts?

No plans right now, but we may consider it in the future!

asks:

I see all these wonderful essays and people doing great work. Honestly, I would love to interact with the community + become part of it in some capacity. Is there a channel where people who are interested in the ideas that Cosmos is working towards but not necessarily are academics or builders can communicate with the fellows and the team?

Not right now, but we are thinking about whether there’s something we can do here!

asks:

I’m persuaded by the concern about cognitive risks and the need for “AI for epistemics,” “deliberative AI,” etc. Do you know of organizations developing benchmarks around the goal of bolstering critical thinking and improving epistemic processes and outcomes?

We have some grant projects that have focused on this. Two that come to mind are DeliberationBench, which assesses AI persuasion in comparison with diverse human discussion, and Priori, a tool that surfaces hidden assumptions when you are interacting with an AI model. Two of our grantees (Steven Molotnikov and Cathy Fang) are running a research study on how Priori and related human oversight interfaces work in practice.

I think there is a wave of energy in this area. Various orgs are thinking more about AI for Human Reasoning (with Future of Life Foundation funding work in this area, Forethought writing about it, and Elicit working on directly in the for-profit space). Also anecdotally I hear researchers thinking more about ideas like “epistemic security” or “cognitive security” or “cognitive sovereignty” as well as ways to improve information environments without restricting speech and expression.

I share your enthusiasm for more work in this area – both on benchmarking but also technology that better enables open contestation of ideas (inspired by classical liberal premises, and Mill’s ideas on this). If readers are working on this please do reach out!

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

Thank you for all your responses – you can now see the responses to the AMA here.

This week, we crossed 20,000 subscribers on Substack. Thank you to everyone who has read, shared, and engaged with our work.

We’ve written about Claude Boys to Coasean bargaining to the perils of liberal nudging. Reading the comments has often been as rewarding as writing the posts. To mark the milestone, I’ll be answering your questions on Wednesday April 15.

Drop your question in the comments below and upvote the ones you want answered. I’ll start responding next week and I’ll try to take as many as I can.

There are a few things I’ve been thinking about that we haven’t written about yet. This seems like the right place to start.

Ask about Cosmos, human autonomy, AI x philosophy, or what people in our network are building. Especially questions that are hard, that relate to how we approach AI as builders, or that challenge our assumptions.

- Brendan

Leave a comment

In the autumn of 1809, Prussia was a country that no longer knew what it was for. Three years earlier, Napoleon had destroyed its army in an afternoon and walked into Berlin without resistance. The king fled. Half the territory was gone, the treasury empty. French soldiers were still garrisoned in the capital.

As Prussia began rebuilding from the wreckage, most people assumed it needed more officers, administrators, and engineers. People who could do things. The task of designing the new system of education fell to a thirty-two-year-old diplomat named Wilhelm von Humboldt. He gave them something else entirely.

In a series of memoranda written over the next year, he laid out a vision for a new university in Berlin organized around Bildung. The word has no English equivalent. “Education” is too narrow, “self-improvement” too thin. “Formation” gets closest but still misses its moral weight.

Humboldt’s Bildung means the free, harmonious development of a human being’s powers into a complete and consistent whole, through encounter with the world in its variety and resistance.

Mill, who took the idea from Humboldt, put it more simply: a human being is more like a tree than a steam engine.

Humboldt proposed a university where professors and students would be joined in the pursuit of knowledge, unconstrained by political demands. In a defeated nation hungry for officers and administrators, he was arguing for formation before function.

The ideal of the modern research university, with its union of teaching and inquiry, its seminar culture, and its commitment to academic freedom, descends from what Humboldt designed in those desperate months.

Today, the culture of Bildung that animated the university survives only at the margins, sustained by people stubborn enough to work against the grain of the institutions around them. Credentialism twisted the university into a vendor of certificates, and the formation of the student as a complete human being came to seem anachronistic. The cathedrals remain, but not the faith.

We are in a moment that rhymes with Humboldt’s own. Technological pressure is once again pushing education toward the practical. The evidence of institutional collapse is everywhere: flagship universities slashing PhD admissions, hundreds of degree programs have been eliminated, dozens of small colleges have closed, and a decline in the college-age population still lies ahead.

The loudest responses to the crisis have come from outside the university. Alex Karp tells young people to skip college and learn a trade. Marc Andreessen argues the university is a credentialing middleman and should be disintermediated. Both are right that the university is failing. But if the answer to a broken formation system is to skip formation altogether, you have already conceded that education is justified only by utility.

Neither is asking the question Humboldt asked: What is a human being, that education should serve it?

Subscribe now

The Tree and the Trap

Before Humboldt, the Prussian system assigned students to a vocation, trained them in it, and delivered them into their function. Even on those terms, Bildung wins. Broadly cultivated judgment produces better doctors and better engineers. Technical skill on a foundation of general cultivation is more resilient and more humane than technical skill resting on nothing.

But if that were the whole justification, Bildung would be nothing more than a roundabout way of minting skilled professionals. A pedagogy can be justified by output. Bildung cannot.

A tree does not exist in order to produce lumber. You can make lumber from it, and good lumber is nothing to sneer at. But if you look at a tree and see only lumber, you have missed what is standing in front of you. Something is growing there under its own power, toward its own form, and the growing is not a means to some further end.

Humboldt’s claim about human beings is the same shape. A person is a self-developing being whose worth is not exhausted by function.

Bildung braids together the Kantian claim that no person is merely a means, the Greek ideal of harmonious excellence across mind and body and character, and the Romantic conviction that we develop through encounter with what resists us.

You have had the experience even if you never had a word for it. Real engagement with something that has its own demands—a hard problem, a serious book, a gifted teacher—changes who you are. You could not have planned the person you became.

Such formation is not the property of any particular university department. This is not a “save the humanities” argument. A coder who, after tackling a hard systems-design problem, comes out thinking differently about complexity, tradeoffs, and the limits of formal reasoning has undergone a kind of Bildung—but only if the encounter changed who they are, not just what they can do.

The resistance to all this is understandable.

In a world where your economic value can evaporate overnight, “Become a whole person” sounds like advice from someone who has never worried about next month’s rent. The utilitarian case for education has the force of necessity behind it. For millions of people, making yourself useful is what responsibility to their families demands.

Yet if the response to being replaceable is always to train for a different function, you have entered a race you structurally cannot win. The principle that makes your education valuable is the same principle that makes you disposable the moment the function migrates.

The scramble into computer science was an early sign of the trap: students rushed toward the field that seemed safest, and then AI began destabilizing the very functions it trained them to perform. The flight to function looks rational from inside it; that is what makes it a trap.

Solitude and Freedom

Bildung cannot be specified in advance. Once you define what the formed person looks like, you have replaced formation with training. So how do you build institutions around it?

Humboldt’s solution was to design an environment rather than a curriculum. He based the University of Berlin on two principles: solitude and freedom, though they meant something precise in his hands. The university would not answer to the demand for immediate use and inquiry would follow the question wherever it led, unconstrained by predetermined ends.

The undoing of that design came in two phases.

First, a slow hollowing. The German research model crossed the Atlantic when Johns Hopkins was founded in 1876 on Humboldt’s model. But disciplines hardened into guilds and career tracks, postwar federal funding built research enterprises increasingly detached from teaching, and mass enrollment turned the degree into a sorting mechanism. The university no longer had a single organizing purpose, and no one was left responsible for the student as a whole person. By the mid-twentieth century, the university president Clark Kerr could describe the resulting “multiversity” as a collection of unrelated enterprises held together by a common heating plant.

Then, a rejection of the cure. Recently, at the University of Tulsa, the philosopher Jennifer Frey built a Great Books honors college from scratch. It was a serious formation program inside an institution organized around credentialing. The university removed her, restructured the program, and replaced the deanship with a directorship.

Back to Scholé

But even if the institutions had held, formation requires something they could never secure at scale: time.

Aristotle called it scholé. Humboldt had a related word for it: Muße. Both named a kind of structured freedom for the work of becoming, and for most of history that freedom was radically exclusive.

Aristotle could imagine the highest forms of human flourishing only for those relieved of labor by wealth and the work of subordinates and slaves. The good life required freedom from necessity, and in his world only a few could have it. But in the first book of the Politics he imagined something stranger: that if shuttles could weave by themselves and picks could play the lyre, craftsmen would need no subordinates and masters would need no slaves. The “self-guided machine” would mean that the material basis for leisure no longer depended on the unfreedom of others. It is one of the oldest thought experiments in Western philosophy, and we are now enacting it.

Aristotle did not celebrate the prospect. He understood that freedom from necessity does not automatically yield the pursuits that make such freedom worth having. In his account, those with wealth and leisure often turned to unlimited acquisition or bodily gratification rather than to the activities that justify leisure in the first place.

With AI, we are building something like self-guided machines. Whether these systems liberate or merely displace is not settled. But the possibility of leisure at scale is real enough to become a serious question.

If AI can compress parts of instruction, it may deepen learning where it is used and clear ground for formation where it gives time back. But only if it preserves productive struggle rather than bypassing it.

The alternative is already visible: autocomplete for life. Not just help with expression, but the slow outsourcing of judgment itself. That is Bildung’s antithesis.

Worse, the same technological society enabling leisure is also shaping the desires of the people who receive it. If our dispositions have already been trained toward optimization and outsourced judgment, the freed hours may arrive in hands that no longer know what to do with them.

For most of history, the conditions of formation were reserved for the few. The capacity for it was not. If scholé at scale is now possible, refusing to pursue it ratifies a world in which full human development remains the privilege of those who can afford time.

In the Afterglow

Bildung is, at its normative core, anti-servility: the effort to form people who cannot be reduced to instruments of external authority, whether state, market, or algorithm.

The people who use AI well right now are drawing on judgment they formed before these tools became ambient. They know when to trust an LLM’s output and when to push back because they learned to read, argue, and sustain attention under conditions in which those acts were not so easily outsourced. They bring something the tool cannot supply.

Nietzsche thought secular liberals were living off the moral capital of a Christianity they had officially abandoned: inherited capacities that could persist for a time even after the culture that formed them had ceased to renew them. The kind of judgment this essay is defending may be a similar afterglow, formed in a world before AI mediated everything.

Without that judgment, you get agency without autonomy.

If the capacities required for non-servile life in an AI world were all formed in a pre-AI world, what happens when that formation stops? You can live on an inheritance for a while. You cannot educate a civilization on inherited judgment forever.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

The university is under siege.

Harvard, sitting on a $57 billion endowment, slashed 75 percent of its science and 60 percent of its humanities PhD admissions. Indiana mandated the elimination of over 400 degree programs at its public universities. Classics, comparative literature, and foreign languages at the state flagship are gone. At the University of Tulsa, a philosopher built a Great Books honors college from scratch, grew enrolment 500 percent, and attracted a quarter of each freshman class to a curriculum running from Homer to Arendt. The university removed her.

Sixty-four small colleges have closed since 2020, and the demographic cliff, a 13 percent decline in college-age students projected through 2041, hasn’t even started.

Meanwhile, 52 percent of recent graduates are working jobs that don’t require the degree that they borrowed $30,000 to get. Palantir CEO Alex Karp is telling young people to skip elite colleges entirely, saying the only paths left are skilled trades or neurodivergence.

AI will accelerate this. The skills the degree was supposed to impart – researching, analyzing, drafting, coding – are increasingly things a machine can do.

As the old institutions break down, a new generation of educational founders is running experiments. New schools are emerging, each making a different bet on two questions that anyone building an educational institution has to answer:

• The first question is about purpose. Do you focus on specific, trainable skills, or on a broader, if more intangible formation, which includes the development of judgment, attention, and moral seriousness?

• The second question is about technology. Do you build AI into the core of the educational experience or keep it out because the difficulty is where the learning happens?

These two axes produce a landscape.

At one end, programs like Gauntlet train elite AI engineers in ten weeks: 80-100 hours a week of building, a guaranteed $200K job offer at the end, costs paid entirely by employers.

At the other, institutions like St. John’s College run intensive liberal arts programs built on sustained attention and close reading, with no AI in sight.

Arizona State has embedded AI from coursework to advising and has struck partnerships with a number of high-profile AI companies, but the core teaching model hasn’t changed.

The University of Austin has proposed splitting the day between device-free seminars and intensive AI work.

Meanwhile, the bulk of the higher-education system is either banning AI from the classroom or pretending that it doesn’t exist.

These questions are not new. They go back to the founding of the first American colleges and European research universities.

In partnership with Liberty Fund, Cosmos held a seminar to think through them with a group of founders, scholars, and institution-builders. The participants included philosophers from UT Austin and Ohio State, researchers from MIT Media Lab, RAND, and leading AI labs, education policymakers from the Texas Higher Education Coordinating Board, and builders from Fractal Tech and Alpha School.

The reading list we assembled traces these tensions to their origins: from the earliest American proposals for public education through the founding documents of the research university, the mid-century debates about liberal learning, and the first serious writing about what computers might do to the relationship between learner and knowledge.

Subscribe now

Session I: The Promise and Crisis of the University

We are not the first to ask what the university is for.

This session combined five voices, spanning three centuries. Franklin wants a practical institution for a practical republic. Humboldt argues that the university exists for the development of the whole person, not the production of useful professionals. Oakeshott insists that education is how a human being becomes one. Bloom diagnoses what happens when that inheritance is abandoned. And Caplan asks whether the whole enterprise is a $240,000 receipt that signals you can show up on time.

If Caplan is even partly right, the reformers need to explain what they are offering that a credential cannot capture. If Humboldt is right, the reformers need to explain why two centuries of institutions built on his model ended up producing exactly the credentialism he warned against.

• Benjamin Franklin, Proposals Relating to the Education of Youth in Pennsilvania. (link)

• Wilhelm von Humboldt, “On the Internal and External Organization of the Higher Academic Institutions in Berlin”. (link)

• Michael Oakeshott, The Voice of Liberal Learning, Introduction.

• Allan Bloom, “Our Listless Universities.” (link)

• Bryan Caplan, The Case Against Education, Chapter 1, “The Magic of Education.”

Session II: What Formation Requires

If the first session asked what the university is for, the second focused on what actually changes a person. Weil makes the strongest claim: attention is all you need. Klein argues that liberal education requires putting your own opinions genuinely at risk, and that the resistance to doing so is rooted in something specifically human: we build our identities on what we think we know. Gadamer argues that practical wisdom cannot be taught as a curriculum; it grows out of ethos, the character already formed by living in a particular community.

But what if the boundaries of the person aren’t where we think they are? Clark and Chalmers ask whether cognition is even confined to the skull. If the mind extends into the tools and environments we think with, what counts as “the student”? Shanahan complicates the picture further: LLMs create a compelling illusion of understanding, but they are fundamentally unlike us. If formation requires genuine encounter with other minds, what happens when the most available interlocutor is a machine?

• Simone Weil, Waiting for God, “Reflections on the Right Use of School Studies with a View to the Love of God.”

• Jacob Klein, “The Idea of Liberal Education” in The Goals of Higher Education, ed. W.D. Weatherford, Jr.

• Hans-Georg Gadamer, “The Socratic Question and Aristotle”, Continental Philosophy Review.

• Andy Clark and David Chalmers, “The Extended Mind”, Analysis. (link)

• Murray Shanahan, “Talking about Large Language Models”, Communications of the ACM. (link)

Session III: The Institutional Question

Good ideas about education are easy. Institutions that embody them are hard. This session focused on the gap. Jefferson proposes education as grafting: implanting something new onto the wild stock of human nature. Newman argues that the value of a university lies in the sheer density of its intellectual community, that students gain from living among those who represent the whole circle of knowledge even if they can never study it all.

Karlsson brings this into the present. His argument is sobering: AI tutors will be held back by culture, rather than technology. Motivated learners embedded in high-growth communities will use AI to accelerate, while everyone else will use it to avoid difficulty. The real challenge will be building strong cultural norms against taking the path of least resistance.

• Thomas Jefferson, “Draft of the Rockfish Gap Report of the University of Virginia.” (link)

• John Henry Newman, The Idea of a University, Discourse 5, “Knowledge its Own End.” (link)

• Henrik Karlsson, “AI tutors will be held back by culture.” (link)

Session IV: Education and the Machine

This session turned directly to the question of what technology does to thinking. Licklider, writing in 1960, imagined a partnership: machines handle the routine cognitive work, freeing humans for insight and decision. Papert envisaged a more profound interplay. He argued that when a child learns to program, the relationship between learner and knowledge is fundamentally transformed. The child is no longer receiving explanations but building things, and the building changes how she thinks.

Matuschak and Nielsen pick up the thread sixty years later and find it frayed. The pioneering visions of tools for thought are treated as nostalgia in technology circles. There is little determined effort to build tools that genuinely transform how people understand. The discussion focused on whether the current wave of AI represented a chance to revive the Licklider-Papert vision or whether tools for thought were likely to become an even more distant memory.

• JCR Licklider, “Man-Computer Symbiosis,” IRE Transactions on Human Factors in Electronics.

• Seymour Papert, Mindstorms: Children, Computers, and Powerful Ideas, Ch. 1.

• Andy Matuschak and Michael Nielsen, “How can we develop transformative tools for thought?” (link)

Session V: Building the New Academy

The final session focused on what comes next. Flexner’s defense of useless knowledge, written from the Institute for Advanced Study, is the purest case for protecting inquiry from the demand for application. Bloom’s portrait of Hutchins at UChicago shows what it looks like when someone actually tries to build an institution around these convictions, and the political will it requires. Simondon challenges the premise that technology and culture are opposed, arguing that the hostility between them is a sign of ignorance.

Engelbart closes the list with a constraint AI is testing: “The entire effect of an individual on the world stems essentially from what he can transmit to the world through his limited motor channels.” As those channels widen, the bottleneck shifts to whether you have something worth transmitting.

• Alexander Flexner, “The Usefulness of Useless Knowledge,” Harper’s Magazine. (link)

• Allan Bloom, “Hutchins’s Idea of a University,” Times Literary Supplement.

• Gilbert Simondon, On the Mode of Existence of Technical Objects, “Introduction.”

• Douglas C. Engelbart, “Augmenting Human Intellect: A Conceptual Framework,” SRI Summary Report AFOSR-3223.

Is there an institution that protects the conditions for deep formation and teaches students to master the tools that will define their century?

Subscribe now

Cosmos Institute is the Academy for Philosopher-Builders, with programs, grants, events, and fellowships for those building AI for human flourishing

“Artificial Intelligence will revolutionize science.” The phrase is usually invoked by researchers when discussing the promise of thinking machines to help us understand the natural world. In the early 2020s, I subscribed to, and was perhaps even evangelizing for, the role of AI in scientific research. Like many of my colleagues, I was gripped by a new tool that promised so much.

More recently my view has changed. I still believe that AI will be fantastically useful, but not necessarily in the way we think. Discovery, after all, is not the same as understanding. As scientists, we need to engage with the process of inquiry to truly make sense of what we learn about the world. Only then can we understand.

The Long History of “AI for Science”

In the summer of 1953, physicists Enrico Fermi, John Pasta, and mathematicians Stanislaw Ulam and Mary Tsingou ran the first “numerical experiment” on MANIAC, one of the early electronic computers built at the Los Alamos National Laboratory after World War II. Computers were new and scientists were excited to explore their use in solving research problems like simulating the dynamics of atoms and molecules, studying tumor cell populations and investigating numerically fluid dynamics. They even created the first documented chess-playing program that defeated a human in the game.

This group decided to model a chain of oscillators (identical masses connected by springs), add a small nonlinearity (a quadratic or cubic term), and see what would happen. The expectation was that the system would reach equilibrium (common sense would predict that there is some wiggling around, but ultimately energy ends up equally distributed throughout the springs). But the results were surprising. Instead, the system showed a recurrent behavior where the energy spreads throughout the chain, then comes back, before spreading out again. This observation sparked interest in the study of nonlinear systems that would later lead, among other things, to the discovery of solitons (waves that travel freely preserving their shape) and the development of chaos theory in the 1960s-1970s.

If one was to assign a birthyear to computational physics, that honorific should go to 1953 where Fermi and colleagues made for the first time an unexpected discovery through a purely computational approach. Computers had been used for physics and astronomy calculations before – for example, in the 1930s mechanical IBM accounting machines were modified to solve the differential equations of planetary motion by numerical integration – and had played a key role in the development of the atomic bomb during the Manhattan project. In the following decades computers would transform scientific research. Today, there are almost no advances in physics that are not enabled by some aspect of computer simulation.

Three years after the “first numerical experiment” the term artificial intelligence was coined and a first attempt was made to use automated reasoning to prove mathematical theorems through the Logic Theorist system in 1956. The first AI “expert” system, DENDRAL, was created in 1965. Combining a knowledge base with a reasoning engine, it was capable of determining the molecular structure of a compound from its mass spectra. Here, “expert” refers to a kind of AI system that encoded domain knowledge from human experts as rules and applied them through an inference engine.

Other examples include LHASA, a program designed in 1972 to discover sequences of reactions to synthesize a molecule; and the Automated Mathematician in 1977, a heuristic-based program designed to discover new mathematical concepts and theorems; and other expert systems like MYCIN (to identify bacteria) and PROSPECTOR (for mineral analysis). In the late 1980s and early 1990s, artificial neural networks were used to tackle problems in particle physics, astrophysics, and –– in a glimpse of what was to come –– predicting protein structure. By the 2000s, AI methods were in use in many areas of physics and even assisted data analysis leading to the discovery of the Higgs boson particle.

In the 2010s and 2020s, scientists began to use a wave of powerful neural network-based tools to solve research problems. Some of these advances, such as Google DeepMind’s AlphaFold protein prediction system, made headlines around the world. Others were important but less glamorous. Machine-learned interatomic potentials, for example, significantly speed up materials and chemistry simulations yet remain unknown outside the research community.

But if “AI for science” has a long history, why has it only recently started to make the news? In his 2003 book, Douglas S. Robertson coined the term “phase change” to describe a radical change that an instrument makes possible compared to the prior state of the art. AI may be a phase change in how we do science, but not only because of powerful individual tools. It is their extreme accessibility, from large language models to the AlphaFold Protein Structure Database, that sees advances in one field become instruments of exploration in others.

Epistemic Enhancers

Scientific instruments provide extensions to humans’ senses. Telescopes and microscopes allowed us to see the far and the small. This type of enhancement is known as extrapolation. Another type of epistemic support is conversion, that is, transforming one modality into another (like sound into a visual image). Finally, we have those instruments that extend our capacities by giving access to phenomena that elude our senses, such as detecting radiation or magnetic fields. This is known as augmentation. These three moves – extrapolation, conversion, and augmentation – are all types of epistemic enhancers.

A central motivation behind scientific practice is understanding. But what is scientific understanding? The 2009 book Scientific Understanding edited by Henk W. de Regt, Sabina Leonelli, Kai Eigner suggests that there is no universal definition of scientific understanding. Understanding in physics differs from understanding in biology or engineering. Even within my own discipline, physics, what we mean by understanding is not straightforward. While clearly related to the explanation of a phenomenon, understanding is not precisely the same thing. There is a difference between understanding the phenomena and understanding the theories or models that explain that phenomenon.

A prerequisite for understanding is discovery insofar as one cannot understand a phenomenon that has not been observed. Discovery is the process or product of successful scientific inquiry. Objects of discovery are things, events, processes, causes, and properties as well as theories and hypotheses and their features. The 1987 book Scientific Discovery proposed that discovery in science can be broken down into problem solving tasks and therefore can be automated with computers.

The authors of the book built four types of programs to look for quantitative or qualitative laws and structural models. Then they used them individually and combined them to rediscover many laws in physics and chemistry. While the authors recognized that there is no unique process that accounts for scientific discovery, they showed that most of these can be cast as problem solving tasks that can be tackled with heuristics. Their method, based on tree searches, was general in theory but limited in practice. This was because the combinatorial explosion of possible paths made them intractable for problems beyond a modest scale.

Both traditional numerical methods and modern AI tools enhance epistemic extrapolation and conversion. The application of such methods represents a difference in kind as well as magnitude for scientific practice. As Paul Humphreys put it: “This extrapolation of our computational abilities takes us to a region where the quantitatively different becomes the qualitatively different.” This is because these simulations cannot be carried out in practice except in regions of computational speed far beyond the capacities of humans. This dual articulation of the quantitative and the qualitative could be taken as an argument for why modern AI tools ought to usher in a new way of doing science.

In the 1960s mathematicians Bryan John Birch and Peter Swinnerton-Dyer used the EDSAC-2 computer at the University of Cambridge Computer Laboratory to run numerical calculations that allowed them to state a conjecture about the set of rational solutions to equations defining an elliptic curve. This is now known as Birch and Swinnerton-Dyer (BSD) conjecture and is one of the seven Millennium Prize Problems. The computer helped them explore an abstract space so they could formulate the conjecture. This computer-assisted discovery falls in the final category of epistemic enhancers mentioned above: augmentation.

If the formulation of the BSD conjecture is a weaker example of augmentation, more recently AI methods helped mathematicians make a breakthrough towards solving it. Using machine learning, mathematicians discovered unexpected oscillatory patterns in the parameter space key to BSD (of hundreds of dimensions) which led to the “murmuration” conjectures. Progress has also been made towards solving another Millennium Prize Problem as AI tools were used to find potential singularities in the Navier-Stokes equations.

Although automated theorem proving programs have been around for 80 years, these examples do suggest that we are witnessing, in Robertson’s terms, a phase change. The quantitatively different becomes the qualitatively different in a more interesting way. It’s speed, yes, but it’s also breadth. Take AlphaFold and the AlphaFold Protein Structure Database it made possible. What makes it revolutionary is not only the improved accuracy of the protein structure prediction, but also its usability by scientists from many fields (who can access hundreds of millions of protein structure predictions).

We are yet to see the full extent of what AI tools can do for scientific research, but one differentiator between them and traditional scientific instruments, including computer simulation, is that they combine epistemic enhancers to explore vast dimensions of information. Remember DENDRAL, the early AI system determining the molecular structure of a compound from its mass spectra? Now, imagine the power of its modern incarnation which has access to several databases of molecular spectra, can cross-check with published scientific literature, and has the ability to write code to perform additional calculations if needed.

Or consider a powerful telescope that discovers a new exoplanet. Here an AI tool analyzes the spectra, determines the likely composition of the exoplanet atmosphere, compares it with the existing recorded information, runs simulations with the known and inferred parameters to better characterize the exoplanet and adds it to the catalog. All of these individual steps are already done separately and for each the use of AI tools can be seen as an evolutionary improvement. But the combination of these improvements, working together across different sources of information is potentially revolutionary.

If I’m permitted to add a new epistemic enhancer category for AI as an instrument the most appropriate would be integration (as in synthesizing across large amounts of very different types of information). When a tool synthesizes across hundreds of heterogeneous sources simultaneously, the inferential path from evidence to conclusion can become challenging for the scientist receiving the result to trace. The more dimensions integration spans, the wider this gap between discovery and reconstruction becomes. Where earlier epistemic enhancers extended what scientists could see or calculate, integration may increasingly determine what they conclude.

Our Role as Scientists

At the end of his book, Humphreys argued that computer simulation brings a “shift of emphasis in the scientific enterprise away from humans.” By the 1980s, researchers already knew that, at least in part, scientific discovery and problem generation could be successfully automated. Modern AI methods can do the same thing, only much faster and much better. I regularly hear from colleagues that in the coming years AI will take over many of the tasks associated with scientific inquiry.

Use of AI will likely mean more discoveries overall, but to discover is not to understand. Henk de Regt argued that: “scientists achieve understanding of phenomena by basing their explanations on intelligible theories. The intelligibility of theories is related to scientists’ abilities: theories are intelligible if scientists have the skills to use those theories in fruitful ways.” Quantum mechanics is an example of a very effective, yet not particularly intelligible theory, that despite the practical successes still eludes physicists in its interpretation. If AI tools can optimize discovery and generate explanations, they could perhaps also help produce more intelligible theories.

Yet intelligibility is not a property of theories themselves. As de Regt suggests, it depends on the capacities of the scientists who use them. A theory becomes intelligible only when scientists acquire the skills to explore its implications and apply it fruitfully. Even if machines make discoveries and generate explanations, this kind of understanding still depends on the participation of human investigators. Put differently: scientists need science just as science needs scientists.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

In 1962, Victor Glushkov pitched the Soviet authorities on a nationwide cybernetics network to solve the oldest problem in socialist economics: how to allocate resources without private property and market prices. Washington was alarmed enough for the CIA to create a special taskforce.

Just as Soviet industrialization was giving way to stagnation, the first serious computers gave the planned economy a shot in the arm. The central planners believed that they’d struggled to process information fast enough, so increased computing power promised a solution.

In 1969, the Polish economist Oskar Lange described the market as “a computing device of the pre-electronic age,” suggesting we could put “simultaneous equations on an electronic computer and obtain the solution in less than half a second.” In 1971, Chile’s socialist government attempted a version of this managed via telex machine, only for it to be cut short by the coup of 1973.

Thirty years after the fall of the Soviet Union, enthusiasm for these ideas has returned. In 2016, Jack Ma predicted that “the planned economy will become increasingly big … because with access to all kinds of data, we may be able to find the invisible hand of the market.” Marxist economists have written with surprising enthusiasm about Walmart and Amazon, viewing them as technologically-enabled planned economies.

The greatest excitement has been reserved for advanced AI. Zvi Mowshowitz has argued that AI “can embody the preferences and knowledge of many or even all humans, in a way an individual human or group of humans never could.” Meanwhile, Erik Brynjolfsson and Zoë Hitzig have made the case that, by combining immense processing capacity with the ability to codify tacit knowledge through computer vision, language, and sensor data, AI could erode the traditional advantages of decentralization.

The optimists attack the case for traditional markets and decentralization from multiple directions: AI can match or exceed the information-processing advantages of markets, capture knowledge embedded in human judgment, simulate competition without running it, assess outcomes markets model badly through proxies, or simply replace the human participants whose limitations created the problem in the first place.

Despite their diversity, many of these arguments fall into the same traps. They routinely misstate the case for decentralization and flatten the distinction between different kinds of knowledge, while treating any unsolved problems as an engineering detail.

The pursuit of knowledge

The most influential case against central planning was made on epistemological grounds by Friedrich Hayek. Oskar Lange and many of his successors read him as making a simple point about transmission: the useful knowledge in any economy is spread across millions of minds, so no central authority can collect it fast enough to act on it.

This is dangerously wrong. In “The Use of Knowledge in Society,” Hayek distinguishes between two different types of knowledge. The first is scientific or theoretical knowledge, which can be stated in general rules or principles. In theory, this kind of knowledge could be effectively concentrated in a single mind or system – arguably, LLMs already do this very well.

The second type is what he calls “knowledge of the particular circumstances of time and place.” This is knowledge that is embedded in practice, judgment, and context. For example, a farmer may know that a specific field drains poorly in its southeast corner or a sales rep may notice a change in body language with a long-standing client. This kind of knowledge is derived from the experience of a specific context, rather than from theoretical training or by performing a regression analysis. The person who has this knowledge will frequently struggle to explain how they acquired it.

This concept of tacit knowledge was expressed in more detail by Michael Polanyi, a chemist turned philosopher of science. Polanyi famously formulated tacit knowledge as the idea that “we can know more than we can tell.” There are lots of things we can do that we would struggle to articulate. The theoretical account of riding a bike – adjusting angular momentum through micro-corrections in steering – bears little relationship to the knowledge that you possess and exercise when doing it.

Polanyi’s view is that tacit knowledge is not just knowledge that happens to be unstated, but instead has a distinctive architecture.

Imagine a bank manager in a meeting with a local business owner, asking for an extension to their loan. In the course of this interaction, she senses that something isn’t right.

The bank manager is picking up on a series of cues, such as the business owner’s posture, the rhythm of his speech, or the differences from their past interactions. The bank manager experiences all of this as a single act of perception, rather than a series of data points. If we tried to unpick this knowledge and asked the bank manager to list out all the data points she picked up on, it would be akin to asking a pianist to state the precise angle of each finger while she’s playing.

Of course, the planner at this point could argue that even if the bank manager can’t articulate these cues, we could train a model across video, audio, and biometrics to detect the same patterns she’s detecting. The model doesn’t need to have the same experience as her, it just needs to produce the same outputs. For example, driving was long held up as an example of inalienable tacit knowledge; Polanyi himself argued that “the skill of a driver cannot be replaced by a thorough schooling in the theory of the motorcar.” Despite this, we now have highly performant self-driving cars.

This only tells us so much. Driving is mostly about seeing things and moving your body in response. The environment is physically constrained and the action space is narrow. While complicated, it is markedly less ambiguous than navigating a dense web of human intentions and social meanings. AI excels at chess, but falters in complicated social reasoning games. The market is far more like the latter.

The distinction here goes beyond difficulty. A car navigates spatial relationships and physical dynamics, whereas what the bank manager does is categorically different: she is interpreting, drawing on a framework of meaning built up through years of situated experience that organizes her perception before any calculation begins. That framework is the structure through which the interaction becomes intelligible to her at all. More processing power has no bearing on a gap like this.

By being situated in both the conversation and the wider social context, our bank manager also has a few other advantages versus an impersonal system. For a start, she has skin in the game. If she gets this call wrong, her reputation and business could suffer; in extreme cases, her physical safety could be at risk. Secondly, unlike a system observing a bunch of cues, she is an active participant in the interaction. Her tone and method of formulating questions are all part of the interaction. Finally, she’s situated in the community. She knows both the social norms and the realities of running a business in that area.

The snake devours its tail

Eventually, we get stuck in a loop. Even if we could train an AI on these interactions, what would the training data consist of? Someone has to decide to record certain things and not others. For example, we may include the transcript of the conversation, some financial metrics, and the outcome of the loan, but not the handshake or some of the pauses between words. The data is already a selective compression of the interaction, shaped by prior human decisions about what matters.

The tacit knowledge that made those framing decisions is invisible to the system trained on their outputs. No dataset encounters raw reality. It arrives pre-shaped by decisions about what to measure and what to discard. When you tell the system to look for indicators of trustworthiness, you’ve already decided what the relevant features are, which is precisely the judgment you were hoping the system would replicate.

We make direct perceptual contact with the world in a way that AI can’t. We determine the very concepts needed to carve up the world intelligibly, invent new ones constantly, and make normative and aesthetic judgments all throughout. If the data is always post-conceptual, then every training pipeline inherits the tacit knowledge of whoever decided what to measure, which means the system can never fully escape human judgment, even in principle.

The optimist could object here. Perhaps a reinforcement learning function could create something functionally equivalent to skin in the game. A dynamic AI system interacting with clients would also develop its own tone and method of probing. Big enough systems don’t simply compute over pre-specified data – they learn from experience and may develop something akin to internal world models. They may absorb tacit knowledge wholesale without anyone specifying what to look for.

Even granting all of that, the question is whether it can replicate the model of engagement that produces our bank manager’s particular sensitivity. She must sit inside the tension between reputational risk, relationship preservation, institutional obligation, and commercial judgment without the ability to collapse them into a single metric. This is what makes her attention so acute – she can’t keep everyone happy.

This points to something general. A system that lacks our direct perceptual access to the world can detect statistical regularities within whatever framework it’s been given. It won’t recognize the moment when existing categories fail to capture what’s really happening, because its measure of “what matters” is determined by the framework. A reward function can’t evaluate its own weights. Only someone embedded in the situation – who personally feels the weight of competing, immeasurable human risks, obligations, and norms – is capable of making that judgment.

Information versus action

The economist Israel Kirzner tells the story of a mother struggling with a teething child. The mother has tried everything she can to soothe or pacify the child, but to no avail. A travelling salesman knocks on her door and offers her a colorful toy at a price of five dollars. Her child is delighted and calms down. On closer inspection, she is dismayed to realize that the toy is nothing more than a collection of marbles in a clear plastic container – something that she could have assembled in her kitchen for less than a dollar. She “could kick herself for not having done so.”

On one level, this is understandable. The mother knew she had marbles in her kitchen and had the wherewithal to put them in a container, but this knowledge “did not inspire her to action.” In Kirzner’s framing, she had information-knowledge (all the facts), but not action-knowledge (the alertness to act on the information).

Even if you build a system that can collect every production function and every consumer preference in the economy, all you have done is assemble a comprehensive stock of information-knowledge. Alertness is hard to program, because it isn’t a process of inference from known data. A successful entrepreneur acts speculatively in response to a suspected opportunity, which by definition, hasn’t already been recognized.

Hayek went further. He observed that much of the knowledge that matters for economic coordination doesn’t exist at all until the competitive process generates it.

If you are an entrepreneur who tries a new approach, whether you succeed or fail, you have created new knowledge. As a result of the risk you took, people know whether a specific combination of resources, aimed at a set of customers, at a particular price point is viable. This knowledge didn’t exist somewhere in the ether waiting to be discovered by a more powerful algorithm. Instead, it was brought into existence by speculation in conditions of genuine uncertainty. The firm that succeeds reveals the value of its approach retroactively.

The price signals that result are not the transmissions of pre-existing data, but the outputs of a distinct process. The indeterminacy runs deeper still: economic agents are not carrying fixed utility functions. Participation in exchange changes the participants – human interactions themselves generate new kinds of choices. The system a planner would need to model doesn’t hold still, because the process of market coordination is partly constitutive of the preferences and knowledge it produces.

Akio Morita, the co-founder of Sony, launched the Walkman in 1979 against the objections of the rest of the company. Morita had observed that people took large stereos to the beach and listened to music in their cars, and sensed a market opportunity. Sony’s own market research and consumer surveys consistently suggested that there was no consumer demand for a tape player that couldn’t record, no matter how portable it was.

What Morita did was not pattern recognition on a richer dataset. He changed the conceptual space, reconceiving what a music device could be and who it could be for. No amount of data about what consumers said they wanted would have produced the Walkman, because the preference for it was partly a consequence of the product’s existence. The market revealed his conjecture to have value, and it now retrospectively seems obvious. But obviousness after the fact is the signature of knowledge that could only have been created through action.

The launch of the Walkman then had a series of downstream consequences – for competitors, component manufacturers, and music sales alike. The relationship between music and everyday life changed, with enduring social and economic consequences.

If it ain’t broke

Even if many of the foundational technological challenges could be solved by a future system, we would also need to believe that it would be worth the risk. An entrepreneur who bets wrong loses his own capital. A society that dismantles its price system has made an irreversible collective wager. We would need to have confidence that this future model would outperform markets – an order that, given basic institutions, no one has to design.

For the traditional Marxist, the case is straightforward. If technology can finally solve the calculation problem, it vindicates the claim that capitalism contains the seeds of its own succession. But as we saw earlier, the AI alternative to traditional markets increasingly appeals to those who don’t hope for the final triumph of the proletariat.

Oliver Klingefjord and Joe Edelman from the Meaning Alignment Institute argue that advanced AI systems could correct a number of shortcomings in current markets. They believe that markets systematically contract on proxy metrics like hours, subscriptions, engagement, rather than outcomes actually delivered. This is partly because doing so would be prohibitively expensive across millions of consumers, but also because there is an asymmetry of power between suppliers and consumers. Big suppliers can write “take it or leave it” contracts and have huge information advantages versus those that they are contracting with.

Klingefjord and Edelman argue that replacing markets with AI could collapse these measurement and bargaining costs, making it feasible to pay suppliers for delivered benefit via competitive, voluntary AI intermediaries that pool consumers, assess outcomes qualitatively, and negotiate enterprise-level deals.

This is much more sophisticated than technosocialism, but runs into some of the same problems.

This approach tries to maintain a price system, but changes what the prices track. But this entire system relies on an intermediary being able to assess whether a good outcome was delivered. Unlike market prices, which emerge from entrepreneurial bids under genuine uncertainty and personal risks, these assessed prices would reflect a system’s operationalized definition of human benefit. While the user might set the guardrails, the system has to turn these into assessable criteria. In essence, it’s a system’s approximation of a person’s approximation of what constitutes human flourishing.

This would also result in an enormous amount of discretionary judgment embedded in an infrastructure layer that most people would never inspect. While you could try to mitigate this by having a world of competing AI intermediaries, it’s hard to see users choosing between rival theories of their own good – as operationalized by AI systems that they can’t design – as an obvious improvement on choosing between rival products in a traditional market.

Magical thinking

It is, of course, possible to argue that these objections could all be overcome. Maybe we will build systems that can collect all dispersed and local knowledge, model genuine alertness, simulate exchange, and anticipate the outputs of the price discovery process without running it. It may then lead to more efficient resource allocation. But we haven’t built these systems, and nothing in the current trajectory suggests we are close. The burden of proof lies with those who believe otherwise.

The case for planning, by necessity, assumes away all real-world constraints while simultaneously reversing the burden of proof. In response to arguments about the importance of markets, it hypothesizes a system that by stipulation overcomes any individual objection and then challenges opponents to prove that it’s impossible.

Thought experiments that grant one or two premises can be genuinely useful when thinking about advanced technology, but with each additional hypothetical, the value diminishes. Any theory of how the world could work becomes plausible once you assume the existence of an omniscient machine god.

In the end, the CIA didn’t need to worry about Soviet cybernetics. Glushkov’s proposed cybernetics system, budgeted to cost the equivalent of over one trillion dollars in today’s money, never saw the light of day. The Soviet authorities, unconvinced by his argument that the system would pay for itself several times over, balked at the price tag. Cybernetics would not get to play its role in the inevitable triumph of the proletariat.

It seemed that even the Soviets had lost faith in the planned economy. On this, if little else, they were right, even if it was not for reasons that they fully understood. The fundamental obstacle was never processing power or data collection. It was that the economy a planner would need to model is constitutively shaped by the expectations and interpretive frameworks of the people who participate in it. Those frameworks shift in response to the very act of observation and intervention. There is no fixed economy waiting to be measured.

The system that a planner would need to model is the same system that the plan would destroy.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.