This week, we crossed 20,000 subscribers on Substack. Thank you to everyone who has read, shared, and engaged with our work.

We’ve written about Claude Boys to Coasean bargaining to the perils of liberal nudging. Reading the comments has often been as rewarding as writing the posts. To mark the milestone, I’ll be answering your questions on Wednesday April 15.

Drop your question in the comments below and upvote the ones you want answered. I’ll start responding next week and I’ll try to take as many as I can.

There are a few things I’ve been thinking about that we haven’t written about yet. This seems like the right place to start.

Ask about Cosmos, human autonomy, AI x philosophy, or what people in our network are building. Especially questions that are hard, that relate to how we approach AI as builders, or that challenge our assumptions.

- Brendan

Leave a comment

In the autumn of 1809, Prussia was a country that no longer knew what it was for. Three years earlier, Napoleon had destroyed its army in an afternoon and walked into Berlin without resistance. The king fled. Half the territory was gone, the treasury empty. French soldiers were still garrisoned in the capital.

As Prussia began rebuilding from the wreckage, most people assumed it needed more officers, administrators, and engineers. People who could do things. The task of designing the new system of education fell to a thirty-two-year-old diplomat named Wilhelm von Humboldt. He gave them something else entirely.

In a series of memoranda written over the next year, he laid out a vision for a new university in Berlin organized around Bildung. The word has no English equivalent. “Education” is too narrow, “self-improvement” too thin. “Formation” gets closest but still misses its moral weight.

Humboldt’s Bildung means the free, harmonious development of a human being’s powers into a complete and consistent whole, through encounter with the world in its variety and resistance.

Mill, who took the idea from Humboldt, put it more simply: a human being is more like a tree than a steam engine.

Humboldt proposed a university where professors and students would be joined in the pursuit of knowledge, unconstrained by political demands. In a defeated nation hungry for officers and administrators, he was arguing for formation before function.

The ideal of the modern research university, with its union of teaching and inquiry, its seminar culture, and its commitment to academic freedom, descends from what Humboldt designed in those desperate months.

Today, the culture of Bildung that animated the university survives only at the margins, sustained by people stubborn enough to work against the grain of the institutions around them. Credentialism twisted the university into a vendor of certificates, and the formation of the student as a complete human being came to seem anachronistic. The cathedrals remain, but not the faith.

We are in a moment that rhymes with Humboldt’s own. Technological pressure is once again pushing education toward the practical. The evidence of institutional collapse is everywhere: flagship universities slashing PhD admissions, hundreds of degree programs have been eliminated, dozens of small colleges have closed, and a decline in the college-age population still lies ahead.

The loudest responses to the crisis have come from outside the university. Alex Karp tells young people to skip college and learn a trade. Marc Andreessen argues the university is a credentialing middleman and should be disintermediated. Both are right that the university is failing. But if the answer to a broken formation system is to skip formation altogether, you have already conceded that education is justified only by utility.

Neither is asking the question Humboldt asked: What is a human being, that education should serve it?

Subscribe now

The Tree and the Trap

Before Humboldt, the Prussian system assigned students to a vocation, trained them in it, and delivered them into their function. Even on those terms, Bildung wins. Broadly cultivated judgment produces better doctors and better engineers. Technical skill on a foundation of general cultivation is more resilient and more humane than technical skill resting on nothing.

But if that were the whole justification, Bildung would be nothing more than a roundabout way of minting skilled professionals. A pedagogy can be justified by output. Bildung cannot.

A tree does not exist in order to produce lumber. You can make lumber from it, and good lumber is nothing to sneer at. But if you look at a tree and see only lumber, you have missed what is standing in front of you. Something is growing there under its own power, toward its own form, and the growing is not a means to some further end.

Humboldt’s claim about human beings is the same shape. A person is a self-developing being whose worth is not exhausted by function.

Bildung braids together the Kantian claim that no person is merely a means, the Greek ideal of harmonious excellence across mind and body and character, and the Romantic conviction that we develop through encounter with what resists us.

You have had the experience even if you never had a word for it. Real engagement with something that has its own demands—a hard problem, a serious book, a gifted teacher—changes who you are. You could not have planned the person you became.

Such formation is not the property of any particular university department. This is not a “save the humanities” argument. A coder who, after tackling a hard systems-design problem, comes out thinking differently about complexity, tradeoffs, and the limits of formal reasoning has undergone a kind of Bildung—but only if the encounter changed who they are, not just what they can do.

The resistance to all this is understandable.

In a world where your economic value can evaporate overnight, “Become a whole person” sounds like advice from someone who has never worried about next month’s rent. The utilitarian case for education has the force of necessity behind it. For millions of people, making yourself useful is what responsibility to their families demands.

Yet if the response to being replaceable is always to train for a different function, you have entered a race you structurally cannot win. The principle that makes your education valuable is the same principle that makes you disposable the moment the function migrates.

The scramble into computer science was an early sign of the trap: students rushed toward the field that seemed safest, and then AI began destabilizing the very functions it trained them to perform. The flight to function looks rational from inside it; that is what makes it a trap.

Solitude and Freedom

Bildung cannot be specified in advance. Once you define what the formed person looks like, you have replaced formation with training. So how do you build institutions around it?

Humboldt’s solution was to design an environment rather than a curriculum. He based the University of Berlin on two principles: solitude and freedom, though they meant something precise in his hands. The university would not answer to the demand for immediate use and inquiry would follow the question wherever it led, unconstrained by predetermined ends.

The undoing of that design came in two phases.

First, a slow hollowing. The German research model crossed the Atlantic when Johns Hopkins was founded in 1876 on Humboldt’s model. But disciplines hardened into guilds and career tracks, postwar federal funding built research enterprises increasingly detached from teaching, and mass enrollment turned the degree into a sorting mechanism. The university no longer had a single organizing purpose, and no one was left responsible for the student as a whole person. By the mid-twentieth century, the university president Clark Kerr could describe the resulting “multiversity” as a collection of unrelated enterprises held together by a common heating plant.

Then, a rejection of the cure. Recently, at the University of Tulsa, the philosopher Jennifer Frey built a Great Books honors college from scratch. It was a serious formation program inside an institution organized around credentialing. The university removed her, restructured the program, and replaced the deanship with a directorship.

Back to Scholé

But even if the institutions had held, formation requires something they could never secure at scale: time.

Aristotle called it scholé. Humboldt had a related word for it: Muße. Both named a kind of structured freedom for the work of becoming, and for most of history that freedom was radically exclusive.

Aristotle could imagine the highest forms of human flourishing only for those relieved of labor by wealth and the work of subordinates and slaves. The good life required freedom from necessity, and in his world only a few could have it. But in the first book of the Politics he imagined something stranger: that if shuttles could weave by themselves and picks could play the lyre, craftsmen would need no subordinates and masters would need no slaves. The “self-guided machine” would mean that the material basis for leisure no longer depended on the unfreedom of others. It is one of the oldest thought experiments in Western philosophy, and we are now enacting it.

Aristotle did not celebrate the prospect. He understood that freedom from necessity does not automatically yield the pursuits that make such freedom worth having. In his account, those with wealth and leisure often turned to unlimited acquisition or bodily gratification rather than to the activities that justify leisure in the first place.

With AI, we are building something like self-guided machines. Whether these systems liberate or merely displace is not settled. But the possibility of leisure at scale is real enough to become a serious question.

If AI can compress parts of instruction, it may deepen learning where it is used and clear ground for formation where it gives time back. But only if it preserves productive struggle rather than bypassing it.

The alternative is already visible: autocomplete for life. Not just help with expression, but the slow outsourcing of judgment itself. That is Bildung’s antithesis.

Worse, the same technological society enabling leisure is also shaping the desires of the people who receive it. If our dispositions have already been trained toward optimization and outsourced judgment, the freed hours may arrive in hands that no longer know what to do with them.

For most of history, the conditions of formation were reserved for the few. The capacity for it was not. If scholé at scale is now possible, refusing to pursue it ratifies a world in which full human development remains the privilege of those who can afford time.

In the Afterglow

Bildung is, at its normative core, anti-servility: the effort to form people who cannot be reduced to instruments of external authority, whether state, market, or algorithm.

The people who use AI well right now are drawing on judgment they formed before these tools became ambient. They know when to trust an LLM’s output and when to push back because they learned to read, argue, and sustain attention under conditions in which those acts were not so easily outsourced. They bring something the tool cannot supply.

Nietzsche thought secular liberals were living off the moral capital of a Christianity they had officially abandoned: inherited capacities that could persist for a time even after the culture that formed them had ceased to renew them. The kind of judgment this essay is defending may be a similar afterglow, formed in a world before AI mediated everything.

Without that judgment, you get agency without autonomy.

If the capacities required for non-servile life in an AI world were all formed in a pre-AI world, what happens when that formation stops? You can live on an inheritance for a while. You cannot educate a civilization on inherited judgment forever.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

The university is under siege.

Harvard, sitting on a $57 billion endowment, slashed 75 percent of its science and 60 percent of its humanities PhD admissions. Indiana mandated the elimination of over 400 degree programs at its public universities. Classics, comparative literature, and foreign languages at the state flagship are gone. At the University of Tulsa, a philosopher built a Great Books honors college from scratch, grew enrolment 500 percent, and attracted a quarter of each freshman class to a curriculum running from Homer to Arendt. The university removed her.

Sixty-four small colleges have closed since 2020, and the demographic cliff, a 13 percent decline in college-age students projected through 2041, hasn’t even started.

Meanwhile, 52 percent of recent graduates are working jobs that don’t require the degree that they borrowed $30,000 to get. Palantir CEO Alex Karp is telling young people to skip elite colleges entirely, saying the only paths left are skilled trades or neurodivergence.

AI will accelerate this. The skills the degree was supposed to impart – researching, analyzing, drafting, coding – are increasingly things a machine can do.

As the old institutions break down, a new generation of educational founders is running experiments. New schools are emerging, each making a different bet on two questions that anyone building an educational institution has to answer:

• The first question is about purpose. Do you focus on specific, trainable skills, or on a broader, if more intangible formation, which includes the development of judgment, attention, and moral seriousness?

• The second question is about technology. Do you build AI into the core of the educational experience or keep it out because the difficulty is where the learning happens?

These two axes produce a landscape.

At one end, programs like Gauntlet train elite AI engineers in ten weeks: 80-100 hours a week of building, a guaranteed $200K job offer at the end, costs paid entirely by employers.

At the other, institutions like St. John’s College run intensive liberal arts programs built on sustained attention and close reading, with no AI in sight.

Arizona State has embedded AI from coursework to advising and has struck partnerships with a number of high-profile AI companies, but the core teaching model hasn’t changed.

The University of Austin has proposed splitting the day between device-free seminars and intensive AI work.

Meanwhile, the bulk of the higher-education system is either banning AI from the classroom or pretending that it doesn’t exist.

These questions are not new. They go back to the founding of the first American colleges and European research universities.

In partnership with Liberty Fund, Cosmos held a seminar to think through them with a group of founders, scholars, and institution-builders. The participants included philosophers from UT Austin and Ohio State, researchers from MIT Media Lab, RAND, and leading AI labs, education policymakers from the Texas Higher Education Coordinating Board, and builders from Fractal Tech and Alpha School.

The reading list we assembled traces these tensions to their origins: from the earliest American proposals for public education through the founding documents of the research university, the mid-century debates about liberal learning, and the first serious writing about what computers might do to the relationship between learner and knowledge.

Subscribe now

Session I: The Promise and Crisis of the University

We are not the first to ask what the university is for.

This session combined five voices, spanning three centuries. Franklin wants a practical institution for a practical republic. Humboldt argues that the university exists for the development of the whole person, not the production of useful professionals. Oakeshott insists that education is how a human being becomes one. Bloom diagnoses what happens when that inheritance is abandoned. And Caplan asks whether the whole enterprise is a $240,000 receipt that signals you can show up on time.

If Caplan is even partly right, the reformers need to explain what they are offering that a credential cannot capture. If Humboldt is right, the reformers need to explain why two centuries of institutions built on his model ended up producing exactly the credentialism he warned against.

• Benjamin Franklin, Proposals Relating to the Education of Youth in Pennsilvania. (link)

• Wilhelm von Humboldt, “On the Internal and External Organization of the Higher Academic Institutions in Berlin”. (link)

• Michael Oakeshott, The Voice of Liberal Learning, Introduction.

• Allan Bloom, “Our Listless Universities.” (link)

• Bryan Caplan, The Case Against Education, Chapter 1, “The Magic of Education.”

Session II: What Formation Requires

If the first session asked what the university is for, the second focused on what actually changes a person. Weil makes the strongest claim: attention is all you need. Klein argues that liberal education requires putting your own opinions genuinely at risk, and that the resistance to doing so is rooted in something specifically human: we build our identities on what we think we know. Gadamer argues that practical wisdom cannot be taught as a curriculum; it grows out of ethos, the character already formed by living in a particular community.

But what if the boundaries of the person aren’t where we think they are? Clark and Chalmers ask whether cognition is even confined to the skull. If the mind extends into the tools and environments we think with, what counts as “the student”? Shanahan complicates the picture further: LLMs create a compelling illusion of understanding, but they are fundamentally unlike us. If formation requires genuine encounter with other minds, what happens when the most available interlocutor is a machine?

• Simone Weil, Waiting for God, “Reflections on the Right Use of School Studies with a View to the Love of God.”

• Jacob Klein, “The Idea of Liberal Education” in The Goals of Higher Education, ed. W.D. Weatherford, Jr.

• Hans-Georg Gadamer, “The Socratic Question and Aristotle”, Continental Philosophy Review.

• Andy Clark and David Chalmers, “The Extended Mind”, Analysis. (link)

• Murray Shanahan, “Talking about Large Language Models”, Communications of the ACM. (link)

Session III: The Institutional Question

Good ideas about education are easy. Institutions that embody them are hard. This session focused on the gap. Jefferson proposes education as grafting: implanting something new onto the wild stock of human nature. Newman argues that the value of a university lies in the sheer density of its intellectual community, that students gain from living among those who represent the whole circle of knowledge even if they can never study it all.

Karlsson brings this into the present. His argument is sobering: AI tutors will be held back by culture, rather than technology. Motivated learners embedded in high-growth communities will use AI to accelerate, while everyone else will use it to avoid difficulty. The real challenge will be building strong cultural norms against taking the path of least resistance.

• Thomas Jefferson, “Draft of the Rockfish Gap Report of the University of Virginia.” (link)

• John Henry Newman, The Idea of a University, Discourse 5, “Knowledge its Own End.” (link)

• Henrik Karlsson, “AI tutors will be held back by culture.” (link)

Session IV: Education and the Machine

This session turned directly to the question of what technology does to thinking. Licklider, writing in 1960, imagined a partnership: machines handle the routine cognitive work, freeing humans for insight and decision. Papert envisaged a more profound interplay. He argued that when a child learns to program, the relationship between learner and knowledge is fundamentally transformed. The child is no longer receiving explanations but building things, and the building changes how she thinks.

Matuschak and Nielsen pick up the thread sixty years later and find it frayed. The pioneering visions of tools for thought are treated as nostalgia in technology circles. There is little determined effort to build tools that genuinely transform how people understand. The discussion focused on whether the current wave of AI represented a chance to revive the Licklider-Papert vision or whether tools for thought were likely to become an even more distant memory.

• JCR Licklider, “Man-Computer Symbiosis,” IRE Transactions on Human Factors in Electronics.

• Seymour Papert, Mindstorms: Children, Computers, and Powerful Ideas, Ch. 1.

• Andy Matuschak and Michael Nielsen, “How can we develop transformative tools for thought?” (link)

Session V: Building the New Academy

The final session focused on what comes next. Flexner’s defense of useless knowledge, written from the Institute for Advanced Study, is the purest case for protecting inquiry from the demand for application. Bloom’s portrait of Hutchins at UChicago shows what it looks like when someone actually tries to build an institution around these convictions, and the political will it requires. Simondon challenges the premise that technology and culture are opposed, arguing that the hostility between them is a sign of ignorance.

Engelbart closes the list with a constraint AI is testing: “The entire effect of an individual on the world stems essentially from what he can transmit to the world through his limited motor channels.” As those channels widen, the bottleneck shifts to whether you have something worth transmitting.

• Alexander Flexner, “The Usefulness of Useless Knowledge,” Harper’s Magazine. (link)

• Allan Bloom, “Hutchins’s Idea of a University,” Times Literary Supplement.

• Gilbert Simondon, On the Mode of Existence of Technical Objects, “Introduction.”

• Douglas C. Engelbart, “Augmenting Human Intellect: A Conceptual Framework,” SRI Summary Report AFOSR-3223.

Is there an institution that protects the conditions for deep formation and teaches students to master the tools that will define their century?

Subscribe now

Cosmos Institute is the Academy for Philosopher-Builders, with programs, grants, events, and fellowships for those building AI for human flourishing

“Artificial Intelligence will revolutionize science.” The phrase is usually invoked by researchers when discussing the promise of thinking machines to help us understand the natural world. In the early 2020s, I subscribed to, and was perhaps even evangelizing for, the role of AI in scientific research. Like many of my colleagues, I was gripped by a new tool that promised so much.

More recently my view has changed. I still believe that AI will be fantastically useful, but not necessarily in the way we think. Discovery, after all, is not the same as understanding. As scientists, we need to engage with the process of inquiry to truly make sense of what we learn about the world. Only then can we understand.

The Long History of “AI for Science”

In the summer of 1953, physicists Enrico Fermi, John Pasta, and mathematicians Stanislaw Ulam and Mary Tsingou ran the first “numerical experiment” on MANIAC, one of the early electronic computers built at the Los Alamos National Laboratory after World War II. Computers were new and scientists were excited to explore their use in solving research problems like simulating the dynamics of atoms and molecules, studying tumor cell populations and investigating numerically fluid dynamics. They even created the first documented chess-playing program that defeated a human in the game.

This group decided to model a chain of oscillators (identical masses connected by springs), add a small nonlinearity (a quadratic or cubic term), and see what would happen. The expectation was that the system would reach equilibrium (common sense would predict that there is some wiggling around, but ultimately energy ends up equally distributed throughout the springs). But the results were surprising. Instead, the system showed a recurrent behavior where the energy spreads throughout the chain, then comes back, before spreading out again. This observation sparked interest in the study of nonlinear systems that would later lead, among other things, to the discovery of solitons (waves that travel freely preserving their shape) and the development of chaos theory in the 1960s-1970s.

If one was to assign a birthyear to computational physics, that honorific should go to 1953 where Fermi and colleagues made for the first time an unexpected discovery through a purely computational approach. Computers had been used for physics and astronomy calculations before – for example, in the 1930s mechanical IBM accounting machines were modified to solve the differential equations of planetary motion by numerical integration – and had played a key role in the development of the atomic bomb during the Manhattan project. In the following decades computers would transform scientific research. Today, there are almost no advances in physics that are not enabled by some aspect of computer simulation.

Three years after the “first numerical experiment” the term artificial intelligence was coined and a first attempt was made to use automated reasoning to prove mathematical theorems through the Logic Theorist system in 1956. The first AI “expert” system, DENDRAL, was created in 1965. Combining a knowledge base with a reasoning engine, it was capable of determining the molecular structure of a compound from its mass spectra. Here, “expert” refers to a kind of AI system that encoded domain knowledge from human experts as rules and applied them through an inference engine.

Other examples include LHASA, a program designed in 1972 to discover sequences of reactions to synthesize a molecule; and the Automated Mathematician in 1977, a heuristic-based program designed to discover new mathematical concepts and theorems; and other expert systems like MYCIN (to identify bacteria) and PROSPECTOR (for mineral analysis). In the late 1980s and early 1990s, artificial neural networks were used to tackle problems in particle physics, astrophysics, and –– in a glimpse of what was to come –– predicting protein structure. By the 2000s, AI methods were in use in many areas of physics and even assisted data analysis leading to the discovery of the Higgs boson particle.

In the 2010s and 2020s, scientists began to use a wave of powerful neural network-based tools to solve research problems. Some of these advances, such as Google DeepMind’s AlphaFold protein prediction system, made headlines around the world. Others were important but less glamorous. Machine-learned interatomic potentials, for example, significantly speed up materials and chemistry simulations yet remain unknown outside the research community.

But if “AI for science” has a long history, why has it only recently started to make the news? In his 2003 book, Douglas S. Robertson coined the term “phase change” to describe a radical change that an instrument makes possible compared to the prior state of the art. AI may be a phase change in how we do science, but not only because of powerful individual tools. It is their extreme accessibility, from large language models to the AlphaFold Protein Structure Database, that sees advances in one field become instruments of exploration in others.

Epistemic Enhancers

Scientific instruments provide extensions to humans’ senses. Telescopes and microscopes allowed us to see the far and the small. This type of enhancement is known as extrapolation. Another type of epistemic support is conversion, that is, transforming one modality into another (like sound into a visual image). Finally, we have those instruments that extend our capacities by giving access to phenomena that elude our senses, such as detecting radiation or magnetic fields. This is known as augmentation. These three moves – extrapolation, conversion, and augmentation – are all types of epistemic enhancers.

A central motivation behind scientific practice is understanding. But what is scientific understanding? The 2009 book Scientific Understanding edited by Henk W. de Regt, Sabina Leonelli, Kai Eigner suggests that there is no universal definition of scientific understanding. Understanding in physics differs from understanding in biology or engineering. Even within my own discipline, physics, what we mean by understanding is not straightforward. While clearly related to the explanation of a phenomenon, understanding is not precisely the same thing. There is a difference between understanding the phenomena and understanding the theories or models that explain that phenomenon.

A prerequisite for understanding is discovery insofar as one cannot understand a phenomenon that has not been observed. Discovery is the process or product of successful scientific inquiry. Objects of discovery are things, events, processes, causes, and properties as well as theories and hypotheses and their features. The 1987 book Scientific Discovery proposed that discovery in science can be broken down into problem solving tasks and therefore can be automated with computers.

The authors of the book built four types of programs to look for quantitative or qualitative laws and structural models. Then they used them individually and combined them to rediscover many laws in physics and chemistry. While the authors recognized that there is no unique process that accounts for scientific discovery, they showed that most of these can be cast as problem solving tasks that can be tackled with heuristics. Their method, based on tree searches, was general in theory but limited in practice. This was because the combinatorial explosion of possible paths made them intractable for problems beyond a modest scale.

Both traditional numerical methods and modern AI tools enhance epistemic extrapolation and conversion. The application of such methods represents a difference in kind as well as magnitude for scientific practice. As Paul Humphreys put it: “This extrapolation of our computational abilities takes us to a region where the quantitatively different becomes the qualitatively different.” This is because these simulations cannot be carried out in practice except in regions of computational speed far beyond the capacities of humans. This dual articulation of the quantitative and the qualitative could be taken as an argument for why modern AI tools ought to usher in a new way of doing science.

In the 1960s mathematicians Bryan John Birch and Peter Swinnerton-Dyer used the EDSAC-2 computer at the University of Cambridge Computer Laboratory to run numerical calculations that allowed them to state a conjecture about the set of rational solutions to equations defining an elliptic curve. This is now known as Birch and Swinnerton-Dyer (BSD) conjecture and is one of the seven Millennium Prize Problems. The computer helped them explore an abstract space so they could formulate the conjecture. This computer-assisted discovery falls in the final category of epistemic enhancers mentioned above: augmentation.

If the formulation of the BSD conjecture is a weaker example of augmentation, more recently AI methods helped mathematicians make a breakthrough towards solving it. Using machine learning, mathematicians discovered unexpected oscillatory patterns in the parameter space key to BSD (of hundreds of dimensions) which led to the “murmuration” conjectures. Progress has also been made towards solving another Millennium Prize Problem as AI tools were used to find potential singularities in the Navier-Stokes equations.

Although automated theorem proving programs have been around for 80 years, these examples do suggest that we are witnessing, in Robertson’s terms, a phase change. The quantitatively different becomes the qualitatively different in a more interesting way. It’s speed, yes, but it’s also breadth. Take AlphaFold and the AlphaFold Protein Structure Database it made possible. What makes it revolutionary is not only the improved accuracy of the protein structure prediction, but also its usability by scientists from many fields (who can access hundreds of millions of protein structure predictions).

We are yet to see the full extent of what AI tools can do for scientific research, but one differentiator between them and traditional scientific instruments, including computer simulation, is that they combine epistemic enhancers to explore vast dimensions of information. Remember DENDRAL, the early AI system determining the molecular structure of a compound from its mass spectra? Now, imagine the power of its modern incarnation which has access to several databases of molecular spectra, can cross-check with published scientific literature, and has the ability to write code to perform additional calculations if needed.

Or consider a powerful telescope that discovers a new exoplanet. Here an AI tool analyzes the spectra, determines the likely composition of the exoplanet atmosphere, compares it with the existing recorded information, runs simulations with the known and inferred parameters to better characterize the exoplanet and adds it to the catalog. All of these individual steps are already done separately and for each the use of AI tools can be seen as an evolutionary improvement. But the combination of these improvements, working together across different sources of information is potentially revolutionary.

If I’m permitted to add a new epistemic enhancer category for AI as an instrument the most appropriate would be integration (as in synthesizing across large amounts of very different types of information). When a tool synthesizes across hundreds of heterogeneous sources simultaneously, the inferential path from evidence to conclusion can become challenging for the scientist receiving the result to trace. The more dimensions integration spans, the wider this gap between discovery and reconstruction becomes. Where earlier epistemic enhancers extended what scientists could see or calculate, integration may increasingly determine what they conclude.

Our Role as Scientists

At the end of his book, Humphreys argued that computer simulation brings a “shift of emphasis in the scientific enterprise away from humans.” By the 1980s, researchers already knew that, at least in part, scientific discovery and problem generation could be successfully automated. Modern AI methods can do the same thing, only much faster and much better. I regularly hear from colleagues that in the coming years AI will take over many of the tasks associated with scientific inquiry.

Use of AI will likely mean more discoveries overall, but to discover is not to understand. Henk de Regt argued that: “scientists achieve understanding of phenomena by basing their explanations on intelligible theories. The intelligibility of theories is related to scientists’ abilities: theories are intelligible if scientists have the skills to use those theories in fruitful ways.” Quantum mechanics is an example of a very effective, yet not particularly intelligible theory, that despite the practical successes still eludes physicists in its interpretation. If AI tools can optimize discovery and generate explanations, they could perhaps also help produce more intelligible theories.

Yet intelligibility is not a property of theories themselves. As de Regt suggests, it depends on the capacities of the scientists who use them. A theory becomes intelligible only when scientists acquire the skills to explore its implications and apply it fruitfully. Even if machines make discoveries and generate explanations, this kind of understanding still depends on the participation of human investigators. Put differently: scientists need science just as science needs scientists.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

In 1962, Victor Glushkov pitched the Soviet authorities on a nationwide cybernetics network to solve the oldest problem in socialist economics: how to allocate resources without private property and market prices. Washington was alarmed enough for the CIA to create a special taskforce.

Just as Soviet industrialization was giving way to stagnation, the first serious computers gave the planned economy a shot in the arm. The central planners believed that they’d struggled to process information fast enough, so increased computing power promised a solution.

In 1969, the Polish economist Oskar Lange described the market as “a computing device of the pre-electronic age,” suggesting we could put “simultaneous equations on an electronic computer and obtain the solution in less than half a second.” In 1971, Chile’s socialist government attempted a version of this managed via telex machine, only for it to be cut short by the coup of 1973.

Thirty years after the fall of the Soviet Union, enthusiasm for these ideas has returned. In 2016, Jack Ma predicted that “the planned economy will become increasingly big … because with access to all kinds of data, we may be able to find the invisible hand of the market.” Marxist economists have written with surprising enthusiasm about Walmart and Amazon, viewing them as technologically-enabled planned economies.

The greatest excitement has been reserved for advanced AI. Zvi Mowshowitz has argued that AI “can embody the preferences and knowledge of many or even all humans, in a way an individual human or group of humans never could.” Meanwhile, Erik Brynjolfsson and Zoë Hitzig have made the case that, by combining immense processing capacity with the ability to codify tacit knowledge through computer vision, language, and sensor data, AI could erode the traditional advantages of decentralization.

The optimists attack the case for traditional markets and decentralization from multiple directions: AI can match or exceed the information-processing advantages of markets, capture knowledge embedded in human judgment, simulate competition without running it, assess outcomes markets model badly through proxies, or simply replace the human participants whose limitations created the problem in the first place.

Despite their diversity, many of these arguments fall into the same traps. They routinely misstate the case for decentralization and flatten the distinction between different kinds of knowledge, while treating any unsolved problems as an engineering detail.

The pursuit of knowledge

The most influential case against central planning was made on epistemological grounds by Friedrich Hayek. Oskar Lange and many of his successors read him as making a simple point about transmission: the useful knowledge in any economy is spread across millions of minds, so no central authority can collect it fast enough to act on it.

This is dangerously wrong. In “The Use of Knowledge in Society,” Hayek distinguishes between two different types of knowledge. The first is scientific or theoretical knowledge, which can be stated in general rules or principles. In theory, this kind of knowledge could be effectively concentrated in a single mind or system – arguably, LLMs already do this very well.

The second type is what he calls “knowledge of the particular circumstances of time and place.” This is knowledge that is embedded in practice, judgment, and context. For example, a farmer may know that a specific field drains poorly in its southeast corner or a sales rep may notice a change in body language with a long-standing client. This kind of knowledge is derived from the experience of a specific context, rather than from theoretical training or by performing a regression analysis. The person who has this knowledge will frequently struggle to explain how they acquired it.

This concept of tacit knowledge was expressed in more detail by Michael Polanyi, a chemist turned philosopher of science. Polanyi famously formulated tacit knowledge as the idea that “we can know more than we can tell.” There are lots of things we can do that we would struggle to articulate. The theoretical account of riding a bike – adjusting angular momentum through micro-corrections in steering – bears little relationship to the knowledge that you possess and exercise when doing it.

Polanyi’s view is that tacit knowledge is not just knowledge that happens to be unstated, but instead has a distinctive architecture.

Imagine a bank manager in a meeting with a local business owner, asking for an extension to their loan. In the course of this interaction, she senses that something isn’t right.

The bank manager is picking up on a series of cues, such as the business owner’s posture, the rhythm of his speech, or the differences from their past interactions. The bank manager experiences all of this as a single act of perception, rather than a series of data points. If we tried to unpick this knowledge and asked the bank manager to list out all the data points she picked up on, it would be akin to asking a pianist to state the precise angle of each finger while she’s playing.

Of course, the planner at this point could argue that even if the bank manager can’t articulate these cues, we could train a model across video, audio, and biometrics to detect the same patterns she’s detecting. The model doesn’t need to have the same experience as her, it just needs to produce the same outputs. For example, driving was long held up as an example of inalienable tacit knowledge; Polanyi himself argued that “the skill of a driver cannot be replaced by a thorough schooling in the theory of the motorcar.” Despite this, we now have highly performant self-driving cars.

This only tells us so much. Driving is mostly about seeing things and moving your body in response. The environment is physically constrained and the action space is narrow. While complicated, it is markedly less ambiguous than navigating a dense web of human intentions and social meanings. AI excels at chess, but falters in complicated social reasoning games. The market is far more like the latter.

The distinction here goes beyond difficulty. A car navigates spatial relationships and physical dynamics, whereas what the bank manager does is categorically different: she is interpreting, drawing on a framework of meaning built up through years of situated experience that organizes her perception before any calculation begins. That framework is the structure through which the interaction becomes intelligible to her at all. More processing power has no bearing on a gap like this.

By being situated in both the conversation and the wider social context, our bank manager also has a few other advantages versus an impersonal system. For a start, she has skin in the game. If she gets this call wrong, her reputation and business could suffer; in extreme cases, her physical safety could be at risk. Secondly, unlike a system observing a bunch of cues, she is an active participant in the interaction. Her tone and method of formulating questions are all part of the interaction. Finally, she’s situated in the community. She knows both the social norms and the realities of running a business in that area.

The snake devours its tail

Eventually, we get stuck in a loop. Even if we could train an AI on these interactions, what would the training data consist of? Someone has to decide to record certain things and not others. For example, we may include the transcript of the conversation, some financial metrics, and the outcome of the loan, but not the handshake or some of the pauses between words. The data is already a selective compression of the interaction, shaped by prior human decisions about what matters.

The tacit knowledge that made those framing decisions is invisible to the system trained on their outputs. No dataset encounters raw reality. It arrives pre-shaped by decisions about what to measure and what to discard. When you tell the system to look for indicators of trustworthiness, you’ve already decided what the relevant features are, which is precisely the judgment you were hoping the system would replicate.

We make direct perceptual contact with the world in a way that AI can’t. We determine the very concepts needed to carve up the world intelligibly, invent new ones constantly, and make normative and aesthetic judgments all throughout. If the data is always post-conceptual, then every training pipeline inherits the tacit knowledge of whoever decided what to measure, which means the system can never fully escape human judgment, even in principle.

The optimist could object here. Perhaps a reinforcement learning function could create something functionally equivalent to skin in the game. A dynamic AI system interacting with clients would also develop its own tone and method of probing. Big enough systems don’t simply compute over pre-specified data – they learn from experience and may develop something akin to internal world models. They may absorb tacit knowledge wholesale without anyone specifying what to look for.

Even granting all of that, the question is whether it can replicate the model of engagement that produces our bank manager’s particular sensitivity. She must sit inside the tension between reputational risk, relationship preservation, institutional obligation, and commercial judgment without the ability to collapse them into a single metric. This is what makes her attention so acute – she can’t keep everyone happy.

This points to something general. A system that lacks our direct perceptual access to the world can detect statistical regularities within whatever framework it’s been given. It won’t recognize the moment when existing categories fail to capture what’s really happening, because its measure of “what matters” is determined by the framework. A reward function can’t evaluate its own weights. Only someone embedded in the situation – who personally feels the weight of competing, immeasurable human risks, obligations, and norms – is capable of making that judgment.

Information versus action

The economist Israel Kirzner tells the story of a mother struggling with a teething child. The mother has tried everything she can to soothe or pacify the child, but to no avail. A travelling salesman knocks on her door and offers her a colorful toy at a price of five dollars. Her child is delighted and calms down. On closer inspection, she is dismayed to realize that the toy is nothing more than a collection of marbles in a clear plastic container – something that she could have assembled in her kitchen for less than a dollar. She “could kick herself for not having done so.”

On one level, this is understandable. The mother knew she had marbles in her kitchen and had the wherewithal to put them in a container, but this knowledge “did not inspire her to action.” In Kirzner’s framing, she had information-knowledge (all the facts), but not action-knowledge (the alertness to act on the information).

Even if you build a system that can collect every production function and every consumer preference in the economy, all you have done is assemble a comprehensive stock of information-knowledge. Alertness is hard to program, because it isn’t a process of inference from known data. A successful entrepreneur acts speculatively in response to a suspected opportunity, which by definition, hasn’t already been recognized.

Hayek went further. He observed that much of the knowledge that matters for economic coordination doesn’t exist at all until the competitive process generates it.

If you are an entrepreneur who tries a new approach, whether you succeed or fail, you have created new knowledge. As a result of the risk you took, people know whether a specific combination of resources, aimed at a set of customers, at a particular price point is viable. This knowledge didn’t exist somewhere in the ether waiting to be discovered by a more powerful algorithm. Instead, it was brought into existence by speculation in conditions of genuine uncertainty. The firm that succeeds reveals the value of its approach retroactively.

The price signals that result are not the transmissions of pre-existing data, but the outputs of a distinct process. The indeterminacy runs deeper still: economic agents are not carrying fixed utility functions. Participation in exchange changes the participants – human interactions themselves generate new kinds of choices. The system a planner would need to model doesn’t hold still, because the process of market coordination is partly constitutive of the preferences and knowledge it produces.

Akio Morita, the co-founder of Sony, launched the Walkman in 1979 against the objections of the rest of the company. Morita had observed that people took large stereos to the beach and listened to music in their cars, and sensed a market opportunity. Sony’s own market research and consumer surveys consistently suggested that there was no consumer demand for a tape player that couldn’t record, no matter how portable it was.

What Morita did was not pattern recognition on a richer dataset. He changed the conceptual space, reconceiving what a music device could be and who it could be for. No amount of data about what consumers said they wanted would have produced the Walkman, because the preference for it was partly a consequence of the product’s existence. The market revealed his conjecture to have value, and it now retrospectively seems obvious. But obviousness after the fact is the signature of knowledge that could only have been created through action.

The launch of the Walkman then had a series of downstream consequences – for competitors, component manufacturers, and music sales alike. The relationship between music and everyday life changed, with enduring social and economic consequences.

If it ain’t broke

Even if many of the foundational technological challenges could be solved by a future system, we would also need to believe that it would be worth the risk. An entrepreneur who bets wrong loses his own capital. A society that dismantles its price system has made an irreversible collective wager. We would need to have confidence that this future model would outperform markets – an order that, given basic institutions, no one has to design.

For the traditional Marxist, the case is straightforward. If technology can finally solve the calculation problem, it vindicates the claim that capitalism contains the seeds of its own succession. But as we saw earlier, the AI alternative to traditional markets increasingly appeals to those who don’t hope for the final triumph of the proletariat.

Oliver Klingefjord and Joe Edelman from the Meaning Alignment Institute argue that advanced AI systems could correct a number of shortcomings in current markets. They believe that markets systematically contract on proxy metrics like hours, subscriptions, engagement, rather than outcomes actually delivered. This is partly because doing so would be prohibitively expensive across millions of consumers, but also because there is an asymmetry of power between suppliers and consumers. Big suppliers can write “take it or leave it” contracts and have huge information advantages versus those that they are contracting with.

Klingefjord and Edelman argue that replacing markets with AI could collapse these measurement and bargaining costs, making it feasible to pay suppliers for delivered benefit via competitive, voluntary AI intermediaries that pool consumers, assess outcomes qualitatively, and negotiate enterprise-level deals.

This is much more sophisticated than technosocialism, but runs into some of the same problems.

This approach tries to maintain a price system, but changes what the prices track. But this entire system relies on an intermediary being able to assess whether a good outcome was delivered. Unlike market prices, which emerge from entrepreneurial bids under genuine uncertainty and personal risks, these assessed prices would reflect a system’s operationalized definition of human benefit. While the user might set the guardrails, the system has to turn these into assessable criteria. In essence, it’s a system’s approximation of a person’s approximation of what constitutes human flourishing.

This would also result in an enormous amount of discretionary judgment embedded in an infrastructure layer that most people would never inspect. While you could try to mitigate this by having a world of competing AI intermediaries, it’s hard to see users choosing between rival theories of their own good – as operationalized by AI systems that they can’t design – as an obvious improvement on choosing between rival products in a traditional market.

Magical thinking

It is, of course, possible to argue that these objections could all be overcome. Maybe we will build systems that can collect all dispersed and local knowledge, model genuine alertness, simulate exchange, and anticipate the outputs of the price discovery process without running it. It may then lead to more efficient resource allocation. But we haven’t built these systems, and nothing in the current trajectory suggests we are close. The burden of proof lies with those who believe otherwise.

The case for planning, by necessity, assumes away all real-world constraints while simultaneously reversing the burden of proof. In response to arguments about the importance of markets, it hypothesizes a system that by stipulation overcomes any individual objection and then challenges opponents to prove that it’s impossible.

Thought experiments that grant one or two premises can be genuinely useful when thinking about advanced technology, but with each additional hypothetical, the value diminishes. Any theory of how the world could work becomes plausible once you assume the existence of an omniscient machine god.

In the end, the CIA didn’t need to worry about Soviet cybernetics. Glushkov’s proposed cybernetics system, budgeted to cost the equivalent of over one trillion dollars in today’s money, never saw the light of day. The Soviet authorities, unconvinced by his argument that the system would pay for itself several times over, balked at the price tag. Cybernetics would not get to play its role in the inevitable triumph of the proletariat.

It seemed that even the Soviets had lost faith in the planned economy. On this, if little else, they were right, even if it was not for reasons that they fully understood. The fundamental obstacle was never processing power or data collection. It was that the economy a planner would need to model is constitutively shaped by the expectations and interpretive frameworks of the people who participate in it. Those frameworks shift in response to the very act of observation and intervention. There is no fixed economy waiting to be measured.

The system that a planner would need to model is the same system that the plan would destroy.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

This question was on our minds as we gathered 120 thinkers and builders from frontier AI labs, top universities, and cutting-edge institutions for the first Cosmos Symposium. Among them were the creator of Fortnite, category theorists, the principal of Alpha School, researchers from OpenAI and DeepMind, and first-time founders building for human autonomy

But as Brendan said in his opening remarks, no one in the room, or indeed the world, is yet a philosopher-builder.

Over the past two and a half millennia, there have only been a handful of true philosophers who pursued fundamental questions with courage and relentlessness. The number of people who have then applied this energy to institution-building that enhances freedom and inquiry for the individual is even smaller; the best example is Benjamin Franklin.

Nevertheless, there was no shortage of ambition. During the event, attendees shared what they were building on a whiteboard. The responses varied significantly, ranging across human goods like strengthened community, augmented intelligence, and a more enriched life of the mind. Underneath this difference in emphasis was a shared belief that we are building towards a greater goal. Everyone in the room saw more capable technology as a means, not an end.

Before the event, everyone nominated a book that shaped their view of the world. Many chose the classics – Plato’s Republic, Smith’s Wealth of Nations – but we also had science fiction, writings from theologian John Henry Newman, and a history of American nuclear power.

To our surprise, one of the most nominated books by attendees was Martin Heidegger’s famously challenging Being and Time. In Being and Time, Heidegger rejected René Descartes’s traditional account of consciousness, which depicts a detached mind peering out at an external world. Instead, Heidegger insisted that we are already caught up in the world, acting on it before we ever step back to think about it. At a time when it can feel like the development of technology is following a fixed path and the best we can do is act as users or spectators, he may have something to teach us.

As only 120 of us could be in the room, we’re sharing a few of our main takeaways from the event.

1. The importance of ‘becoming’

Shortly before Adam Smith died, he ordered that 16 volumes of his manuscripts be burned. He was so anxious that only his best thinking survive that he destroyed the record of his own becoming: the drafts, notes, abandoned arguments we might have learned from

We tried to do the reverse of this at the Symposium by bringing people together at different stages of their journey. This included:

• Highly accomplished builders, who are pursuing mission-driven projects and can act as inspiration, such as Joe Liemandt, the principal of Alpha School or Paul Meegan, the creator of Fortnite and founder of a new stealth venture;

• Former researchers and engineers at leading tech companies and frontier labs, now building for human goods, like , a 15-year Google and Stripe veteran and , a former Anthropic researcher and ex-head of collective intelligence at Midjourney;

• First-time founders who are building for human autonomy, such as Workshop Labs co-founder ;

• World-class intellectuals who are helping to lay the crucial foundations for future builders and thinkers, such as Philipp Koralus at the Oxford HAI Lab, at the Mercatus Center, UT Austin’s Harvey Lederman, Catherine Project founder Zena Hitz, or , founder of the Machine Intelligence and Normative Theory Lab.

All of these groups had something to learn from each other, whether it was hard-won knowledge about what it actually takes to ship something, fresh insights from the frontier of AI research, or clarity about what’s worth building and why.

2. Building the philosophy to code pipeline

Philosophy can feel remote from the realities of shipping a product. It can be easy to default to obvious ideas, like introducing greater personalization through a system prompt. Attendees explored options that show why steerability is essential but insufficient.

These included tools that surface the assumptions a model makes about us, and ways of aligning models with our second order preferences: not just what we want, but what we want to want.

Some of these projects are still in their early stages, so we can’t share too much detail at the moment, but we were grateful to be joined by members of the Cosmos community who are entering institution-building mode. These included Samuele Marro of the Institute of Decentralized AI, Elian McCarron of Kanonic, and Jasmine Li, who is starting an organization focused on measuring and forecasting human agency as AI capabilities advance.

3. Epistemic humility

As one of our speakers observed, autonomy requires humility. The philosopher-builder will need to constantly ask whether they are doing the right thing and course-correct when the answer is no.

from the Mercatus Center recently wrote an essay on how the same evidence in frontier AI debates can justify sharply conflicting hypotheses. Despite all of this talent in one room, none of us left Austin feeling certain.

The discussions raised vital questions that lack a definite answers:

• How can we build for second order preferences without defaulting to paternalism?

• How can we prioritize human autonomy, while using capital markets to reach hundreds of millions of people?

• What are the limits of decentralization? What high-level coordination or condition-setting is needed to create useful, coherent action in the real world?

Some debates, especially around frontier AI, can feel tribal, which is why it’s important for communities to build strong norms. At the symposium, we saw people treat conversations as conversations. Each interaction felt like an opportunity for two people to learn, and we received no reports of distillation attacks.

4. Structure and spontaneity

Nietzsche thought Greek cultural greatness stemmed from the fusion of the Apollonian and the Dionysian. Apollo, the god of light, represented clarity, structure, and rational boundary-making. Meanwhile, Dionysus, the god of wine-making, represented the dissolution of those boundaries.

In our own modest way, we attempted to replicate this formula. Alongside our structured programming, attendees had time to mingle over drinks, dinner, and across the sprawling, beautiful grounds of the venue. The best conversations were often the spontaneous ones, still running late on Sunday morning, long after the formal programming ended.

This was the first Cosmos Symposium, but it won’t be the last. The Cosmos community is just over a year old and we’re grateful to everyone who has contributed to what we’re building.

If you’re interested in attending a future Cosmos event, register your interest on our seminars and events form.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

If you’re someone who thinks deeply, builds deliberately, and cares about the future AI is shaping—join the Cosmos network.

Subscribe now

John Graunt was a merchant who lived in 17th century London. In 1662, in a city familiar with plague, Graunt consulted parish death records to figure out how long Londoners could expect to live. He could tell you that roughly 36 out of every 100 children born in the city would die before their sixth birthday. This probability could be checked by anyone willing to do the same sums using the same sources.

Three and a half centuries later, AI researchers are calculating a morbid probability of their own. They wonder about thinking machines seeing off the human race, and refer to the likelihood of that possibility as P(doom). The term began life as an inside joke in rationalist spaces in the closing years of the 2000s, but shot to prominence after researchers signed an open letter about the threat posed by AI development in 2023. In the AI world, everyone has their number. Yudkowsky puts it at roughly 95–100 percent. Geoffrey Hinton sits around 10–20 percent, while Marc Andreessen reports a bullish 0 percent (a logical impossibility because you cannot rule out something that has never happened and whose conditions have never been tested). The mean estimate among surveyed AI researchers is about 14 per cent.

We have some who believe that AI is almost certain to wipe out humanity and others who ascribe a 0 per cent probability to the same outcome. This is curious for a supposedly hard-headed prediction made by some of the world’s most credentialed scientists. Imagine a roomful of cardiologists who, given the same scans and the same patient history, exposed to the same training and the same clinical standards, disagreed on whether the probability of heart failure was 0.1 percent or 99 percent.

You would not conclude that this was a hard problem on which reasonable people differed. In fact, you would probably conclude that at least some of our medics may need to visit neurology down the hall. With P(doom), the estimates from AI researchers range from 0 percent to 99 percent (and a few nines then some). When someone tells you they assign a 15 per cent probability to AI wiping out humanity, it is worth asking what kind of claim they are actually making. It sounds like a statement about the world. It is not.

To be clear: my objection is not to probabilistic reasoning, which is straightforwardly indispensable in domains where estimates can be calibrated against outcomes over time. My problem concerns what we might call proximate and ultimate formulations. Proximate estimates for COVID were things like the hospitalization rate and case severity, while an ultimate estimate was the infection fatality rate. For P(doom), proximate estimates include the conditions under which models exhibit deceptive behavior and the observed failure rates of current alignment techniques. Here, the ultimate estimate is the thing itself.

Someone might know that current models can be made to exhibit deceptive alignment under laboratory conditions or that existing alignment techniques have specific failure modes. But to cram those findings into a container labeled “15% chance of extinction” involves a series of judgments about unprecedented transitions for which there is no model connecting proximate estimates to their ultimate counterpart. Knowing that fine-tuning on specific tasks can induce broader misalignment tells us little about how well alignment solutions will generalize or how geopolitical actors will respond. Compare with COVID, where hospitalization and case-severity data fed epidemiological models that could be tested against actual deaths.

A natural objection is that some people are good at formulating predictions in lots of different domains, so perhaps their assessment of P(doom) ought to carry weight. After all, some superforecasters assign 17 per cent to various events and see those events happen roughly 17 per cent of the time. You can check a forecaster’s guesses across many different predictions in such a way as to mean we’re dealing with a property of the forecaster’s overall policy rather than of any individual estimate. When a superforecaster says 17 per cent for a ceasefire in some conflict, that number is useful because they also say 17 per cent for hundreds of other things and roughly 17 per cent of those things happen.

Superforecasters earn their calibration (that is, the extent to which confidence lines up with their track record over time) using predictions with short time horizons, a history of similar events to draw on, and a mechanism that allows the forecaster to correct course when needed. For P(doom), there is no source of comparable predictions against which to calibrate. A forecaster might be superbly calibrated on elections and geopolitical high drama, but that tells us nothing about whether their number for an unprecedented event is any good.

What I Talk About When I Talk About P(doom)

There are two influential interpretations of probability: subjective and objective. Where the former deals with how confident you are in a claim, the latter stresses what actually happens when you test it. Anyone who has spent even a little time thinking about the reliability of P(doom) metrics probably has a sense that the calculations ought to be taken with a pinch of salt. Many in rationalist or rationalist-adjacent circles are well aware of the difference between these two accounts, but lots of people who absorb these views are not.

That goes for those who adopt the terminology to signal their membership of the in-group, and those who hear talk of P(doom) on the news and wonder why one of the world’s most cited researchers thinks there’s a one in five chance that AI will wipe out the human race. When a chief global strategist at one of the largest investment research firms tells CNBC it’s 50/50 whether AI destroys humanity by mid-century, the average viewer — incorrectly but understandably — assumes that figure estimates the likelihood that it will happen.

Every single P(doom) you hear is a subjective probability. It is a measure of the degree of rational belief one holds in a proposition given the available evidence — a property of an agent’s epistemic state rather than of the world. When you say “I think there’s a 50 per cent chance this meeting will be a waste of time,” you’re not drawing on a frequency table of past meetings. You’re expressing how confident you feel given what you know about the agenda and who’s attending. All the framework requires is that your degrees of belief are consistent with one another. Beyond this internal consistency, there is no further requirement that your probabilities correspond to the shape of reality.

If your beliefs are internally coherent, you can multiply your probability for any outcome by the value of that outcome — however you choose to measure it — and arrive at its expected value, a single number that tells you how good or bad a bet looks on average. From there you can derive a utility function and compare any set of actions on a common scale. The framework of cause prioritization, wherein resources are allocated to whichever cause produces the greatest expected good per dollar, inherits this apparatus. This approach is popular with rationalist-adjacent communities, especially Effective Altruism, because it transforms questions of moral life into properties that can be measured and optimized.

Objective accounts of probability take a different approach. Here probability is what actually happens in the world when you repeat something many times. This species of probability is a physical property of systems, one that is empirically testable. If you flip a coin a thousand times roughly half will land heads, so we can say the probability is that ratio. The same logic applies to complex phenomena, like radioactive decay, where about half the atoms in a given sample will decay within the half-life period (and you can go and check).

The subjective interpretation is internally coherent and mathematically elegant. But coherence should not be confused for empirical content. My objection to P(doom) flows from the simple fact that subjective probability claims are, by their nature, unfalsifiable. That’s generally fine in many instances — like casual everyday judgments or betting on the Super Bowl — but not when the stakes are high enough to redirect billions of dollars and make or break government policy. Statistical reasoning about unlikely but potentially devastating scenarios is obviously useful, but only when properly grounded in claims we can falsify.

Your Problem Too

For a probability to tell you about something other than the speaker’s state of mind, it needs a collection of similar events from which a ratio can be drawn. To say that a coin lands heads 50 percent of the time, we need a set of coin flips to draw from. This is our reference class.

Clearly there is no reference class for human extinction. As far as every AI researcher in the world knows, the event is singular and unprecedented. Traditionally, the reference class problem has been associated with the objective interpretation of probability. This is because objective probability is calculated according to the ratio of how many times something happened out of how many times it might have happened. If the event has never happened and can’t be repeated, then there’s nothing to compute.

But the reference class problem isn’t only relevant for objective accounts of probability. Alan Hájek famously made the case that it affects every interpretation of probability when applied to singular events. As he put it, “the reference class problem is your problem too.” The argument goes something like this: before you can assign a probability to a one-off event, you have to decide what kind of event it is. Is AI-caused extinction a case of “new technology going wrong,” “species-level catastrophe,” or “unprecedented transition in the nature of intelligence”? Each framing suggests a different probability, and there is no principled way to choose between them. The number you arrive at depends on how you describe your target.

The way to tell a useful probability from a useless one is to ask whether the world can correct it.

This is because good explanations resist variation. If Graunt had doubled his estimate of childhood mortality, the parish registers would have corrected him. If an epidemiologist in early 2020 had put the infection fatality rate at 10 per cent rather than 1 per cent, the incoming hospital data would have shown the estimate to be wrong. The explanation is enmeshed with the world in such a way that it cannot be led to wherever you would like it to go. With P(doom), you can swap the causal pathway from deceptive alignment to resource competition and inflate the number from 5 per cent to 7.5 per cent. The estimate can contort to accommodate any figure and any causal chain because the only constraint it faces is internal coherence.

In the context of P(doom), defenders might say something like “We don’t need a reference class because we’re expressing a degree of personal belief.” This is perfectly coherent on the subjective view, but it is also the move that severs the claim from its empirical content. If your probability doesn’t need to correspond to any feature of the world, then it can’t be wrong about the world. And a claim that can’t be wrong about the world tells you nothing about it.

We might call the result a Gettier probability, a credence that is internally justified — and that might even correspond to reality — but that lines up with the world through luck rather than judgement. In epistemology, a Gettier case is a belief that is true and justified but not true because of its justification. Any P(doom), where the justification and the truth are aligned, will (assuming the estimate was correct) have the same structure. A researcher’s estimate might happen to match whatever the actual risk turns out to be, but the match would be accidental relative to the method.

Gettier probabilities show up everywhere, like when a consulting firm estimates that a given market will grow at 7.3 per cent and lo and behold they are correct. The same is true for a geopolitical risk score that looks precise but cannot be updated by any observation short of the catastrophe it purports to predict. P(doom) is distinctive only in that the stakes are high enough that the probability escapes containment and ends up on the morning news.

Suppose someone’s P(doom) is 30 percent. What outcome would show this was wrong? If AI goes well, they get to enjoy the benefits of the 70 percent figure. If it goes catastrophically — and anyone is still alive to update their priors — the 30 percent gets paid off. This is one of the more frustrating features of the subjective framework, which allows pretty much anyone with an idea about AI risk to have their tokens and eat them.

Bayesian probability demands that credences be precise. When estimates range from 0% to 99% with no mechanism to adjudicate, the framework is not being applied so much as invoked. P(doom) manages the impressive trick of satisfying neither the Bayesian standard (precise credences that can be adjudicated between) nor the falsifiability standard (empirical claims that can be proved wrong).

I’m not arguing that we ought to stop reasoning about novel events. Subjective probability is useful because it helps us think under uncertainty without good data. Some of the best calls in recent AI history were made by researchers who formed strong priors about scaling laws and bet on them before the evidence was in. Those bets were vindicated because they were testable. Scaling laws either held or they didn’t; capability thresholds either arrived on schedule or did not. My concern is that P(doom) borrows the confidence of claims like these without sharing their vulnerability to refutation.

I Am Sure You Are Very Sure

Forget, for a moment, whether P(doom) can be tested. Consider instead whether it is the kind of thing that can be meaningfully quantified in the first place. In 1921, the economist Frank Knight drew a distinction between risk and uncertainty. Risk is what you face when you can calculate the odds as with flicking a small ball onto a roulette wheel. Uncertainty is what you face when you can’t, even in principle, because the situation is too poorly understood to yield a number.

Knight argued that the two are different in kind.

The Bayesian tradition’s response to Knight was to argue that subjective probability does away with the distinction: if you can always express a degree of belief, then there is no situation that resists quantification. P(doom) is a textbook application of this move, but the dissolution works only if the resulting number can be tested via real-world feedback. For P(doom), the Bayesian move converts Knightian uncertainty into a figure that has the form of risk without any of the mechanisms that make risk estimates trustworthy.

The same process, where uncertainty becomes risk, happens wherever a decision-making framework demands a numerical input and none is available. Firms routinely assign precise probabilities to market scenarios that are genuinely unprecedented. They don’t have any evidence that warrants a specific number, but they do have a spreadsheet with cells that need filling. The defense that such figures are “rough heuristics” does not help. The problem is that they are treated simultaneously as a casual shorthand and as inputs into value calculations depending on whatever is most convenient. This is why the “shmrobability” move doesn’t fix our problem. It concedes we’re not really talking about probability, while trying to preserve its practical content. Either the number is a rough heuristic and we should stop plugging it into expected value calculations, or it is a serious input to policy and we should hold it to the standards that implies.

If P(doom) is a subjective credence, there is no principled way to adjudicate between two people who disagree. A P(doom) of 95 per cent and one of 50 per cent are both internally coherent insofar as neither violates any rule of the subjective framework. You can be confident in your P(doom) and I can be confident in mine, and there is no observation or outcome that could settle which of us is right. Attempts to resolve this from within the Bayesian model, from Aumann’s agreement theorem onwards, all founder on conditions that do not hold (like shared starting points, perfect rationality, or common knowledge of each other’s beliefs).

The problem compounds when these estimates enter expected value calculations, where even a vanishingly small probability multiplied by the stakes of human extinction produces numbers large enough to dominate decisions. The rationalist and rationalist-adjacent communities recognize this dynamic as Pascal’s mugging, while groups like GiveWell have cautioned against taking such estimates at face value.

The reader should not take from this essay that AI risk is unworthy of attention. Rather, my point is that P(doom) is the wrong epistemic instrument for thinking about it. The alternative is to formulate falsifiable conjectures about the world and then try to refute them, which is exactly what AI safety researchers do in their actual work. They propose that particular training regimes will produce deceptive behavior and they predict that models beyond a certain capability threshold will resist correction. They hypothesize that misalignment in one domain will generalize unpredictably to others, and they can test this by measuring the transfer of learned strategies across task distributions. They ask whether models that pass safety evaluations in deployment conditions will behave differently when those conditions change. Each of these lines of research produces claims that can fail, and this vulnerability is what makes them worth paying attention to.

Individual claims about deceptive behavior or capability thresholds do not by themselves tell you how much to spend on alignment versus pandemic preparedness, yet real institutions have treated P(doom) as though it could. The effective altruist community has directed hundreds of millions of dollars toward AI existential risk reduction. Open Philanthropy, the field’s largest funder, used subjective probability frameworks to weigh AI risk against global health, biosecurity, and farm animal welfare. Whether or not that assessment was correct, the reasoning behind it is only as good as the numbers it relies on.

A figure that ranges from 0 per cent to 99 per cent depending on who you ask and with no way to adjudicate between them is about as useful as no number at all. Aggregating these estimates might seem to help inasmuch as a mean of 14 per cent looks more sober than bouncing between extremes. Aggregate or individual, the problem is that neither incur a cost for being wrong. A bookmaker’s odds are constrained by the market just as an insurer’s premiums are shaped by the frequency and nature of claims made. In both cases, the issuer’s assessment improves over time because they suffer when it is incorrect. There is no market to punish a mispriced P(doom) and no settlement date on which the estimate is tested.

We might say something like: “Even if P(doom) is imprecise, it’s still better than no number because we need to allocate resources somehow.” But here we’re assuming that internal coherence still outperforms the absence of a number when you have to act. We can live with imprecision under these conditions, but not at the expense of claims that cannot be falsified that get steamrolled by the headline rating.

Governments fund pandemic preparedness efforts without a big round number for the probability of the next pandemic. Institutions allocate resources under genuine uncertainty all the time by doing things like funding a portfolio of approaches, setting thresholds for observable harms, identifying the cheapest reversible interventions, demanding observable milestones before scaling commitment, and building the capacity to pivot as new evidence arrives.

The physicist Wolfgang Pauli was famously unforgiving of bad theory. When a colleague asked him to assess a young physicist’s paper, his verdict was that it was “not even wrong.” He meant that its central claim was so removed from evidence that it could not be proved false. P(doom) is not even wrong. Graunt’s mortality tables may have been technically crude, but they were accountable. They could be checked against next year’s parish records, and if they were wrong, the records would show it.

Imagine a system that knows you eat sensibly all day and fall apart at 10pm. That knows a calorie label won’t help you, because the evidence says calorie labels mostly change the behavior of people who already eat well.

This system knows the difference between your hunger and your boredom. It leaves you alone when you don’t need it. It preserves every option on the menu. It just rearranges the choice so you’re more likely to do what you’d do if you were thinking clearly.

Now extend this to every significant decision you make. Cars, insurance, investments, medical treatment, career moves. A system that corrects for your specific biases, knows what you’ll regret, and steers you toward what it calculates you really want. The full menu is technically available, but the system is right often enough that overriding it starts to feel like pride, and convenient enough that you stop wanting to.

Cass Sunstein—co-author of Nudge, former White House regulatory czar, the most influential regulatory thinker of his generation—calls this a “Choice Engine.” His arguments tend to become law. I think the Choice Engine’s greatest danger is that it works.

His new paper “Liberal AI,” prepared for a lecture at Oxford this May, makes the strongest case yet that AI-powered choice architecture can respect autonomy while improving welfare. The logic runs like this. Classical liberalism, following Mill and Hayek, grounds freedom of choice in an epistemic claim: the chooser knows best. AI disrupts that claim by knowing better. Better than the individual about her own informational gaps, her biases, what she’ll want next year. So Choice Engines can personalize the nudge, correct the bias, and never technically remove the freedom to choose otherwise. Liberal AI is no oxymoron.

Sunstein flags the risks. He talks about manipulation, self-interested designers, AI’s own biases. He calls for regulation. He distinguishes liberal Choice Engines from illiberal ones. But even the liberal version is steering people toward choices they wouldn’t have made on their own. Sunstein knows this is paternalism. He thinks preserving opt-out keeps it libertarian rather than something worse. I argue that it cannot.

Mill’s Music

Sunstein’s Mill is an information theorist. In On Liberty, Mill argues that each person is “the person most interested in his own well-being,” possessing “means of knowledge immeasurably surpassing those that can be possessed by anyone else.” When outsiders intervene, they rely on “general presumptions” that are likely wrong or misapplied.

On this reading, the case for freedom of choice is instrumental: people should choose for themselves because they know their own circumstances best. If the case for freedom rests on the chooser’s informational advantage, and AI erases that advantage, then the case for freedom weakens. AI knows your medical history better than you remember it, your spending patterns better than you track them, your recurring mistakes better than you admit them. The epistemic ground has shifted. So far, so honest.

But Sunstein himself notices something that should give him pause. He observes that “the words of On Liberty are epistemic, but the music is romantic.” Then he proceeds to work exclusively with the words.

The words say: people should choose freely because they know best. The music says something else entirely. It says that the process of choosing is what makes a person a person. Mill:

The human faculties of perception, judgment, discriminative feeling, mental activity, and even moral preference, are exercised only in making a choice. He who does anything because it is the custom, makes no choice. He gains no practice either in discerning or in desiring what is best. The mental and the moral, like the muscular powers, are improved only by being used.

This is an argument about formation. Mill is not saying that choosers make better decisions (the information argument). He is saying that choosing makes better choosers—people with developed faculties of perception, judgment, and what he calls “discriminative feeling,” the ability to tell the difference between what matters and what doesn’t. These faculties, like muscles, grow through exercise and atrophy through disuse. The quality of any particular decision matters less than what the process of deciding does to the person who decides.

Once you hear the music, Sunstein’s proposal sounds different.

Consider a person who consults a well-designed Choice Engine for a car purchase and accepts its recommendation. She may end up with a better car than she would have chosen on her own. Sunstein counts this as a welfare gain.

But if the evaluative work was done by the Engine, if she did not weigh the tradeoffs, interrogate her own priorities, or struggle with the uncertainty of not knowing what she really wants, then her faculties of judgment were not exercised. She received an outcome without undergoing the process that develops the capacity to arrive at such outcomes independently.

Do this once and the effect is trivial. Do this for every significant choice across a decade, and you have a person whose formal freedom of choice is perfectly intact, whose welfare as measured at each decision point is maximized, and whose capacity for independent evaluation has atrophied through sustained disuse. She can always opt out, but she no longer has the developed faculties that would make opting out a meaningful exercise of judgment.

Mill has a name for this person. “One whose desires and impulses are not his own, has no character, no more than a steam-engine has a character.”

She may be doing well by every metric Sunstein tracks. But the evaluative standards by which “well” is determined have migrated from her judgment to the algorithm. She is content without being, in any sense Mill would recognize, free.

Half of Hayek

Hayek’s case for markets rests on a claim about knowledge: both that markets aggregate it efficiently, and that participation in institutions like markets is what produces it.

Sunstein takes the first half and drops the second. He reads Hayek as making an argument about information processing, where markets aggregate dispersed knowledge more effectively than central planners. If AI can aggregate that knowledge even more effectively, the Hayekian argument for leaving people alone weakens.

But the knowledge Hayek cares about is not the kind of thing that can be aggregated. The knowledge of “the particular circumstances of time and place” is not sitting in a warehouse waiting to be collected. It comes into existence through the activity of people navigating uncertainty, bearing consequences, and adjusting. This is not a computational problem.

The structure in which choosing happens is itself doing something irreplaceable. Remove the individual from the process and you do not just lose one person’s contribution. You lose the conditions under which that person would have developed the knowledge worth contributing. The loss compounds.

AI would have to live your life to fully replace your epistemic role in it. But partial replacement may be enough to leave you unable to perform it yourself.

When Hayek writes that coercion “eliminates an individual as a thinking and valuing person and makes him a bare tool in the achievement of the ends of another,” Sunstein reads this as an argument against coercion, which Choice Engines avoid because they preserve opt-out. But read the sentence again. The operative phrase is not “coercion.” It is “eliminates an individual as a thinking and valuing person.” That elimination can happen without coercion. It can happen through substitution, through a system that does the thinking and valuing for you so effectively that your own faculties are never called upon.

Hayek’s nightmare was central planning, but his deepest fear was not the planner’s malice. It was the replacement of a distributed process of discovery with a system that delivers answers without requiring anyone to find them. Choice Engines are that system. No one is coerced, and no one means harm. And yet it is Hayek’s nightmare, built by people who thought they were honoring him.

The Missing Axis

Sunstein’s paper operates on a single evaluative dimension. Every intervention is assessed by two criteria: does it improve welfare, and does it preserve freedom of choice? Both are real goods, but they sit on the same axis. Call it agency: the capacity to get what you want, measured by outcomes and options.

The axis Sunstein does not have is autonomy. Not autonomy in the loose sense of “having choices,” which is just agency again. Autonomy in the strict sense: self-rule. Authorship of the evaluative criteria by which options are assessed. A person has agency — agere, to act — when she can get what she wants. She has autonomy — auto nomos, self-rule — when she determines what to want, through her own exercised faculties of judgment in Mill’s terms, or through her own participation in the knowledge-generating processes Hayek describes.

These come apart. AI can increase agency while eroding autonomy. A Choice Engine that tells you what is best for you may be right. It may get you better outcomes. But if the standard by which “best” is determined has migrated from your judgment to the Engine, your autonomy has decreased. You are more powerful, but you are less free.

Sunstein’s paper treats both rows as one. He assumes that improving agency while preserving the option set is sufficient to protect autonomy.

His Choice Engines produce movement downward. Into the lower left. Claude Boys are powerful without being self-governing, formally free at every point, and yet not authoring the evaluative criteria by which a life is organized. Silicon Valley’s “you can just do stuff” brigade lives here, though it doesn’t know it.

This quadrant is easy to miss because it doesn’t look like a problem. It looks like liberal AI working exactly as designed. And all the while, the capacity that makes opting out meaningful is eroded, because it is no longer required.

Constitutional Drift

The phrase that does all the work in Sunstein’s paper is a counterfactual. Choice Engines steer people toward what they would choose if they were “adequately informed and free from behavioral biases.” The Engine corrects for departures from this idealized chooser.

But who defines the ideal? Whether something counts as a bias or a preference, adequate information or noise, a self-control failure or a legitimate present-oriented value: the answer, necessarily, is the Choice Engine. Which means the Engine is authoring the normative standard against which the person’s actual choices are evaluated. The person experiences this as help. She believes she is getting closer to what she really wants. But “what she really wants” is a construction of the system, not an independent fact about the person.

This is Constitutional drift, the migration of priority-setting from the person to the system. It is experienced from inside as self-improvement. The person feels she is becoming more rational. She is becoming more aligned with the Engine’s model of rationality. These feel identical from the inside, but may not be the same thing.

And the drift compounds. At time one, the person’s preferences are her own. She consults the Engine, accepts its recommendation, and her welfare improves. At time two, her preferences have been partially shaped by her previous Engine-assisted choices. By time ten, the convergence is nearly complete. The Engine recommends what she wants and she wants what the Engine recommends. The idealized chooser, “adequately informed and free from behavioral biases,” has become the Engine’s model of her rather than anything she authored herself.

This convergence is a ratchet. The more the Engine outperforms her independent judgment, the more rational it is to defer. The more she defers, the less her capacity is exercised, and the more the Engine outperforms her. At every point in this cycle, Sunstein’s framework reports success. It cannot represent the difference between a person whose welfare is high because she exercises excellent judgment and a person whose welfare is high because an Engine exercises excellent judgment on her behalf. They are in opposite positions with respect to their own freedom.

There is early evidence this is already happening. Bastani et al. (2025) found that high-school students using ChatGPT scored 48% higher on practice assignments and 17% lower on exams taken without it. The work improved but the learning did not. Becker et al. (2025) found that experienced developers using AI coding assistants were 19% slower on tasks while believing they were 24% faster, a 43-percentage-point gap between perceived and actual performance. Fernandes et al. (2025) reported that higher AI literacy correlated with worse metacognitive accuracy about one’s own AI-assisted performance. The people most embedded in AI-assisted work were most wrong about what the assistance was doing to their independent capacity.

The habit of deferring to a well-designed Engine is the same habit a manipulative one exploits. The person whose evaluative capacity has diminished through years of benign delegation is the person least equipped to detect when an Engine shifts from serving her interests to exploiting them. Sunstein’s liberal AI creates the prey that his illiberal AI hunts.

The effect will not be evenly distributed. People with strong habits of deliberation will use Choice Engines as tools, retaining evaluative authority while extracting information. People without that formation will not supplement their judgment but cede it. These populations map onto existing inequalities in parenting and education. The structural effect of Choice Engines may be to concentrate the capacity for self-rule in those who already have it. This is the deepest inequality, and the least measurable.

The Tutelary Power

Sunstein frames his paper with Mill and Hayek. The thinker he needs, and does not cite, is Tocqueville. In the final pages of Democracy in America, Tocqueville describes a tutelary power that

takes charge of assuring their enjoyments and watching over their fate. It is absolute, detailed, regular, far-seeing, and mild. It would resemble paternal power if, like that, it had for its object to prepare men for manhood; but on the contrary, it seeks only to keep them fixed irrevocably in childhood… It does not destroy, it prevents things from being born.

The Engine does not coerce and it does not break wills. It softens, bends, and directs. It seeks not to prepare persons for independent judgment but to keep them, gently and perpetually, in a state of epistemic childhood. The capacities at risk here are not taken. They go unexercised until they are gone.

Every argument in “Liberal AI” presupposes an author with intact evaluative capacity. Sunstein weighs Mill against Hayek, assesses the evidence on nudges, evaluates the risks of manipulation, reasons about what would constitute genuinely liberal AI. He is arguing from inside a house his proposal would slowly empty. But the capacities that make his paper important are the capacities that a generation raised on Choice Engines may never develop, because the Engines are so helpful, so liberal, and so respectful of freedom that the difficult and sometimes painful process of developing independent judgment is no longer necessary.

Mill understood this. The worst threat to liberty is not the tyrant who forbids you from thinking. It is the benevolent power that makes thinking unnecessary.

Sunstein’s Liberal AI is not an oxymoron. It is something more dangerous: a coherent program that achieves everything it promises while producing persons who can no longer do for themselves what it does for them. A generation raised on Choice Engines will never develop the evaluative capacity that was, in all prior human experience, the unavoidable byproduct of having to choose. We are building the conditions for that generation now.

The paper asks whether liberal AI can make life more free. It cannot ask what kind of persons will be left to be free, or whether freedom, for such persons, will mean anything at all.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

In 1972, researchers offered children from the Bing Nursery School at Stanford a simple choice. Left alone with a single marshmallow, they were told they could eat it now or wait fifteen minutes and receive two instead. Years later, the scientists tracked down the original participants to find that children who had waited longer at age four seemed healthier, more sociable, and even higher performing at school.

Gleefully amplified by journalists and pop psychology gurus, a popular interpretation held that the children already had certain traits that predicted how their lives might unfold. Regrettably for the original group, the results – as is often the case in the social sciences – didn’t stick. In 2018, a new team of researchers conducted another study with a similar task and long term follow-ups. Drawing on a much larger dataset and controlling for confounding variables like parental education or household income, they found only a mild correlation between waiting time and positive outcomes.

What gives? The answer turns on the extent to which you think the self is made or given. The original interpretation assumed self-control was innate rather than learned, but the replication seemed to show that a child’s capacity to wait depended on background conditions (and so could not be wholly inborn after all). A child from a chaotic household, where promises are routinely broken, has good reason not to trust that the second marshmallow will materialize. Their failure to wait reflects a reasonable inference.

If this is correct, then the capacity to resist immediate gratification in favor of longer-term goods may be cultivated over the course of a life. This seems true enough for anyone who has learned an instrument or a new sport, where you start off having a bad time but stick with it until you eventually get better. That’s not to say we are all blank slates, but rather that the persons we become are in part the product of environment and experience. The ancient Greeks had a term for what happens when the child reaches for the marshmallow despite knowing she should wait: akrasia, sometimes translated as “weakness of will.”

Kramer vs. Kramer

Akrasia presupposes that self is divided and choice collapses at the moment of action. There are multiple sources of motivation pulling in different directions, though they are not made equal. The agent identifies with the judgment and recognizes it as “better” or more wholly theirs, whereas the impulse is deemed alien or at least insufficiently representative of the person they want to be. The defining feature of akrasia is that the judgment persists even as you act against it. You know you’re failing which is why you lament that you “knew better.”

Three conditions must hold for this self-opposition to occur:

• There must be some standard to act against. The agent must believe that some alternative is better. Without this prior judgment formed through experience and reasoning, acting on an impulse is neither virtuous nor vicious.

• The agent must be aware of deviation. The agent knows, in the moment or shortly after, that they have failed by their own standard. This is why akrasia is accompanied by regret of having done what one resolved not to do.

• There must be some struggle involved. The agent who resists effortlessly by definition cannot exercise self-control. But the person who gives in and the person who resists both experience a form of temptation.

These conditions remind us that temptation is in some respects good for autonomy. This is an old idea, articulated by the likes of Aristotle and the ascetics, that takes the moments in which we are tempted as the moments where autonomy is strengthened. To resist is to become resistant and to abstain is to become abstentious.

Today’s temptress is AI. It’s on hand to help us with tasks big and small, from compiling a Twitter thread to vibe-coding that app you never got around to working on. Most of the time that’s great. There have been a few projects that I quite simply would have never done without Claude. And yet Claude tempts me to summarize the paper rather than read it or to automatically add notes to my Notion when I finish a new book instead of puzzling out what it meant to me.

I think about AI, fantastically useful as it is, as a kind of meta-temptation: a temptation to remove the conditions under which ordinary temptation occurs. This is different to a kind of “digital akrasia” where, for example, you get notifications from Uber Eats to order something which might not have otherwise been done. Pizza for supper instead of that ensalada caprese you were planning.

This kind of effect is less interesting to me, and doesn’t really have anything to do with AI-as-meta-temptation. The more compelling instance — where we skip the need to form our own judgments — does. AI temptation encourages us to stop deliberating, which is worrisome precisely because deliberation might let us see temptation for what it is. Consider two types of akrasia: clear-eyed and means-end.1

In clear-eyed Akrasia, you see the situation accurately and fail all the same. You think “I shouldn’t smoke” while lighting the cigarette. Afterwards you feel guilty because you saw clearly and failed anyway. In means-end Akrasia, you cloud your own vision. You still believe “I should be healthy” while telling yourself a story that disconnects this action from that judgment. You say things like “one cigarette won’t hurt” or “I’ll quit next month” so you don’t have to see this particular instance as a violation of your own standard.

AI temptation is distinctively means-end akrasia, but it goes further because it short-circuits opportunities for deliberation altogether (the situations in which we either strengthen our autonomy or succumb to akrasia). When a person convinces themselves “AI is just a tool!” or “the core ideas are mine!” or “I’ll review it anyway!,” she disconnects its use from the need to do her own thinking. That would normally be plain old means-end akrasia, except that a) the thing you’re outsourcing here is deliberation and b) deliberation is the capacity you need to see your rationalizations clearly.

When you ask Claude or ChatGPT to draft an email, you haven’t yet decided what you think. You are tempted to outsource whatever thinking was required and, assuming you just hit send without re-reading, don’t weigh its content. The obvious response here is something like: fine, but if I review, edit, or rewrite then aren’t I still exercising judgment? This example only works for extreme passivity, and lots of people don’t just hit send and move on.

The difference is that when you review an AI draft, you tend to be evaluating rather than generating. This is the basic idea of writing-as-thinking, which holds that blank pages are there to help further your understanding rather than be filled with size eleven arial. Would you have written the same email? Maybe. But you didn’t muddle through the process that might have surfaced alternatives, so you’ll never know.

Granted, it is difficult to say what counts as “generative” in this context. One such scenario might see a person think about a given topic and weigh the tone and key points to make. Your prompt might be as specific as possible and leave very little room for the model to go off-piste. But finding the right word or rhythm isn’t only about getting the execution right. “I’m sorry but I can’t make it” versus “unfortunately I won’t be able to attend” versus “I wish I could be there, but” aren’t necessarily fungible. Even in the low stakes messages, I often figure out what I really mean through the act of fumbling around for the right words.

More than that, for all of the above I’m assuming the lowest stakes kind of writing and the maximal amount of resistance on the part of the author. If you always use AI like this then you must have a kind of saintly disposition. With longer or more complex writing it’s much more tempting to take the easy way out, which is what many people do much of the time.

This argument, the one spelled out over the last few paragraphs, is a very modest instance of exercising deliberative capacity. Let’s say I thought about using Claude to make this case for me. We have two kinds of decisions to make:

1. Do I use Claude or not?

2. If so, how much do I engage with Claude’s output?

Temptation operates at both of these levels, with the former a simple binary (use or don’t) and the latter a gradient consisting of things like prompting more carefully, reviewing more diligently, engaging for longer, and calibrating the threshold at which you’re willing to accept “good enough.” The person who rewrites is tempted to revise, the person who revises is tempted to edit, and the person who edits is tempted to skim. At every point along the spectrum we’re being tempted with another opportunity to do less.

Athletes of God

In the fourth-century, the Christian philosopher Anthony the Great withdrew to the Egyptian desert in search of discipline. The desert was a gymnasium, a place where you trained the will by struggling against it. These ascetic monks called themselves “athletes of God” because they believed that the conditions that allow us to err also allow us to stay the course and become stronger for it.

The Desert Fathers thought that temptation was necessary for growing into the people we want to be. They chose to do the hard things because they were hard. Today, doing the hard thing is less necessary than ever. Unlike the ascetics, who chose once and lived with it, technology continually presents us with ways of doing less.

A clear-eyed akratic at least knows she is falling short. A means-end akratic, aided by a tool that makes the rationalization feel reasonable, does not. Meta-temptation is a distinctively means-end form of akrasia that allows you to convince yourself that Claude or ChatGPT was only helping, one that goes further because you outsource the faculties you need to see through these post hoc rationalizations.

No one wakes up one morning and decides to stop thinking. We start by outsourcing the stuff that doesn’t matter and discover, weeks or months later, that the boundary between trivial and meaningful is getting harder to spot. The email becomes the memo becomes the proposal becomes the argument. Each step is small enough to rationalize and the rationalization is always the same: one more time can’t hurt?

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Every builder’s first duty is philosophical: to decide what they should build for. This series asks 9 questions to founders who are building towards their vision of the human good.

Today’s guest is Zena Hitz. Zena is the founder of Catherine Project, which builds communities of learning through online courses and reading groups.

Zena is also a Tutor at St. John’s College and the author of Lost in Thought: The Hidden Pleasures of an Intellectual Life.

1. What are the core questions or beliefs driving your work?

I founded Catherine Project because I had the sentimental idea that learning and thinking for its own sake is a basic human need. Since I've founded it, and we've seen thousands of people come to us, I believe it is actually true. Our educational institutions have largely forgotten this fact, if they ever knew it.

The other reason I wanted the Project to be open to anyone who was interested was to see for myself how much interest there really is. My conclusion: there's a lot. I decided not to charge tuition, because learning only happens when the learner decides to do it. It can’t be bought, even if some opportunities for it can be. If we charged tuition, they’d expect us to do their learning for them.

2. What future are you building for?

I can see two futures. In one, colleges and universities lose their last remnants of serious learning, and organizations like Catherine Project keep practices of serious learning alive for future generations.

In another future, organizations like mine display the enormous demand for liberal education that all of the educational leaders have been denying. That drives the old institutions to measures of reform and so colleges and universities return (or partially return) to their original mission.

3. What commonly held belief in the tech community do you believe is wrong?

Everyone seems to want “scale” and “impact” and seems to think that means having a massive operation. But doing something right on a small scale breeds imitators—it is like a seed. Great Books institutions were always small, but they have been enormously influential. There’s nothing more “scalable” than wheat or bread, but no one owns all the wheat nor the bread-making recipes. It doesn’t take much thinking to see that this is for the best.

Technology only works for human flourishing if we choose to design it and to use it that way. We need to think about how we want to live—in common as well as individually—and choose accordingly. Likewise, money isn’t the most important thing. To build trust, you need to care about something that recognizably benefits more than just you and your friends.

4. What are your main philosophical influences?

For me the biggest puzzle is how intellectual excellence can and should shape our everyday choices and our public or communal projects.

Plato and Aristotle argued eudaimonia or human flourishing was constituted by the pursuit of theoretical philosophy. Moreover, they thought the wholehearted pursuit of theory, paradoxically, was the best guide to life. In part that's because eudaimonia is always the best guide to life. Mixed motives are not stable, and tend toward the dominance of the strongest and worst motive in the mix. There's only one pursuit that matters most to us. Since that motive shapes everything else, there's nothing more important than getting it right.

5. What does human flourishing mean to you?

It means the exercise of our best capacities, the power to think and the power to love. We develop those capacities by cultivating the virtues like courage, generosity, prudence, moderation, wisdom.

We need communities on our scale—think large families or small towns—to provide training courses in virtue and meaningful paths of exercise for our activities. Poor and humble people often flourish more fully than the rich and powerful because they rely more heavily on their communities and live more intimately with them. That’s where the best of life is. I came to see this from joining the Roman Catholic Church but I think the other major religions teach it. I’m not sure anyone else does.

6. What’s one book you’ve read recently that you’d recommend?

I’m not alone in thinking the Robert Caro biographies of Lyndon Johnson are incredible.

They display perfectly the tensions between character and ambition. Caro is never glib and he never takes an easy answer.

7. What’s your most irrational belief?

That I—my will and my talents—am the chief cause of my success. In fact it was mostly luck, grace, and animal spirits. Relatedly, I have the repeated delusion that whatever the problem is, I am the person best equipped to solve it. No matter how many times I see that proved false, the unsinkable belief always rises again and plagues me when I need it least.

8. What’s the most interesting tab you have open right now?

I have a tab listing the famous social science studies that have not been replicated. We swim in fake science, so much that we can’t even make a simple argument without appealing to it. We so sorely need the arts of study, listening, reading, thinking, and judgment. “Studies show...” cannot replace a solid, well-informed human judgment.

9. Who is one writer or thinker today who you think is underrated?

Well, to add a spin on the question…we are living, active beings who can feed our reflection and imagination on anything that has been thought or said.

And so I think the most underrated thinker is whatever great thinker of the past you haven’t read yet—or wherever your understanding is most limited.

For me, it's the Enlightenment authors I know least. I have been gradually untangling Descartes' philosophy and science through my teaching at St. John's and I find it very exciting.

Thanks to Zena for answering “What Will You Build For?”

To get in touch, find her on X or at Catherine Project.

This is the third installment in this interview series, see our first interview with AI Underwriting Company co-founder Rune Kvist and our second interview with ex/ante founder Zoe Weinberg.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

If you’re someone who thinks deeply, builds deliberately, and cares about the future AI is shaping: join the Cosmos network.

To nominate someone for “What Will You Build For?” leave a comment below, or send us a DM.

Every builder’s first duty is philosophical: to decide what they should build for. This series asks 9 questions to founders who are building towards their vision of the human good.

Today’s guest is Zoe Weinberg. Zoe is the founder of ex/ante, an early-stage venture fund backing technology that advances human agency and digital freedom. She talks more about her work in this short video.

Before ex/ante, Zoe worked on ethics and policy issues at the National Security Commission on Artificial Intelligence, Google AI, and the World Bank.

1. What are the core questions or beliefs driving your work?

It’s tempting to fall into standard narratives of technology as bringing either salvation or apocalypse. It’s all a bit Manichean—a black-and-white view of technology as good or evil, often in those stark terms.

I’m more interested in questions of how technology can subtly reshape human agency, human rights, and human flourishing, all depending on how it’s built. Technology can democratic or authoritarian. It can expand freedom or constrict it. It can advance open societies. Or not. There’s very little that’s preordained. There’s very little that’s not ultimately in our hands as builders, funders, and users.

2. What future are you building for?

Through my work at ex/ante, I’m trying to contribute to a future that centers individual autonomy, freedom, and choice, counteracting a digital world that feels increasingly circumscribed and controlled by governments and a handful of large tech companies.

This means a world where systems make knowledge and capability more accessible, and where the friction and difficulty that often accompanies deep understanding isn’t eliminated, but is part of what creates meaning and joy.

One version of this was captured recently in the Resonant Computing Manifesto. This suggested that artificial intelligence can give rise to digital experiences that nourish, rather than drain, us if we build with intention and thoughtfulness. I’m building for the future where innovation isn’t just for innovation’s sake, but toward the creation of a healthier and more free society.

3. What commonly held belief in the tech community do you believe is wrong?

In the tech world I often hear people say that technology is value-neutral: it’s how people use technology that can have good or bad consequences. I disagree.

Technology is purpose-built, and therefore is embedded with intent. What’s its purpose? How’s it meant to be used? Technical architecture always embodies a set of values-laden choices: the design of protocols, interfaces, and systems that don’t just enable certain actions but actively shape the distribution of power, the visibility of alternatives, and the very modes by which users understand their options.

There’s a million little decisions that go into product- and company-building that ultimately have a moral valence. To make some of those decisions easier, ex/ante and the Ford Foundation released Builder Blueprints, a library of templates and documents for start-ups to raise the bar in terms of privacy and user rights. It’s just one piece of the equation, but it is all part of the foundation on which a company is built.

Calling technology value-neutral is a cop-out.

4. What are your main philosophical influences?

I will never forget the first time I read Thomas Kuhn’s Structure of Scientific Revolutions. I am often reminded that basic assumptions about the world that we take as “science” and “fact” are in reality socially and historically-contingent. Science and progress are hardly linear, and new (often contrarian) perspectives are critical to improving our understanding of the world. In the context of my work at ex/ante, I often think about what technological developments on the far edges of the horizon could substantively change my mental model of how the world works or contribute to a true paradigm-shift.

Many of the other thinkers and writers that have influenced my work are represented in the Tech for Human Agency Library, including John Stuart Mill, Alexis de Tocqueville, Michel Foucault, Marshall McLuhan and many others.

5. What does human flourishing mean to you?

For me, human flourishing is rooted in human agency: the ability to make substantive decisions about one’s life. It requires a social structure that enables individual freedom, individual judgment, and pursuit of individual purpose.

“Individual” is the key word. Any infrastructure (technological, cultural, political or otherwise) that imposes uniformity diminishes the conditions for human development and progress. Self-authorship emerges from the struggle to understand and solve genuinely hard problems, to navigate moral complexity, to confront perspectives that are at odds with one’s worldview. On a societal level, human flourishing is necessarily plural.

The question for our moment is whether technology can expand or contract the possibilities for self-authorship and whether we can design these systems to preserve the productive friction that gives rise to human flourishing. On an individual level, human flourishing is less about achieving any particular state of happiness and more about having the capacity for self-authorship.

6. What’s one book you’ve read recently that you’d recommend?

Second Life: Having a Child in the Digital Age, by Amanda Hess.

I recently became a parent, so I appreciated this memoir about the ways that the digital world has shaped one of life’s biggest milestones, by an internet culture critic for The New York Times.

7. What’s your most irrational belief?

My irrational prediction of an irrational belief is that we will see the emergence of a religious movement centered on worship of AI and superintelligence. A great awakening for the 21st century.

8. What’s the most interesting tab you have open right now?

The website for the Center for the Alignment of AI Alignment Centers. I am a big believer in satire and humor as a method for sense-making. I also think creator Louis Barclay is hilarious and brilliant.

9. Who is one writer or thinker today who you think is underrated?

An idea that is underrated is the Free-Market Welfare State. It’s a concept described by Samuel Hammond in 2018 policy paper, suggesting that a strong social safety net can encourage more innovation, risk-taking, and entrepreneurism.

By giving citizens guaranteed support, such as health insurance and retirement security, we remove the pressure for protectionism and intervention during periods of technological and economic disruption. At the same time, we empower individuals to take risks, knowing they and their families won’t slip through the cracks. The proposal doesn’t fit in a neat partisan box, which perhaps is why it does not get nearly the attention it deserves as an economic policy solution.

Innovation and progress depend on the willingness of individuals to take risks, so policies that make taking a leap of faith easier are ones I’m likely to be in favour of.

Thanks to Zoe for answering “What Will You Build For?”

To get in touch, find her on LinkedIn or at ex/ante.

This is the second installment in this interview series, see our first interview with AI Underwriting Company co-founder Rune Kvist.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

If you’re someone who thinks deeply, builds deliberately, and cares about the future AI is shaping—join the Cosmos network.

To nominate someone for “What Will You Build For?” leave a comment below, or send us a DM.

Pepper was once a luxury. Medieval merchants traveled halfway around the world to bring it to European tables. Now it sits in shakers at every restaurant, free for the taking. It became cheap because people wanted to put it on everything.

Intelligence is becoming pepper.

As Sam Altman puts it, intelligence “too cheap to meter.” Open-source models already make raw cognitive power free at the point of use. Soon it will be everywhere, layered over every interaction, and available for every task. Our operative intelligence, the combination of biological and artificial neural networks, will expand dramatically. But as artificial intelligence becomes cheap, biological intelligence may decline. A society can grow smarter while its members’ skills atrophy.

Most people find this prospect irredeemably bad. I think some atrophy is acceptable and sometimes beneficial. This thesis is surprisingly moderate. We already accept strategic cognitive atrophy when the trade-offs are right. Calculators have led to the atrophy of mental arithmetic capacity, yet mathematicians today are better at mathematics than ever. They gave up tedium and they gained something greater: the ability to solve harder problems.

Mathematicians now routinely use computer algebra systems (such as Mathematica) and programming languages (such as Lean) to support mathematical reasoning. This lets them explore problems they could not before. The four-color theorem was proven in 1976 with computer assistance, as no human could have checked all the cases themselves. This pattern is ancient. Writing hurts our memory, but no one wants to stop people from writing. This precedent shows that cognitive offloading can enhance human flourishing when properly structured.

AI will cause some cognitive atrophy. Offloading is inevitable when intelligence becomes free. We’ll need to decide which forms of atrophy we should accept given their inevitability.

Trade-offs

John Stuart Mill and Aristotle help clarify when these trade-offs prove acceptable. Mill distinguished between higher and lower-order pleasures. By higher pleasures he wasn’t talking about “refined” tastes or elite entertainment. He meant the pleasures that engage our distinctively human capacities: reasoning, imagination, and moral feeling. As Mill put it in Utilitarianism, we “assign to the pleasures of the intellect, of the feelings and imagination, and of the moral sentiments, a much higher value as pleasures than to those of mere sensation.”

Consider someone who has experienced the satisfaction of a nice-tasting meal and the satisfaction of solving a difficult problem. If she is honest, she will rank the second higher. These provide a deeper satisfaction.

This is Mill’s “competent judges” test. Ask anyone who has genuinely experienced both kinds of pleasure which they would give up. They will sacrifice bodily pleasures first. The fool may be content. But Socrates, even when dissatisfied, has something the fool will never have.

Aristotle made a similar observation over 2,000 years prior. He distinguished three lives: the life of pleasure, the life of political action, and the life of contemplation. The life of pleasure seeks bodily satisfaction. The life of action seeks honor and achievement. But the contemplative life seeks truth itself.

Contemplation is not passive. It is the most intense activity the mind can perform. It means thinking about the highest things: the structure of reality, the nature of the good, and the order of the cosmos. This activity is not labor-intensive like farming or manufacturing, and it’s not leisure either. It is the mind at full stretch.

Aristotle thought most people could not sustain a contemplative life. The demands of survival intervened. But he also thought that insofar as we can contemplate, we should. It is what we are for.

This is where AI’s promise shines through. It can handle lower-order cognitive tasks, freeing us for activities that engage our highest capacities. A researcher who once spent hours hunting for sources can now spend those hours thinking about what the sources mean. A writer who labored over formatting can focus on whether the argument is true.

Of course, people may not do this. We all know that AI can seduce us into passivity. But I would argue that it also elevates the opportunities for virtue. The mind freed from drudgery really can rise. It can shift us from the mechanical to the meaningful. If we follow the competent judges test, as Mill would have us, we know that most people do not prefer the mechanical once they have moved beyond it. Many people really do have the time to cultivate virtue and devote themselves to pursuits that satisfy them and that achieve their highest values. AI gives us the chance to do that.

Following Mill and Aristotle, cognitive offloading is acceptable only when it preserves these capacities. It can become noble when it enhances them.

What to Offload

Consider two patients facing the same diagnosis. The first asks an AI what treatment to pursue and then simply does it. She has outsourced both the information-gathering and the judgment itself. Her capacity for medical reasoning atrophies because she never exercises it. She cannot evaluate what the AI told her. She cannot ask her doctor the right questions. When the AI errs, she has no way to catch its mistake.

The second patient uses AI differently. She conducts an extensive medical review, reading research reports that the AI surfaces. She generates a list of questions for her doctor based on what she learned, brings the reports to the appointment, and discusses them. The doctor makes the call, but with a better-informed patient in the room.

Both patients offloaded cognition. But the second preserved her deliberative capacity. She did not hunt for the research herself (there’s no free cognitive lunch), but she did something harder: evaluate it. Her speed at reading dense medical literature may atrophy. But her ability to weigh trade-offs, to question authority, and to integrate information grows stronger.

Offloading exists on a spectrum. The same pattern emerges in education, but with deliberation more fully intact. A student can have AI do her homework and then pass it off as her own. She speaks fluently about concepts she doesn’t understand. The appearance of intelligence replaces its reality. Like muscles that atrophy without use, her reasoning capacity withers.

Or she can use AI to challenge herself. Study mode forces her to work through problems rather than receiving answers. The AI asks questions, corrects misconceptions, and refuses to simply hand over solutions. Each session builds her capacity. She’s still using AI. But she’s building human thinking on top of AI thinking.

Autopilot shows what happens when atrophy goes wrong. Modern pilots offload constantly, and for good reason; automated systems handle airspeed, altitude, and navigation almost all of the time. The pilot watches the machines work, which reduces fatigue. Flights have become far safer as a result. Still, there is a loss. As we know from the movies, hand-flying an aircraft keeps pilots in the loop, letting them feel the aircraft’s responses intimately. That helps create a continuous mental model of where the plane is located and what comes next. Autopilot allows for drift, and the mental model fades.

Air France 447 is illustrative. In 2009, over the mid-Atlantic, the aircraft’s pitot tubes iced over, and the autopilot disconnected. The pilots had to hand-fly in unusual conditions and made simple mistakes. The aircraft stalled, but the pilot pulled the nose up. The flight had 228 people. None survived.

The pilots had thousands of hours of flying time between them, but their critical skills atrophied from disuse. Automation makes us safer while making rare emergencies devastating. The mathematician who offloads arithmetic still knows how to solve hard problems. The pilots who offloaded flying could not fly when it mattered.

What distinguishes good offloading from bad? The examples suggest an answer. Offload mechanical cognition: the tedious, repetitive operations that don’t require judgment. Preserve core deliberative capacities: the ability to evaluate, choose, and reason through hard cases. Make sure evaluating AI outputs requires active engagement. Expand higher-level intellectual activity. And preserve cognitive sovereignty: maintain the ability to contest what the AI tells you, to understand how it reached its conclusions, and to exit when you need to.

Permissible atrophy

In my previous essay for Cosmos, I argued that intelligence environments require contestability, transparency, and exit rights. Permissible atrophy is the next question, and it is not just a personal one.

We are designing intelligence environments for one another. A teacher who assigns AI tools shapes what her students become. An employer who deploys AI assistants shapes what their workers can do. These choices become the formal and informal rules that govern how we think.

We have freed people from drudgery before. Some wasted the freedom. Others turned toward higher things. AI can go either way. Build it wrong, and we may create sophisticated parrots: fluent, confident, and hollow.

Pepper became cheap because people wanted to put it on everything. Intelligence is becoming cheap for the same reason. How we embed it will determine who we become.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Technological progress is forcing a question: what kind of people do we want to become in a world where thinking is optional?

Cosmos Institute exists to answer that question, by developing philosopher-builders who can translate the principles of a free society into code. Now we’re growing our team.

In our year-end letter, we shared what we built in 2025:

• 140+ grantees through our fast grants program

• 16 fellows working at the intersection of philosophy and frontier AI

• 14 seminars with partners like Oxford, DeepMind, and Liberty Fund

• 8 philosopher-builders creating new companies

We’re looking for people who believe AI systems should promote human autonomy, enable truth-seeking, and resist central control, and who want to shape the institutions that make that vision real.

At Cosmos you’ll have direct exposure to top AI thinkers and builders, freedom to design how you work, and the chance to help shape the institutions and companies that will define the AI age.

We’re hiring for three core roles:

• Head of Operations

• Head of Finance

• Head of Talent and Network

Details below ↓

Head of Operations

Location: Remote-friendly (preference for Austin, New York, or London)

This role builds the operational infrastructure that allows Cosmos to scale: systems that work, a team that delivers, and a culture built on fast feedback loops.

You’ll own operational delivery for seminars, fellowships, grants cycles, and formation programs, including coordination for The Academy launch. You’ll build and maintain operations systems across Cosmos: payroll coordination, compliance, vendor management, HR systems, and reporting. You’ll establish the operational cadence that works across a distributed team, and build systems that make every cycle better than the last.

You’ll have budget for 1–2 direct reports in Year 1 and contract support as needed. As Cosmos scales, you’ll shape what the operations function becomes.

We’re looking for someone with 5-10+ years experience in operations or high-responsibility chief of staff roles who has personally touched payroll, HR systems, vendor management, and compliance. You take ownership of outcomes, have exceptional follow-through, and know the difference between what should be systematized and what should remain judgment-driven. Read more →

Apply

Head of Finance

Location: Remote-friendly (preference for US-based)

This is Cosmos’s first dedicated finance hire: hands-on enough to run daily workflows, strategic enough to build a modern finance operating system that scales.

You’ll own core finance (accounting, AP/AR, close, reporting, controls, audit/990 coordination) and forward-looking finance (budgeting, forecasting, KPI dashboards). You’ll set up our in-house finance function, lead the relationship with our external audit firm, ensure clean tracking of restricted funds and program costs, and support fundraising with credible numbers. You’ll make finance approachable: clear docs, fast responses, calm execution.

We’re looking for a CPA (or equivalent) with 4–8+ years in accounting/finance, ideally with nonprofit exposure. You treat donor and grant dollars with real seriousness, and you’re a hands-on operator who’s happy to process transactions and design better systems. Read more →

Apply

Head of Talent and Network

Location: Remote-friendly (preference for Austin or London)

Over 2025, Cosmos developed a network of hundreds of top technologists and thinkers: 16 Fellows spanning frontier AI, startups, and philosophy (including economist Tyler Cowen and Anthropic co-founder Jack Clark); 140+ grantees building prototypes and research; and 200+ attendees of deep-dive programs with partners like Oxford, Aspen Institute, and Liberty Fund.

This role takes that early strength and builds a compounding network, where the best people feel supported, make progress faster, publish and ship more, and pull in the next wave of talent.

You’ll identify and source exceptional people, assess and route them into the right pathways, and deliver value through introductions and opportunities that unlock projects, jobs, mentors, and collaborators. You’ll run the network operating system: the database, metrics, and workflows that turn messy relationship work into clean processes, freeing up your time for high-touch work.

We’re looking for a high-energy connector with 3–7+ years in talent, community building, partnerships, or accelerator/fellowship programs. You have taste and judgment: you can tell the difference between “impressive on paper” and “actually exceptional.” You can speak with researchers, founders, funders, and philosophers. Read more →

Apply

Pitch us a role

If you have a thesis on how you can contribute to our team outside of these areas, please get in touch.

We’re likely hiring soon for an operations role focused on the build out of an in-person Academy, and a researcher to contribute to work on AI, human autonomy, and societal impacts.

If our mission resonates, we’d love to hear from you.

Pitch us a role

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

There are two things that I rarely take off: my wedding ring and my Garmin watch. The former stays on because it seldom crosses my mind to remove it. To tell the truth, I’m not even sure I could take it off without the right combination of soapy water, elbow grease, and patience. The latter, the Garmin, stays put because I have a regrettable obsession with bodily metrics. I like to know my heart rate, stress levels, and especially the state of my body battery (a wildly inaccurate read about how much fatigue I ought to be feeling at any given moment).

I’m not one of those life-hacking optimization types, but I wear the watch all the same. It helps me track runs and workouts, which I review every day or two when I lack enthusiasm for another trip to the gym. People often tell me my preoccupation with these figures probably makes me more stressed. They might be right. Nonetheless, I keep using it because I worry that without constantly reviewing breaths-per-minute or VO₂ max I’d be tempted to call time on this wellness thing.

At least in my imagination, fitness – and so Garmin – is a part of my personal quest for eudaimonia (usually translated as ‘flourishing’ or ‘happiness’). Eudaimonia is less a feeling or a state than the activity of living in accordance with virtue. I think that fitness helps me become someone who’s disciplined and fairly healthy, and that’s the kind of person I want to be. In this light, the question that my smartwatch poses is an uncomfortable one: can flourishing be optimized for or must it be cultivated from within?

Garmin’s algorithms are neither sophisticated nor totalizing. They stay in their lane and have a rather narrow set of powers of suggestion. They don’t tell me how to exercise anymore than a pencil tells me what to write. Others, like recommender systems or large language models, aren’t so reserved. They suggest where to go, what to have for breakfast, what to watch on TV, and even how to broach a difficult subject with someone we care about. At the limit, these systems may make all of our decisions for us in a scenario we call “autocomplete for life.”

The Wood from the Trees

In On Liberty Mill tells us something about the human condition: “Human nature is not a machine to be built after a model, and set to do exactly the work prescribed for it, but a tree, which requires to grow and develop itself on all sides, according to the tendency of the inward forces which make it a living thing.” For Mill, human flourishing is developmental. It cannot be achieved through external optimization because the process of growth is essential for the good life. This is why he describes human nature as a tree. The metaphor captures the idea that we grow from latent capacities and inclinations that must be discovered and shaped through self-directed action. Those tendencies suggest paths, but they require autonomy to be actualized.

In the Nicomachean Ethics, Aristotle tells us that flourishing flows from virtue and virtue through habituation. Rather than asking “what should I do?” he encourages us to ask “what kind of person should I become?” Grounded in the way people act in the real world, Aristotle’s approach to ethical life focuses on character development over rule-following or achieving the greatest good for as many people as possible. His “golden mean” (finding virtue by balancing extremes) requires phronesis or practical wisdom, the capacity to discern the right action in particular circumstances developed only through living in the world.

For both Mill and Aristotle, human flourishing is a journey we make every day. Try as we might, there are no shortcuts we can take. The activity of living a self-directed life can never be automated because it is constitutive of what it means to flourish in the first place. A life in which all the right choices were made for you might be pleasant, even enviable, but it would not be yours in the sense that matters. This belief, that flourishing cannot be optimized from the outside, is based on three separate but related ideas: (1) flourishing is a process of being; (2) it requires careful self-authorship; and (3) it must be maintained solely from within. An optimized outcome can simulate contentment, but it cannot substitute for active exercise or resilient capacities.

Go Forth and Optimize

Machine learning systems are trained by optimizing a loss function, but in deployment most function as pattern-completion engines rather than goal-directed agents. When these systems engage humans, they typically optimize for an external objective (e.g. engagement, throughput, or compliance) and treat the user as the site of optimization rather than its author. Even if the objective was human flourishing, we would remain the patient of optimization rather than its architect. The technical apparatus assumes that a desirable end-point can be specified in advance so that the system constructs outcomes rather than growing them. That is unproblematic for the most part – it’s just how the technology functions – but it begins to surface some curious problems when it comes into contact with our efforts to live deliberately.

You can see this dynamic in play with recommender systems that suggest what you want before you have had a chance to reflect on it. These systems learn from aggregate behavior to predict and shape individual choices, like suggesting the optimal route or by pushing you towards television shows or music that “someone like you” ought to like. A kind of overfitting to past behavior can restrict users to previously expressed interests, generating recommendations that whittle away the variety of options needed to recognize the opportunity costs of choice.

Core to this process is the “nudge” wherein the presentation of choices is shaped by an outside actor. Unlike earlier static nudges (such as placing healthy food at eye level), AI-driven nudges operate continuously and adaptively. Given enough scale, this transforms what was once soft paternalism into something closer to soft totalitarianism as fine-grained personalization makes interventions harder to collectively resist. The more criteria are centrally set, the greater the attack surface for those who would exploit them, and the more these systems inhibit the decentralized adaptive learning through which individuals and societies discover what works. When the environment is engineered to produce predictable choices, the capacity to exercise choice becomes harder to sustain.

A common defense of nudges holds that they preserve freedom of choice while improving outcomes for society as a whole. Proponents suggest that nudges influence behavior mostly for our benefit, and that they are easy to avoid if we put our mind to it. If people can always choose otherwise, their autonomy is not compromised. The rub is that true autonomy deals with at least both first-order choices (I want a cigarette) and second-order reflection on those choices (I wish I could stop smoking). Nudges often bypass reflection – how many of us are content with taking the first movie recommendation we see? – preserving formal freedom of choice while undermining the authorship that makes them our own. Defenders of nudges might say something like “if someone’s going to unthinkingly pick the first movie, it might as well be a good one.” But this line of thinking doesn’t engage with the fact that nudges make deliberation harder in the first place. When we engage in critical reflection, we weigh alternatives and formulate our own reasons about what to do. That process is harder to initiate when a “good enough” option is presented to us that bypasses the need for deliberation.

More than that, the idea that nudges are easy to avoid fails given enough altitude. Even granting that individual nudges may be possible to avoid, like walking past that salad bar and opting for a cheeseburger, an ambient ecology of personalized nudges is not. This gets at one of the curious aspects of autonomy: for such a personal thing, it depends on social life and the opportunities to contest and compare it brings with it. Autonomy fails when exit is nominal or when commitments become irreversible. Of course all choices exist within a constellation of potential options, but it is possible in principle to structure those potentialities through systems that keep inquiry distributed rather than centrally engineered.

Another critique of the developmental view of human nature (that is, Mill’s tree) suggests that the problem is not optimization per se but poorly specified objectives. If we could align AI systems with genuine human goods, including autonomy, then optimization might in principle be made compatible with human flourishing. One influential proposal, associated with AI researcher Stuart Russell, introduces a simple framework based on three related principles: (1) the AI’s only objective is to maximize the realization of human values; (2) the AI is initially uncertain what those values are; and (3) human behavior is the primary source of information about values.

But if flourishing really is an activity rather than a state, then it may not be the kind of thing that can be maximized by an external agent. Like happiness in Aristotle’s account, it might simply be a good that must be exercised from within. In practical terms, learning values from behavior must grapple with its own circularity. If current behavior reflects preferences already shaped by prior optimization, the system is learning from wants that we haven’t fully endorsed.

More fundamentally, observed behavior may be the wrong place to look. Conduct reveals who we have been, not who we are trying to be. The developmental view suggests that we are always undergoing growth, and the distance between our actions and our aspirations is where character development takes place. No amount of behavioral data, however rich, can capture what we have not yet become. That said, Russell’s humility principle points toward something important. It reminds us that systems that surface uncertainty and return judgment to the user are more autonomy-preserving than those that do not.

Another problem deals with adequately specifying preferences for AI systems in line with the developmental view. Current models focus on revealed preferences (what we do) in service of a goal that they set (maximize time on platform in the case of recommendation engines or assist in a “helpful, honest, and harmless” manner for language models). Simply telling the system what you want to want isn’t enough; these stated preferences may not necessarily reflect the kind of person you are trying to be. Deeper approaches to character development will be needed for systems that help us realize our potential.

A final objection to the developmental reading of human nature might stress that all technology assists human action. Writing supports memory and calculators aid arithmetic (though overreliance can enfeeble both). If external assistance inherently undermines autonomy, shouldn’t we be wary of tools altogether? But the difference between AI and other technologies turns on whether assistance extends the agent’s deliberation or replaces it. A map or pencil extends my autonomy, a chatbot that makes decisions for me does not.

The Constant Gardener

Am I using the system to pursue ends I have reflectively endorsed within, or is the system shaping my ends from without? Even when you supply preferences, a system must decide how act on them according to its own constitution. If we have little ability to override those rules, well-meaning efforts at governance may inadvertently prevent systems from supporting our personal quests for eudaimonia. Consider a recommendation system that learns my preferences from behavior and serves content to maximize engagement. The system optimizes for my revealed preferences. But there’s no reason a recommender system needs to operate this way. It’s easy enough to build a system that allows users to think carefully about the type of person they would like to become by using the same technology. The former might serve you reality TV because you watched a cooking show; the latter might populate your feed with guides for cooking dishes you’ve always wanted to make.

This is better, but even this kind of second order endorsement is insufficient. If you tell a system the kind of things you want to be interested in, then you may find that the system quickly optimises for them in a way that does little to help you grow as a person. Watching more arthouse cinema and fewer Netflix originals has little bearing on the type of life you want to live. Instead, systems ought to deal with the stuff that matters. Personal agents, for example, could be designed to sift interests from from character development. They could help us think about who we want to become rather than engineering the outcomes we think we want.

In practice, that might mean an AI assistant that makes its influence legible. Rather than covertly curating your information environment, a character-supporting agent might show you the paths not taken. It might introduce the unfamiliar because growth requires encounters beyond the experiences you’ve already had. At minimum, such systems should make exit realistic through portability and contestability and ensure we sign off any changes that may influence our higher order commitments. Rather than simply checking whether your goals have changed, they could surface tensions between your commitments to help you live more deliberately.

Optimizing flourishing from the outside is a non-starter because flourishing is active and authored. But if the locus of deliberation remains with us, perhaps AI can help us live a little more wisely.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Radio arrived in the early twentieth century as pure possibility. Electromagnetic waves carried human voices across continents. Within decades, three versions of the same technology had helped produce three different kinds of citizen.

In the 1930s and 1940s, the British Broadcasting Corporation (BBC) built programming that assumed its listeners wanted to become more than they were, with challenging content and genuine debate, designed to stretch rather than soothe. The funding model made this possible: residents paid a license fee for all BBC programming regardless of what they watched. No one had to maximize time-on-station. They could pursue something else.

American commercial radio faced a different pressure: advertising. Audiences were larger when programming demanded less. The market cleared at entertainment. What sold ads was what held attention, and what held attention was not what elevated the listeners.

Soviet radio under Stalin did not bother with the pretense of serving listeners at all. The technology became an instrument of state narrative designed to bluntly manufacture the appearance of consensus and compress the space for independent thought.

The same transmitters and waveforms propagating at the speed of light formed the technical basis in all three cases, but divergent selection pressures yielded divergent equilibria. The funding models were de facto governance regimes, entangled with the political order, shaping what could be said and what could be heard.

Looking back now at the introduction of the radio, it is easy to see that the question of whether “the radio” is conducive to the good or whether it advances human autonomy is hard to meaningfully address without considering funding models. Yet whenever new technology is deployed, public discourse tends to fall back to discussing the technology in itself without reference to its economic implementation. For example, people are largely happy to discuss the ills and benefits of social media, and even legislate on that basis, as if social media could be evaluated in a test tube for carcinogens in the way we might evaluate cigarettes.

Technology only defines a possibility space: the affordances, constraints, default pathways, and scaling properties that structure our interaction with the world around us. Institutional choices and funding models determine what position in that space becomes reality. Moreover, perhaps more importantly, institutional choices and funding models determine what position can remain in equilibrium over time. An institutional aim to pursue a particularly virtuous part of technological possibility space cannot be maintained if the funding model forces the institution over time to either change the aim or die.

Consider modern recommender systems. The possibility space is vast: systems that surface what users would endorse on reflection, systems that maximize time-on-site, or systems that optimize for learning or serendipity or connection. But the economics of advertising-funded platforms select for a narrow band of that space: the band where engagement can be monetized regardless of whether it tracks anything users actually value. The narrowing isn’t unique to ads; ads are just the clearest case where the reward proxy is orthogonal to reflective value.

Billions of minds are routed by recommender systems daily. The unexplored regions of the surrounding possibility space are not technologically inaccessible, but they are economically unselected. We think of economics here broadly: all inputs required for a system to carry on. Even a non-profit social media site endowed in perpetuity or supported by the state would still need to capture enough attention to remain “social” in any meaningful sense. The distinctions we’re drawing are orthogonal to narrow critiques of capital.

What Institutions Compute

The cognitive scientist David Marr proposed a simple framework for how to analyze an information processing system, regardless of whether it is a human brain or an AI system. He argued that there are three families of questions that need to be answered to have a full understanding of such a system, which he thought of as three levels of analysis.

1. Aim: What is the system trying to compute? What problem is it solving?

2. Mechanism: What procedures does it use? What’s the algorithm?

3. Substrate: What physical substrate implements those procedures? What’s the hardware?

Marr observed that each of these questions has some independence. What you are trying to compute does not fully determine your algorithm or your hardware, and so on. Because of this, we can study facts about aims, mechanisms, and substrates somewhat separately. Yet, Marr also observed that how we answer one family of questions constrains how we get to answer the others. You cannot simply declare that a system has some aim. The aim must be achievable by some procedure, and the procedure must be realizable in the substrate that is available. If the substrate of your system does not support any mechanism that could achieve your favorite aim, that aim can’t be part of the right explanation of what the system does.

This framework transfers to institutions. A company and tech products are also information-processing systems, that can be understood through three analogous families of questions:

1. Aim: What is the organization trying to achieve? What problem is it solving?

2. Mechanism: What procedures does it use? What are the routines, metrics, and product mechanics?

3. Substrate: What implements those procedures? What is the shape of structures like revenue, ownership, incentives, user participation, survival pressures?

As in Marr’s case, we get constraints across levels of questions: the lower levels shape what’s possible at the higher ones. A founder might genuinely aim for VC-funded virtue-in-a-box. But if the company’s survival makes sense over time, the avowed aim is unlikely to remain what best explains its behavior. The substrate constrains what aims are stable. The substrate does not uniquely determine aims, but it sets the boundary of the viable.

Consider a game studio that sets out to build something meaningful. Let’s say the aim is “earned mastery.” What earned mastery requires is not just skill at the game, but the formation that comes from genuine challenge: learning to lose, to persist, to improve through practice. The mechanisms of the game serve that aim, with matchmaking, difficulty curves, and progression that rewards effort. The economics are aligned. Players pay once, and the only path to power is getting better. Aim, mechanism, substrate, all pointing the same direction.

Then growth becomes the mandate. Loot boxes convert well. Players want to feel powerful now, not after fifty hours. As a result, mechanics emerge that let money substitute for skill.

No one at the game studio explicitly decided to abandon the mission. Many probably saw where things were heading. But each choice was locally rational. What got measured became what mattered; what mattered became what survived; what survived became the real aim. The specter of Goodhart’s law looms large and the feedback loop doesn’t care about the pitch deck. It optimizes for what it can see. By the end, “earned mastery” appears in the mission statement and nowhere else. The aim became a ghost. Not abandoned, just no longer part of the best explanation of the company as a system, or the best model for decisions within it.

A distinction matters here. Engagement that tracks value (e.g., mastery, truth, community, or genuine satisfaction) is what good products should generate. The problem is engagement engineered through compulsive behavior, asymmetry, and the exploitation of human weakness. In the worst case, we substitute habit for growth, money for skill, manufactured urgency for actual importance.

Returning to radio, we might say that the BBC avoided structural drift in its heyday because license fees created no pressure to hold attention for advertisers. The economics supported the stated mission rather than corroding it. American commercial radio found other priorities because advertising revenue required capturing attention and capturing attention rewarded different mechanisms than cultivating it. Those mechanisms eventually became the point. This is not a case for license fees as such—the BBC subsequently developed other pathologies, and any funding model creates its own selection pressures. The point is the structural logic, not the specific policy.

The framework we described is analytically neutral. It tells you how aims, mechanisms, and economics can be articulated independently yet alerts you to how they constrain each other in real systems, particularly as they persist over time. It doesn’t tell you what aims are worth pursuing. You could use it to help design effective propaganda as easily as effective education.

Every information environment is a training regime: it determines what people practice noticing, what they practice ignoring, and therefore what they become capable of judging. The institution that shapes attention is in turn shaping the citizen.

Descriptive frameworks are useful if they help us understand systems, and more useful when they help us articulate what systems we want. When we say the BBC “cultivated” and commercial radio “captured,” we are claiming that some outcomes are better than others. Specifically, preserving and developing human judgment matters because judgment is the foundation of autonomy. Systems that cultivate judgment expand what a person can do and be. Systems that atrophy judgment reduce the agent to a consumer of impulses by executing preferences they never made their own and optimizing for ends they never deliberately chose.

If we are going to reason about institutional design at all, we need to be explicit about what we are designing for. That is a prior question, and ignoring it just means your values operate without scrutiny. For our part, we take human autonomy as our prior commitment.

From this perspective, the question for any technology that touches human judgment is not “is this technology good or bad?” The question is: in what institutional arrangements is the technology implemented, and do those arrangements create structural pressure toward or away from the cultivation of human judgment?

In practice, this means asking:

• What aim does the economic structure sustain over time? The mission statement or the aim that survives contact with the feedback loops?

• What mechanisms does that aim require? Do the proposed mechanisms only work in a vacuum to deliver your aim, or could they stably work to deliver your aim given the economics that you are prepared to implement?

• What do those mechanisms do to the humans who use them? The effect you designed for, or the effect at equilibrium?

Many companies cannot answer these questions honestly. The mission statement says one thing; the incentive structure produces another. The founders may believe in the stated aim; the economics select for something else.

If autonomy is the commitment, we can ask what design criteria this would entail. One way to proceed is then to articulate those criteria as constraints or tests. We suggest two tests below as minimal starting points:

1. The Transparent Choice Test: Could this product survive in a suitably nearby world of users who fully understand how the product works and could easily select an alternative?

A product fails this test if its economics depend on the gap between informed choice and actual behavior: it only works because users don’t fully understand it or can’t easily leave. Note that the test brackets market power. Whether it is easy to switch in the actual world is separable from whether people would switch if they could, in a suitably nearby world. A virtuous product might still be a monopoly; a monopoly might still make a good product. Whether monopolies are bad for other reasons is a separate question.

2. The Candid Aim Test: Is the stated aim key to the best explanation for how the system actually behaves and how its parts are put together?

If the stated aim is real, reference to it should make the system’s behavior intelligible and more predictable. It’s a red flag when a different aim explains more, particularly so if it is an aim that is not just orthogonal but contrary to the stated aim. Consider a supermarket with a cash register that makes subtle addition errors at the threshold of detection, always in the store’s favor. If this has been going on for years, “fair dealing” stops being a plausible aim of the store. Or consider a recommender system that reliably foments polarization. At some point, “building community” explains less than “maximizing clicks.”

Neither the Transparent Choice Test nor the Candid Aim Test is passable long-term if the economics pull against the aim.

Structure, Not Will

Structural drift cannot be resisted at the level of individual will. A founder with integrity, operating within a misconfigured stack, will be selected against by the institutional arrangements that structure their team, product, and market. If you have spent years inside one of these companies advocating for users over engagement and eventually burned out, it wasn’t a failure of will. The economics select for certain equilibria and not others. You can resist that gradient for a while, but you cannot reverse it through conviction alone.

The intervention must be structural, not only personal. This is the work of what we call philosopher-builders: configuring the stack itself.

A skeptic will ask: if a firm adopts constraints that reduce short-term competitiveness, won’t the market select it out? Isn’t anything outside of profit maximization founder self-indulgence? But this assumes profit is already defined before we’ve chosen what game we’re playing. The time horizon, the measurement regime, and what gets externalized are not givens. The builder—the philosopher-builder—chooses them.

Some commitment devices reduce competitiveness while others become the competitive advantage. Substack’s bet on writer ownership creates lock-in through loyalty rather than switching costs. The Bloomberg terminal succeeds not despite its commitment to decision-quality over engagement, but because of it. Patagonia’s environmental constraints built a brand that commands premium pricing.

Not every structure survives in every market. Some commitment devices only work with certain capital structures, or in markets where trust is differentially valuable, or with customers who can recognize quality. That’s precisely why governance, capital philosophy, and culture matter. The economics of the endeavor are not given. They are chosen—by founders, by investors, by the people who set the constraints within which everyone else optimizes.

This reframes what it means to work on technology that matters.

The question is not whether you’re an engineer or an executive. The question is whether you’re operating at the level of institutional design going all the way from formulating aims to economic implementation, or only at the level of features. Both matter, but the institutional level in this broad sense is prior. It determines what the feature level can achieve.

An engineer who designs a brilliant recommendation system but ignores the incentive structure it operates within has ceded the most important decisions to someone else. An engineer who understands how funding models shape optimization targets, how metrics become attractors, how governance locks in or erodes mission is operating at the level where outcomes are actually determined.

Every investment thesis contains an implicit theory of what is of value. Every term sheet is a design document for what will be optimized. Every board seat carries influence over what gets optimized. These decisions ripple outward into mechanisms, and mechanisms ripple outward into the lives of everyone who uses what gets built.

The question is whether this influence gets exercised deliberately or by default.

Default means optimizing for what’s measurable, what’s familiar, what’s already worked. Default means engagement metrics and growth curves and the patterns that produced returns last cycle. Default means structural drift toward whatever the economics select for, regardless of stated intentions.

Deliberate means asking the three questions. It means applying the Transparent Choice Test and the Candid Aim Test. It means designing economic structures that can sustain aims worth pursuing, and locking those structures in before the drift begins.

At the present stage of AI development, this framework is not an abstract lens but a live description of the game being played. Frontier systems are expensive to build and are increasingly financed by capital invested on roughly the thesis that the technology will ultimately be applicable to almost every aspect of the economy. In that sense the capital is not buying a product so much as underwriting a general-purpose substrate and hoping to later discover the dominant interfaces through which it captures value. That option-like thesis makes the “aim” unusually plastic: the same underlying model can become tutor, co-worker, companion, bureaucrat, sales engine, surveillance layer, or propaganda instrument. Which of those becomes stable is a property of the institutions, financing models, and culture that wrap it.

Because the return story is “almost everything,” the equilibrium remains underdetermined. Selection pressure is already operating, but often through indirect channels: control of distribution, accumulation of compute, narratives that justify scale, the ability to externalize risk, and the power to set default contracts and defaults-of-use.

Different capital structures will stabilize different aims. Usage-based pricing tends to reward systems that are instrumentally useful; subscription models reward systems people would still choose under conditions of autonomy more than attention-economy models do; enterprise procurement tends to reward auditability and legible governance; state funding tends to reward control.

The degree of freedom is real precisely because the equilibrium has not yet fully formed. Decisions made at this pivotal moment by technologists, founders, and investors can still dramatically shape outcomes we care about.

The attention economy shows what happens when the substrate hardens and aims collapse into a single attractor. If revenue is selling predictable attention to advertisers, then maximizing attention capture becomes the stable aim, engagement proxies become the mechanism, and surveillance plus lock-in becomes the substrate that keeps the loop from breaking. The equilibrium is narrow and self-reinforcing, and the possibility space remains technologically available but economically unselected.

Frontier AI is not yet locked into that attractor, yet it could inherit it quickly if general-purpose models are primarily deployed as next-generation attention routers. If autonomy is the commitment, this is exactly the moment when financing, ownership, and governance choices matter most: before the underdetermined equilibria congeal into infrastructure and the aim becomes a ghost again.

What got built around radio in the twentieth century shaped what kind of citizens could emerge. What gets built around AI now will do the same, at greater scale and speed.

The window is still open. Which citizens are you building for?

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

This year I watched my four- and six-year-old flourish at Alpha School, where AI personalizes their learning and frees time for what machines cannot supply: curiosity, character, play. I saw AI systems produce new mathematics, reshape creative work, and hand ordinary people powers that once required armies of engineers.

And yet I’ve noticed something in myself I don’t much like. The readiness to accept an AI’s first draft of an email, a plan, a decision, because it’s faster, smoother, and good enough. An invisible waltz in which I take one step and then forget that I am no longer leading.

In many domains I’ve welcomed the flow and eagerly entered the new paradigm. In others I’ve pushed back: the forming of beliefs, the making of judgments, the hard labor of learning to think at all. For the deepest questions, what it means to live well, I’ve tried to refuse it entirely.

The more I talked with others this year, the more I understood this was not my private neurosis. Something is shifting. People feel it, even when they cannot name it.

At our seminars with Oxford, Aspen, Liberty Fund, St. John’s College, DeepMind, Palantir, and Microsoft, and at our AI for Truth-seeking Symposium with FIRE, a pattern declared itself: the people most anxious about preserving human judgment are often those building the systems. They see how capable these tools are becoming, and they are asking questions the rest of us would prefer to postpone.

At Anthropic’s headquarters in San Francisco, looking out over the city, co-founder Jack Clark told me he journals more than ever, writing down his thinking before consulting any AI. He knows the outputs are better with the machine. What he wants to know is whether he is still developing. This is the difference between effectiveness and judgment.

Technology that expands what is possible may narrow who is possible. But it need not.

Adam Smith understood both halves of this. You become a good thinker by doing the thinking, badly at first, then less badly. The practice is constitutive. This is the best antidote I know against becoming a Claude Boy. And the kind of people who develop through their own judgment are precisely the kind we need if we are to sustain a free society.

Twenty-five hundred years ago, Pericles stood before the mothers of Athenian dead and described what their sons died for: a city where citizens cultivate beauty and wisdom, debate openly before acting, and take responsibility for public life. A free society.

Since then, the principles of free societies have been transmitted through debates, texts, and institutions. We are entering an era where they must be embodied in code—or they will become inert. The philosophy-to-law pipelines of old have given way to philosophy-to-code.

Tocqueville saw the stakes: vigorous self-governance in which citizens grow through freedom or a softer servitude in which we gradually surrender it for comfort and convenience, becoming, in his phrase, “a flock of timid and industrious animals.” The choice is ours to make, while we still remember what it means to choose.

This is our civilizational moment. It requires people who can translate the principles of a free society into the actual systems and institutions being built.

The philosopher-builder

We named the philosopher-builder archetype in July: the kind of person who, like Benjamin Franklin, combines philosophical reflection with practical wisdom and builds institutions that embody their deepest convictions about human flourishing.

I expected this to resonate with a few hundred people. Instead, some of the most impressive founders I know sent notes saying they’d finally found language for what they were trying to do. Researchers said it changed how they thought about their careers. What I came to understand is that the division between “thinkers” and “doers” has impoverished both, and many people had been quietly waiting for permission to be whole.

The philosopher-builder is an answer to a transmission problem. Whereas the great reformers of old wrote pamphlets, today’s are writing code. They will give the principles of a free society technological expression: what autonomy looks like in API design, what decentralization means for architecture choices, what truth-seeking requires in model development.

This year Ivan Vendrov, Zoe Weinberg, Séb Krier, Jason Zhao, Alex Komoroske, Lisa Wehden, and Joel Lehman all spoke about their work through this lens. We published Summer and Winter reading lists drawing on recommendations from across our network. And we started to see a community coalesce around the conviction that philosophy and building aren’t separate activities—they’re the same activity, done with different hands.

What we built this year

Fourteen seminars, including with Oxford, Aspen Institute, Liberty Fund, St. John’s College, and Edge, bringing builders into conversation with Hayek and Polanyi on the use of knowledge in society, with Socrates and Mill on truth-seeking, with Smith on moral development.

On YouTube and other video channels, our interviews with 11 AI experts and entrepreneurs on their philosophical influences reached over a million subscribers. On Substack, our long-form essays on AI and philosophy reached over 17,000 readers.

And voices across our network produced pieces we keep returning to: Tyler Cowen on how AI will change what it is to be human, Jason Crawford’s Techno-Humanist Manifesto, Caitlin Morris on social tinkering, Gavin Leech’s “The Scaling Era” with Dwarkesh Patel, and Jack Clark’s “Silent Sirens, Flashing For Us All.”

We grew our fast grants program to 140 builders and researchers. Through initiatives including a $1M AI for Truth-seeking program, many projects like Campus, Szdat, Kanonic, and Authorship began to take shape.

We supported 16 fellows working at the intersection of philosophical depth and frontier AI, many of whom did so at Oxford HAI Lab. This led to 34 research papers, from “The Philosophic Turn for AI Agents” to “Full Stack AI Alignment” to “Martingale Scores.”

Through our incubation fellowship, Samuele Marro launched a new non-profit called the Institute for Decentralized AI. We backed 8 philosopher-builders to create new companies that take their ideas to world-changing scale. And together with IHS, we started funding AI tool use for scholars in philosophy and the humanities: researchers like Kevin Vallier and Seth Lazar integrating these tools into serious intellectual work.

What matters more than numbers is the community that emerged: people who share a conviction that the principles of a free society must be translated into the systems we’re building, and that the window for doing so is shorter than most people realize.

What I don’t know

The practices I’ve developed personally feel almost monastic: blank-page journaling before consulting AI, deliberately choosing the harder cognitive path, regular reading groups. Preserving the independence, force, and originality that remains to us seems to require a level of intentionality we often tell ourselves we don't have time for.

I believe the classical liberal tradition has resources for this moment. But translation is hard: principles like truth-seeking and autonomy don't map easily onto architecture choices and agent interfaces. And even when the direction is clear, building well requires judgment that can only come from practice. The ideas and the formation have to come together.

What I’ve come to believe is that this practical wisdom has to be developed in community: through seminars, collaborations, and shared building. And probably through intensive multi-month human formation, which will require new thinking.

The philosopher-builder isn’t a solitary figure. The archetype only works if there are institutions that cultivate it. That’s what we’re trying to build.

Looking forward

This year, Harry Law and I are writing a book on what it takes to be a free agent in the AI age. I’m going to work closely with Philipp Koralus on research at Oxford’s Human-Centered AI Lab, and deepen connections with collaborators at the top labs as things move fast.

We are bringing this community together in person. More gatherings, more informal meetups, especially in Austin. And we’re training the next generation of philosopher-builders through a new format we’ll announce soon.

We’re hiring and will share roles soon. If our mission resonates, get in touch.

The question I keep returning to is simple: What kind of people do we want to become in a world where thinking is optional? The answer will determine what we build.

The flame of a free society has been passed from debate to debate, text to text, institution to institution for 2,500 years. Now it needs to live in code.

To the donors, fellows, collaborators, and readers who made this year possible:

Thank you.

Brendan

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

“If I asked people what they wanted, they would have said faster horses.” The idiom, a widely circulated but likely apocryphal line attributed to Henry Ford, stresses the distance between our ability to picture the future and our ability to make it real. It reminds us that technologies loosen the constraints that shaped past expectations, that deeper shifts usually enact variations in kind as well as magnitude.

“Faster horses” is a shorthand for folk logic that seems bulletproof at the time but quaint in hindsight. Television as radio with pictures, film as photographed theater, early mobile phones as portable landlines, and the internet as a digital library were all kinds of faster horses. They tell us that big swings don’t often play well with existing categories, and that new language, heuristics, or classifications are often needed to make sense of them.

Today, many of those wondering about the downstream impact of thinking machines are on the lookout for AI that can function as a “remote drop-in worker.” This refers to a system that replaces a human employee, in essence, by doing roughly the same things under the same conditions. Here, the future appears as a more seamless version of the present rather than something that dramatically changes the shape of work.

The idea flows from the observation that the majority of jobs in the information economy revolve around making computers do what we want. Word processing, desk research, data analysis, creating presentations, running marketing campaigns and many other tasks are all the end product of keyboard strokes and cursor movements. This is why some long-time AI watchers reckon Claude Opus 4.5, especially its instantiation within Claude Code, can reasonably be described as an early realization of Artificial General Intelligence (AGI). The same might eventually be true of humanoid robots, especially given they can slot into existing infrastructure without costly redesign, but our focus here is solely on knowledge work.

As others have pointed out, the response to a common AGI litmus test (a system that can outperform humans in most economically valuable work) turns on what we categorize as “economically valuable work.” If we define that as “stuff done on a computer,” then it’s plausible that one day soon the models will cross that threshold (if Claude 4.5 Opus hasn’t already). And if a model can be said to be generally capable, then the remote drop-in worker shouldn’t be too far behind.

Whether a single model can do a job in isolation is a useful question to ask, but it doesn’t tell us much about how such systems, interacting with many people and agents of their own, might rearrange patterns of coordination and the shared assumptions that guide them. In some ways, the conservative bet is that the drop-in worker is a stronger account of our present than it is our future. Technologies that matter rarely honor the roles we assign them. If the future is anything like the past, the drop-in worker may prove to be a faster horse: a story that made sense before the true nature of the agent economy became visible.

The Wisdom of the Crowd

Traditional accounts of AGI development often describe the emergence of an isolated system capable of completing the vast majority of cognitive tasks, sometimes referred to as a “singleton”. An alternative scenario, now seriously considered by AI developers, imagines that capabilities may be manifested through the coordination of “sub-AGI individual agents” with complementary skills and affordances. This scenario concerns an ecology of semi-specialized agents whose combined behavior outstrips anything they could do alone (and tallies up to something that we could describe as AGI at a high enough level of abstraction).

You might have a code agent that builds, a negotiation agent that handles scheduling or purchasing, and a compliance agent that checks your work. On top of these sits a manager that breaks goals into subtasks and shunts each to the right agent for the job. We state an objective, the system spins up a network of agents, they pass data between them, and a synthesis function presents the output of the collective for review.

Imagine launching a new software feature. The drop-in worker functions like a high-speed freelancer insofar as it writes code, pauses to check for bugs, and writes documentation sequentially within a single stream. It is a linear acceleration of a human workflow. The agent ecology, however, behaves more like a stack of mini-organizations. When the objective is stated, an “architect agent” drafts the structure while a “red team agent” simultaneously attacks that design to find security flaws before a line of code is written. A “compliance agent” cross-references regional data laws in the background. These agents operate in parallel to create an adversarial loop where the output is the sum of many small interactions. The result is an ecosystem capable of the kind of concurrent processing that individual minds, biological or synthetic, may struggle to achieve by themselves.

But this is only a partial picture. The share of human-agent and agent-agent interactions in the economy will increase over time, with agents engaging in price negotiation, placing orders from one another, coordinating supply and demand, and even rating each other to assign trustworthiness scores.

In some ways, the patchwork AGI thesis is another episode in a long-running story about how intelligence behaves at scale. Markets outperform planners because knowledge never exists in concentrated or integrated forms, but as incomplete and contradictory perspectives dispersed across individuals. Hayek reminds us that “planning” happens all over the place through individual agents, which is why he distinguishes it from “economic planning” that deals with state-backed forms of enterprise management. The agent economy doesn’t represent a toss-up between planning or ad-hoc action but rather an older question about whether planning ought to emanate from within or from without.

Aristotle raised the question that still haunts proponents of collective intelligence: can the many, combining their partial virtues, outperform the excellent few? In Book Three of Politics, he writes:

“For it is possible that the many, though not individually good men, yet when they come together may be better, not individually but collectively, than those who are so, just as public dinners to which many contribute are better than those supplied at one man’s cost; for where there are many, each individual, it may be argued, has some portion of virtue and wisdom, and when they have come together, just as the multitude becomes a single man with many feet and many hands and many senses, so also it becomes one personality as regards the moral and intellectual faculties. This is why the general public is a better judge of the works of music and those of the poets, because different men can judge a different part of the performance, and all of them all of it.”

For Aristotle, groups become smarter when they successfully combine different aspects of competence into a single body. Consider a jury that sees people with different experiences and biases pool their judgment to reach a fairer conclusion than any juror might in isolation. Or England’s common law, where centuries of small decisions by judges produce a legal order with more adaptability than one made by decree.

The same is true of Wikipedia, peer review, or nimble companies. In each case, the quality of the outcome rests on a kind of distributed deliberation wherein perspectives clash, revise, correct, and eventually settle into a stable state. It echoes the Athenian assembly and the medieval disputatio, both of which treated the good we call judgment as the product of structured disagreement.

American writer Howard Rheingold coined the term “smart mobs” to describe groups of people who are able to organize and coordinate quickly through the use of mobile communication technologies like the internet or mobile phones. The term “mob” is deliberately ambivalent, a framing he uses because of its darker connotations (he explicitly notes mob mentality can be for good or ill).

Rheingold thought smart mobs worked because low-cost communication let individuals share context and act in concert without central control. These groups represented an idealized version of accelerated coordination built from minuscule signals that could be aggregated over huge numbers of agents. The mob framing reminds us that coordination capacity increases faster than deliberative capacity, and as a result, the key variable becomes governance of the communications substrate.

But mobs aren’t smart by default. There are a whole set of coordination problems that flow from distributed decision-making, from free-riding (where individuals benefit from a group’s effort without contributing to it) to information cascades (where people copy others’ choices even when their own judgment points elsewhere). Many of us recognize some of these problems when we spend too much time on social media. We see outrage spread through networks faster than facts, and know how easily a crowd can be steered by sentiment rather than the hard work of judgment.

We could say that groups become dumb when they fail to properly synthesize knowledge, and they become smart when divergence is preserved and integrated. Whether or not we benefit from the wisdom of the crowd often depends on the structures that keep the mob in check. Markets do this with prices and labs with peer review. Rheingold might say that smart mobs emerge when communication structure and incentives reward decentralized coordination rather than herd behavior.

Society of Mind

In the 1980s, the AI researcher Marvin Minsky wrote about what he called the “society of mind.” What he meant was that unified intelligence is a loose federation of smaller processes, each narrow, each fallible, yet together capable of producing something that looks like coherent thought given enough altitude. For Minsky, intelligence emerges from many mindless “agents” coordinated in special ways, with the mind employing something like a computational and explanatory strategy whose power is a product of messiness, cross-connection, coordination, and resolution.

Today’s AI models demonstrate unified intelligence at two levels: as a byproduct of statistical learning and in the way models are housed within larger constellations that we refer to as “systems.” Transformers likely generalize because they compress huge corpora into representations that let them improvise solutions on the fly. Intelligence is in some sense a property of compression plus scale, an analog of Aristotle’s crowd insofar as it concerns many partial signals integrated into a single effective whole. As for the systemization of models, we can view each as a constellation of individual expert functions like tooling or multi-modal functionality.

But why stop there? If we accept that AI, like all intelligence, benefits from the interactions between discrete units, it follows that its capability should also be treated as a property of a larger constellation in which many systems operate together. Variance creates productive tension as models surface alternative interpretations and explore distinct solution paths. When those paths are combined through orchestration layers, tool use, or various other kinds of agent frameworks, the result is a system that searches the problem space more effectively than a lone model. Once multi-model systems interact with one another — coordinating, passing intermediate results, or checking each other’s claims — a kind of higher-order intelligence bubbles up from the sum of interactions across several layers. More powerful models are great, but superior ecologies are better.

People each hold fragments of truth, most of it tacit and hard to articulate, which is why spontaneous order tends to get the better of best laid plans. Polanyi might remind us that a drop-in worker presumes that competence is formalizable into explicit tasks and checklists. His work tells us that competence lives partly in the realm of tacit knowledge, that “dropping in” workers will face the same context problems faced by central planners the world over.

Of course, recent progress in AI development does typically try to provide models with the context they need to work effectively (specifically through the use of reinforcement learning techniques to make models good at human work in human settings). We might also say that, even if multi-agent systems matter internally for model cohesion, their deployment could still take the form of a remote-worker analog. Things like permissions, accountability, compliance, budgeting, and change-management favor inserting agents into existing workflows as opposed to redesigning them from the ground up.

These objections are useful, but they don’t allow us to skirt the core problem with the “remote drop-in worker” metaphor: that it treats intelligence as solely a property of individuals. It presumes the unit of analysis is the solitary agent carrying out tasks one after another, when everything we know about complex work suggests otherwise. Real capability comes from the knots of relationships, feedback loops, constraints, and opportunities that bind us together. First within models, then as agent systems, then eventually as agent-agent systems. Collective intelligence is prefigured by how information moves through a system and how the residue of experience accumulates across many small decisions made by each of us.

The remote drop-in worker may prove to be a transitory moment at best and a category error at worst, one that treats AI as an incremental addition to familiar workflows rather than a force that will reshape the nature of those workflows. That is “faster horses” thinking. We’re projecting today’s limitations onto tomorrow’s world and overlooking the fact that new capabilities alter the constraints that make what happens today seem natural. More accurate accounts tend to lead somewhere else, in a form we often only recognize with the benefit of hindsight.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

And so, with Christmas fast approaching, we asked 12 more of the sharpest minds we know for their philosopher-builder recommendations for you to settle down with over the holidays.

From ‘how thinking emerges’ to ‘Universal Robots’ to ‘LLMs as the death of the author,’ here are 12 picks from top AI entrepreneurs and thinkers to kickstart your ideas for 2026.

Read on for reflections from each recommender + a bonus recommendation from our Editorial Lead, Harry Law.

1. The Invention of Science

by David Wootton

It’s a readable and staggeringly erudite work of intellectual history. I’ve read it once, listened to it once, and have embarked on a second reading.

Before the discovery of America, Wootton argues convincingly, Europeans lacked the concept of discovery. No language but Portuguese even had an adequate term. European intellectuals assumed that any new invention or idea must be a recovery of something previously known but lost or forgotten. (Never before reading this book had I realized just how weird the Aristotelean picture of our planet was and thus how disruptive the voyages of discovery were to its factual claims.)

On my second reading, I was struck by Wootton’s description of Diderot: “Diderot had one great advantage over us: graduating from the Sorbonne in 1732, he had been educated in the world of Aristotelian philosophy. He knew how shocking the destruction of that world had been, for he had experienced it at first hand. From a bird’s-eye view—the historian’s view—the Scientific Revolution is a long, slow process, beginning with Tycho Brahe and ending with Newton. But for the individuals caught up in it—for Galileo, Hooke, Boyle and their colleagues—it represents a series of sudden, urgent transformations.”

As we anticipate sudden, urgent transformations from the deployment of powerful AI, The Invention of Science offers a deep dive into a parallel heady, disruptive process.

,
Author, a columnist for Works in Progress, and an Abundance Institute fellow. She and Charles M. Mann will release a podcast series on the history of everyday technologies in early 2026.

2. Vehicles

by Valentino Braitenberg

Now over 40 years old and long predating modern AI, Vehicles challenges us to think about the relationship between behaviour and intelligence.

Through a series of thought experiments, Braitenberg shows us that many things that look like “thinking” can emerge from simple, deterministic rules. As we increasingly interact with machines that (look like they) think, having good mental models of where complex behaviour comes from will be an essential component of epistemic hygiene!

,
Co-Founder of Entrepreneurs First, and Chair of the Advanced Research + Invention Agency (ARIA).

3. The Knowledge Machine

by Michael Strevens

I am glad I read Michael Strevens’s The Knowledge Machine: How Irrationality Created Modern Science. I bristled at some parts of it, but I did appreciate its clarity.

Strevens writes in a stripped-down way about what makes modern science different: his “iron rule” that we’re supposed to bracket off things like beauty, morality, and metaphysics and let only publicly checkable evidence into our arguments.

I’m not persuaded that this is in any way “irrational” (hell, Hume did more damage with rationality than that), and I think we should all probably be rereading Karl Popper’s The Open Society and Its Enemies if we want the full-throated defense of liberal science. For how we actually build better decision- and knowledge-making systems at the institutional level, I’d also send people back to Montesquieu and The Federalist Papers, which treat checks, balances, and structured disagreement as epistemic tools as much as political ones. But The Knowledge Machine is a lucid, accessible, and genuinely interesting thought experiment. For anyone who cares about how we know what we know (especially technologists building new “knowledge machines” of their own), it’s well worth a read—ideally with you arguing with it in the margins.

,
President & CEO at the Foundation for Individual Rights and Expression (FIRE). Author of The Coddling of the American Mind.

4. The Listening Society

by Hanzi Freinacht

For transformative AI to help humans flourish, much depends upon the wisdom of those designing and deploying it. Yet most technologists are blindly running on obsolescent ideas while firmly believing they are not — captured by unifaceted ideologies about progress or power. The Listening Society proposes a philosophy of flourishing that can meet the complexity and nuance of the world. It’s one of the most impactful books I’ve ever read. Even if you don’t like it, at the very least it will provoke new thoughts.

Joel Lehman,
Former OpenAI co-team lead on openendeness. Author of Why Greatness Cannot be Planned. Cosmos Fellow.

5. What Is Political Philosophy?

by Leo Strauss

In What Is Political Philosophy?, Strauss highlights how modern thought has substituted key classical concepts with new ones - most notably replacing the classical idea of virtue with the modern notion of responsibility.

The implication of this shift is that we now frame our behavior and moral standards in ways that make them easier to satisfy, even as we erode the substance of the ideals they once represented, diminishing the demanding excellence at the core of virtue.

I would encourage our generation of technologists to examine our industry’s terminology in light of this provocation. When we invoke words like courage or freedom, what work are they doing for us - and do they still call us to excellence, or serve to comfort us with a veneer of virtue?

,
Founder and CEO at Plymouth Street. Former President of the Oxford Union.

6. R.U.R.

by Karel Čapek

Karel Čapek’s 1920 R.U.R., which introduced the word “robot” in its modern sense, has food for thought on almost every aspect of our current technological landscape. There’s a scientist who built robots because he wanted to play God, and his nephew, a founder, who just wanted to make money. There’s a General Manager who believes his product will liberate humans from the drudgery of work, and an engineer who loves his job. There’s a humanitarian who worries about the robots’ souls, while widespread infertility decimates the human population. In the background of it all, there’s the question: what is it, really, to be human? And what, if anything, is that humanity worth?

,
Professor of Philosophy at UT Austin. Writer of ChatGPT and the Meaning of Life.

7. Aspiration

by Agnes Callard

Aspiration is a thoughtful book, especially useful as an indirect and unintended commentary on much discourse about AI alignment. This discourse often assumes what Callard calls a “decision-theoretic” model of values as fixed and unchanging. Instead, Callard argues that value change, not just accidental but sought out, is a core part of how human values work.

(Rudolf also recommended Liberty by Isaiah Berlin, particularly the essays on J.S.Mill)

,
Co-founder of Workshop Labs, ML Engineer, and Writer at No Set Gauge.

8. A Pattern Language

by Christopher Alexander, Murray Silverstein, and Sara Ishikawa

A Pattern Language might be on the shelf of every architect, but it’s more broadly about understanding the conditions that allow human life to flourish.

How do ideas spread? What creates a sense of belonging? Why don’t people dance in the streets anymore? The authors identify a series of “patterns” (which also come across beautifully as a love letter to design details) that create places where people feel alive and connected.

Reading between the lines and beyond just sometimes outdated architecture, these patterns are responses to deep human needs - enabling shifts between privacy and spontaneous encounter, exploring the role of different “distances” of engagement. A vocabulary not just for thinking about physical spaces but about what we’re trying to create when we design for human thriving. This is less of a “read all at once” and more of a “thumb through like recipes” book for me.

(Caitlin also recommended Speculative Everything: Design, Fiction, and Social Dreaming by Anthony Dunne and Fiona Raby)

,
MIT Media Lab Researcher and Educator who designs technology for curiosity-led social learning. Cosmos Grantee.

9. The Conscious Mind

by David Chalmers

The hard problem of consciousness is easy to state: why is there something it is like to be you?

Neuroscience can explain which brain states correlate with which experiences. But the hard problem of consciousness comes from recognizing that correlation isn’t identity, as we can presumably have the correlated brain states without the corresponding conscious states. We can have brains without the stuff of experience (qualia).

In The Conscious Mind Chalmers works through this puzzle with famous clarity. The payoff for technologists is striking. It can help readers see how, for all we know, AI could be conscious even if the mind is immaterial.

,
Professor of Philosophy & Director of Research at the Institute for American Constitutional Thought and Leadership, University of Toledo.

10. Language Machines

by Leif Weatherby

Leif Weatherby’s Language Machines argues that AI is not agentic intelligence but the death of the author made concrete. Structuralist theorists like Jakobson and Lacan argued that language was a generative system of signs, which existed independently of the ground truths that it described. Weatherby marries their ideas to those of AI researchers such as Claude Shannon, Walter Pitts, and Warren McCulloch. The conclusion: LLMs are not individual intelligence or anything like it, but language as a system, made capable of speaking.

(Henry also recommended The Unaccountability Machine by Dan Davies)

,
SNF Agora Professor of International Affairs at Johns Hopkins University. Writer at Programmable Mutter.

11. The Beginning of Infinity

by David Deutsch

The Beginning of Infinity reshaped how I think about progress, knowledge, and what’s actually possible. He makes optimism rigorous.

Deutsch argues that knowledge creation can continually solve problems and transcend apparent limits, challenging us to reject fatalism about constraints we mistake for destiny.

Progress isn’t guaranteed, but it’s achievable. What drives it? Conjecture, criticism, and relentless error-correction, not certainty or dogma. The idea that humans are “universal explainers” capable of understanding anything in principle feels both humbling and empowering; we’re limited only by what we haven’t yet figured out.

Yet his most useful insight centres on explanation itself: good explanations are hard to vary while still accounting for what we observe, like a key cut to fit one lock. Change one element and the explanation collapses. The clarity of distinguishing real understanding from mere description or correlation matters because it shows us where to focus. We’re building knowledge that compounds.

,
Founder of Exponential View and investor, author of The Exponential Age.

12. How the World Became Rich

by Mark Koyama and Jared Rubin

The tech world has rediscovered big questions about growth: why did the industrial revolution happen where it did, what role did states play, and why do some transformations stick while others don’t?

How The World Became Rich explores the competing explanations without forcing a single answer — a useful corrective for anyone whose model of progress starts and ends in the lab, or the state.

,
Frontier Policy Development Lead at Google DeepMind.

Bonus: The Turing Test Argument

by Bernardo Gonçalves

In The Turing Test Argument Gonçalves explores one of the most well known, and most misunderstood, episodes in the history and philosophy of AI: Alan Turing’s imitation game.

This little book argues that the test was a response to swirling controversy surrounding Turing’s debates with physicist Douglas Hartree, chemist and philosopher Michael Polanyi, and neurosurgeon Geoffrey Jefferson.

Gonçalves thinks that Turing’s focus on learning and adaptability sought to counter Hartree’s view of computers as calculation engines, that tasks like the composition of poetry aimed to address Jefferson’s demands for creative abilities, and that the introduction of open-ended conversation (rather than the use of rule-based games like chess) was designed to confront Polanyi’s concerns that human knowledge couldn’t be formalised by machines.

It’s a short but essential read for anyone interested in deeply understanding a moment that never seems to stop making headlines!

,
Cosmos Editorial Lead, former DeepMind Policy Research, and Cambridge University Researcher.

Thanks to Virginia, Matt, Greg, Joel, Lisa, Harvey, Rudolf, Caitlin, Kevin, Henry, Azeem, Séb, and Harry for contributing to this list.

And let us know your book recommendations for the holidays in the comments!

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Today, Cosmos Institute and the Foundation for Individual Rights and Expression (FIRE) are excited to announce our second cohort of AI x Truth-Seeking fast grants, along with a new set of General Track grantees working on adjacent risks and opportunities (from research into human autonomy to RL approaches to improve AI system alignment).

For this batch we’re excited to be supporting an immensely talented group including: ex. NotebookLM and DeepMind researchers, current Computer Science professors, and student “study influencer.”

All winners receive $1K–$10K in grants, access to other top thinkers in philosophy and AI, and have 90 days to build a prototype.

Our fast grants aim to lower the barriers for bold experimentation, and to accelerate idea and talent development. Since launching we’re backed 100+ projects through the program.

From hundreds of applications for this round, the following builders really stood out for their technical and moral vision. We’re excited for what they will build!

Alexandria by Zizo Bahnasy
A claim-testing engine that supports critical reasoning and long-term research

Zizo is a Harvard Economics & CS graduate working on mechanisms for shared prosperity with AI developments.

Anti Echo-chamber AI by Steve Rathje
AI system that is both non-sycophantic and enjoyable

Steve is an incoming Assistant Professor of Human-Computer Interaction at Carnegie Mellon where he will lead the “Psychology of Technology” lab.

askWisr by Erin McGurk
Gamified micro-lessons to teach effective AI use to students

Erin is a Cambridge Land Economy student who has worked with top AI companies on the use of tech in education.

Bridge by Ben Harris
A search engine for evidence and counter-claims

Ben is a CS masters graduate with a background in finance, security, and system design.

Consider by David Lyreskog and team
AI platform for navigating polarized topics

David is a senior researcher in Neuroethics at Oxford University specialising in philosophy, ethics, and emerging tech.

DeliberationBench by Paul de Font-Reaulx, Luke Hewitt, and Maximilian Kroner Dale
Assessing AI persuasion by comparing it to diverse human discussion

Paul, Luke, and Maximilian bring backgrounds in cognitive and behavioral science to problems around how AI can improve human reasoning.

Hydrangea by Zaria Jalan
Social media labeller that highlights when claims differ from their sources

Zaria is a researcher at MIT Media Lab, with a background as a SWE at Google, Jigsaw and Pinterest.

Mirai by Cathy Fang
AI-powered self-voice for challenging assumptions

Cathy is a researcher at MIT Media Lab, and first author of the first RCT on the psychosocial impact of chatbots on users - in collaboration with OpenAI.

Norma by Jeff Fossett and Jesse Bryant
AI for surfacing the moral values behind political claims

Jeff is a Research Lead at Plurality Institute exploring digital technologies and public life. Jesse is a doctoral candidate in sociology at Yale.

Opinion Faction Remedies by Alice Huang
Algorithms that help people think outside of ideological bundles

Alice is an assistant professor in CS & philosophy who studies the social dynamics of science and collective reasoning.

Proposit by Brian Cefali
AI app to encourage self-questioning when developing arguments

Brian is a software engineering manager with a security background and passion for science and the humanities.

Socratic Trident by Mateo Petel
AI that is both auditable and adversarial

Mateo is an AI research scientist at Stanford and former DeepMind researcher who works on verification for high-stakes AI.

Thought Refractor by Cheng-Wei Hu
AI that turns arguments into interactive simulations and diagrams

Cheng-Wei is an ex-Notebook LM AI engineer and writer who is now building learning tools.

Whitebox Tools by Giovanni Occhipinti
Solutions to ensure AIs accurately report their reasoning

Giovanni is a visiting researcher at Oxford University investigating the failure modes of LLM reasoning.

AIxDemocracy by Fazl Barez
Research on preventing AI-enabled power concentration

Fazl is a Senior Research Fellow at Oxford University working on AI safety, interpretability, and technical governance to build AI systems aligned with human values.

Multi-Armed Bandit Evaluation Lattice by Jaeho Lee and Alon Ragoler
System that pits LLMs against each other to test for manipulative capabilities

Jaeho is a data scientist studying at Brown and Alon is a machine learning engineer studying at Berkeley. They met at SPARC, a summer program for exceptional teenagers.

Pragmatic Autonomy by Daniele Cavalli
Research on human autonomy and technology as a cybernetic process

Daniele is a PhD researcher in Cognitive Science who works on cognitive autonomy and its interaction with AI systems.

Replication Radar by Rhea Karty
A relational AI-generated knowledge graph of academic literature.

Rhea is a pre-doctoral fellow at Harvard’s metareflection lab working on verified programming.

VerbalizedRL by Hunar Batra
RL training that exposes signs of reward hacking and emergent misalignment.

Hunar is an Oxford PhD student working on AI reasoning robustness, with a background at Anthropic and AISI.

WorldWatcher by David Huang
News monitoring to juxtapose divergent views and detect narrative shifts

David is a data scientist in venture investing, and former engineer at Palantir.

The AI x Truth-Seeking grants are made possible through our partnership with the Foundation for Individual Rights and Expression (FIRE).

Thank you to the FIRE team for setting up and scaling this program together, and to Prime Intellect for partnering on compute for projects.

For more on they “why” of this program see:

• Will AI Kill Our Freedom to Think

• A World Unobserved

We’re also seeing demos from our first batch of AI x Truth-seeking winners in January, with grantees having already shared early prototypes of UIs for more fine-grained interaction with AI systems, ways AI can account for language changes over time, and writing tools that maintain your voice.

To follow the projects and get updates on future round openings, subscribe below:

Subscribe now

— The Cosmos Grants (Jason Zhao, Zoe Weinberg, Alex Komoroske, Darren Zhu), and FIRE teams

In 1831, a twenty-two-year-old naturalist boarded the HMS Beagle with a notebook and a willingness to look. For five years, he sailed. He walked the Galápagos, held finches in his hands, noticed that their beaks varied from island to island.

He didn’t know what it meant yet. That would take another twenty years of experiments, of arguing with himself in the margins of his own manuscripts.

In 1859, Charles Darwin finally published. And then, the printing press did its work. The Origin of Species spread across Europe in months. In the process it transformed our self-conception.

It reached Thomas Huxley, who would defend it. It reached Bishop Wilberforce, who would attack it. Within a year, in a famous Oxford debate Huxley and Wilberforce went at each other over whether humans descended from apes. Lady Brewster fainted in the audience.

Generation → Dissemination → Judgment. Humans on both ends, technology in the middle.

For a century and a half, this was the shape of things. Our frameworks for protecting knowledge production, like the free speech tradition, focused on dissemination. Every technology that followed—radio, television, the internet, social media—made that middle faster and wider. But the endpoints stayed human.

Until now.

Generation

Darwin needed twenty years before he had something to say. Years of reading, from Lyell to Malthus—but also picking up rocks, breeding pigeons, dissecting barnacles. He sat with his puzzlement.

Generation remained stubbornly, beautifully human. It was tethered to experience, bound by the slow accumulation of evidence, and disciplined by the resistance of reality. Somewhere in every chain, someone had touched something that wasn’t text. Anchoring generation to the world made it expensive, which meant institutions could handle what made it through. Our entire epistemic infrastructure evolved for this world.

Now that cost has collapsed. Today, AI can ingest trillions of tokens—the aggregate digital imprints of human thought—without the experiences that produced them. More and more of what looks like knowledge production is not anchored to anything outside itself.

AI has our conclusions, but not our warrant for them. A paper that reads like years of research can be produced in seconds. The product might look like Darwin’s, with evidence, caveats, and apparent rigor. It might look better. But there are no rocks behind it. No finches. No twenty years of being in the world. More disembodied, statistical text per day than humanity wrote in centuries. That’s what’s entering the system now.

Judgment

Judgment was the backstop. When too much got through, humans would evaluate it. Darwin publishes, Huxley and Wilberforce argue, the scientific community tests, and over decades the theory survives or fails.

More and more each day, AI participates in how we evaluate and interpret what we encounter. It summarizes documents so we don’t have to read them. It answers questions so we don’t have to research them. Each individual offload usually makes sense. Why spend an hour parsing a report when AI extracts the main points in seconds? Why wade through conflicting sources when the model synthesizes them? The tool saves time. It lowers effort.

But something happens when these reasonable choices compound.

Think about GPS. Each decision to use navigation instead of finding your own way is sensible. You get there faster, with less stress. But over years of doing this, something atrophies. You let slip the capacity to navigate: the ability to build mental maps, to orient yourself in physical space, to find your way when the system goes dark. You stop building internal models because the external model is always available.

I don’t fret much about GPS. Alfred North Whitehead was right that cognitive offloading is how we build civilization. We progress by increasing the number of things we can do without having to think about them.

But notice what we offload: arithmetic, memory, navigation—tasks. Judgment is different in kind. Offloading tasks can be civilizational progress. Offloading the capacity that decides what to believe is civilizational regression.

Judgment is not a task to be completed but a capacity to be a certain kind of knower. Someone who stands behind their conclusions, who can defend them, who revises them in light of evidence. You can outsource your calculations without changing who you are. Can you outsource your judgment and say the same?

Yet each day, we do. Each decision to let AI summarize, synthesize, or conclude seems reasonable in isolation. But in trusting AI to be right, we may lose our ability to discern if it is actually right. You read the summary, adopt the conclusion, and later can’t explain what would change your mind. That’s the signature of outsourced judgment. A person who can’t trace a belief back to reasons can’t really be persuaded or corrected, only swapped from one confidence-provider to another.

Mill said free speech is the lifeblood of truth. This is an argument many know well, that some have spent their careers defending. It rests on a background assumption: that people can evaluate competing claims and reach their own conclusions. That with practice, the capacity for judgment can develop.

What happens when the practice fades across generations—the way, for many of us, our sense of physical direction already has? When the capacity for judgment that AI was meant to assist never takes root in kids? When we’ve built an autocomplete for life?

The Closed Loop

Now put these together. What happens when both endpoints of the chain are no longer reliably human-accountable?

The loop closes. We enter a world that is less audited, and less observed.

Consider how AI learns. It trains on what humans have written—books, articles, conversations. Its outputs then enter the information environment. People encounter them, share them, build on them. Those outputs shape the next generation of AI, which learns from a world already saturated with the previous generation’s outputs.

Human beliefs shape AI. AI shapes human beliefs. Around and around. The cost of producing plausibility has collapsed, but the cost of verifying truth has not. And that asymmetry isn’t static. The trap tightens as it runs. Verification gets harder with each iteration, because you have to trace further back to find something that touched reality. None of this is inevitable. But it’s the direction of travel. And if human judgment is itself being outsourced, there’s no checkpoint left to catch the drift.

Here’s what this looks like. Researchers fed AI chatbots fake medical terms like “Faulkenstein Syndrome” and “Zorstein Virus.” The chatbots didn’t hesitate. They repeated or elaborated on the error in 83% of cases. “Casper-Lew Syndrome,” it turns out, is a rare neurological condition characterized by fever, neck stiffness, and headaches. Except it isn’t, because it’s not real. But a user, if their judgment is shaped by years of trusting AI summaries, doesn’t discern that. They share it. It enters the web, gets scraped into tomorrow’s training data, and the error gets repeated by the next model with even more authority.

Where in this loop does reality enter? Where is the checkpoint?

Darwin read Lyell on geology, and then spent years picking up the rocks Lyell wrote about. A doctor reads the literature but also sees actual patients (none of whom have “Casper-Lew Syndrome”). Somewhere in the traditional chain, someone had contact with something that wasn’t text.

AI has no such contact. AI isn’t in the world the way you and I are. As we use it today, it doesn’t notice when reality fails to match the pattern. If the training data encodes errors, the AI reproduces them with confidence. And as AI outputs flood the information environment, the proportion of content that has ever touched reality shrinks.

The loop can produce errors, but more importantly, it degrades the capacity to detect errors. And it does so invisibly, while looking like knowledge, spending down an inheritance of reality-contact that took centuries to build. How would you detect this from inside the loop? You can’t google your way out of a polluted Google.

Conjecture and refutation can still appear to happen. But they happen inside a hermetic seal. Conjectures generated from text. Refutations drawn from text. The whole cycle can proceed without anyone’s eyes on anything outside.

Researchers have a name for what can happen when models train too heavily on their own outputs: model collapse. Give an image model its own outputs and the image degrades into something unrecognizable. Signal becomes noise.

Is there a civilizational version of the same dynamic? We’re spending down epistemic capital, our stock of reality-tethered knowledge. Current AI sounds smart because it trained on centuries of humans who touched reality: scientists, engineers, doctors, mechanics, writers who lived with consequences and wrote them down. It’s parasitic on that inheritance. But as AI outputs saturate the information environment, the inheritance dilutes. We’re in the good phase—the phase where accumulated capital masks the loss.

We see early signs: the homogenization of prose style across the internet, cultural outputs resembling each other, more people citing summaries of summaries instead of primary sources. What will generation five look like? The experiment is already underway.

Of Knowledge and Knowers

This might seem like a problem for epistemologists—interesting, abstract, not obviously urgent. But the implications extend to how we govern our knowing. The free speech tradition emerged because knowledge is social. It requires not just individual knowers but an ecosystem: publishers, critics, scientists, citizens capable of discerning sense from nonsense. If AI alters the conditions under which that ecosystem functions, free expression is implicated.

The free speech tradition remains essential. But something else is happening that our frameworks don’t quite capture.

Free speech is a shield for expression. But AI increasingly operates upstream of expression—in the formation of thought itself. When the drafting, synthesizing, and evaluating happen before you’ve arrived at something to say, the battleground of free speech isn’t only in our courts, it’s in our minds.

Mill gave us several reasons to protect the clash of ideas. We might be wrong, and need correction. The opposing view might contain partial truth we’d otherwise miss. Even when we’re right, a view that’s never contested becomes dead dogma and loses its vital force.

AI disrupts all of this. When AI drafts our thinking, we skip the work that lets us notice we’re wrong. And a view can’t be truly contested unless the interlocutor can be pinned to commitments, shown evidence, and forced, socially or intellectually, to update. AI bears none of those costs.

A judgment you’ve worked through can survive questioning from any direction. A pre-computed judgment you’ve merely absorbed collapses the moment it’s pressed. AI makes absorption effortless, and what’s effortless becomes default. The work of arriving at a view that’s genuinely yours becomes optional.

The right to think for yourself, and the right to speak for yourself, can remain perfectly intact while the capacity to think for yourself—the capacity that makes these rights meaningful—quietly atrophies.

The 20th century’s biggest threat to free speech was coercion. Today, it is comfort: frictionless belief acquisition that makes judgment optional. This is why intellectual diversity, and the practices that keep judgment sharp, matter more than ever. They are the conditions under which genuine thought remains possible at all.

Darwin spent decades looking at the world before he changed how we see ourselves in it. His theory forced a question: what does it mean to be human, if we’re part of the animal kingdom?

We face a reckoning of similar magnitude. What does it mean to think, when thinking can be automated? What does it mean to judge, when judgment can be outsourced?

How do we respond when the chain of knowledge production that once had humans at both ends is no longer reliably human-accountable?

Let’s get to work.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing.

With FIRE we run a $1M grant program focused on AI x truth-seeking (opening again in early 2026). We also co-hosted a Symposium last weekend moderated by Jonathan Rauch, with participants from Cosmos, FIRE, DeepMind, Stack Overflow, and Oxford HAI Lab, covering the future of free speech, knowledge generation, and live demos from our first batch of grantees, who are wrapping up their projects. The essay above builds on Brendan’s talk from that event.