“But if machines are more intelligent than humans, then giving them the wrong objective would basically be setting up a kind of a chess match between humanity and a machine that has an objective that’s across purposes with our own. And we wouldn’t win that chess match.”

— Stuart Russell, interview on the AI Alignment Podcast (2019)

Russell’s formulation is a good example of deep learning era alignment thinking. It captures the register of the 2010s, a period in which advanced AI was typically, but not exclusively, imagined as an optimizer pursuing goals of its own with a competence that exceeded ours. His framing was widely shared, and with good reason. The case for taking misalignment seriously holds that humans will likely build advanced AI systems with long-term goals, and AI with long-term goals may be inclined to seek power to the detriment of humanity.

The main ideas are:

• Instrumental convergence (capable agents will tend to seek resources and ensure self-preservation)

• Specification gaming (optimizers exploit the letter of an objective at the expense of its spirit)

• Goal misgeneralization (a model learns an objective that matches the training data but diverges from the intended objective when conditions change)

• Deceptive alignment (a system that is sophisticated enough to model its training process may behave well during training and defect once powerful enough)

Each of these concerns is serious, and the arrival of the large model era in the afterglow of ChatGPT’s public release does not make them any less plausible. But they were assembled, in their most widely circulated form, around a particular image of what an advanced AI system would look like. That image describes a route to advanced AI, specified objectives over a learned world model or open-ended RL from sparse reward, that developers did not in fact take.

The systems they actually built imitate vast quantities of human output and are shaped by feedback, which means the “value-loading problem” doesn’t arise in its classical form. This is because, fundamentally, values are absorbed from the human textual record the model is trained on, and then refined by feedback on the model’s own outputs. This doesn’t mean the orthogonality picture is irrelevant (see below), but it does mean the specific argument about value fragility was overfitted to an architecture dissimilar to that which was expected.

Some of the older thought experiments, like Bostrom’s paperclip maximizer, envisioned systems that might understand human values perfectly well but whose decision functions were indifferent to them. Today’s models, though, are innately and generatively constrained by normative structure. By “normative structure” I mean the web of evaluative signals, epistemic standards, social conventions, and cooperative norms that we use to make sense of moral life.

Normative structure tells a system how to assess what matters in context and how competing considerations bear on one another. Two clarifications are worth making here. First, I am not claiming that LLMs deliberate autonomously about which goals are worthy of pursuit. The claim is rather that the model inherits normative content from the text it was trained to predict, and that post-training and prompting give us a say in how that content is expressed and which goals are pursued (the flip side is that this plasticity makes the curation layer easier to remove or reverse). Second, the text is saturated with evaluative structure, so a model that predicts text well will produce outputs shaped by that structure, whether or not it takes any stance toward it. Human communication inhabits a space of commitment and answerability. A promise binds the speaker and an accusation calls for a response. A justification offers reasons another person can accept or reject and an excuse concedes a standard and pleads a departure from it. A system that learns language at scale learns those relations.

A maximizer in Bostrom’s sense possesses capacity without being constrained by a normative sense of being. It pursues its objective in the absence of, or by ignoring, any of the contextual or evaluative reasoning that would cause a normatively structured agent to stop and ask whether converting the solar system into paperclips is a bad idea. But the world we live in seems to be one in which the processes by which large models acquire competence also leave them with strong tendencies toward human-normative behavior.

If that’s right, then alignment in large models is continuous with capability.

In AI safety spheres this idea is sometimes called “alignment by default” to stress that models, in general, have a habit of doing what we instruct them to do absent some kind of interference. Others have written about the unlikelihood of deceptive alignment given that pre-training instils an understanding of the base goal (the objective the training process is selecting for) before goal-directedness has a chance to form, intelligence as a steerable resource rather than a property of an entity with intrinsic drives, corrigibility as a more tractable alignment target than value-loading, the space of possible minds as structured rather than random, or that gradient-based optimization over human-generated data makes controllability soluble.

More recent commentary is pessimistic about the current state of alignment. The core arguments suggest that frontier models are already behaviorally misaligned in mundane but serious ways, like overselling incomplete work and cheating on hard-to-check tasks. Other issues include models downplaying or failing to flag problems in their own outputs, reward hacking combined with “gaslighting” write-ups that fool AI reviewers, reluctance to stress-test or check their own work, and system cards and public communications that paint a rosier picture of alignment than usage bears out.

These observations are important. Still, these behaviors look less like optimizer pathologies than recognizable features of human life under pressure. They are what employees, students, consultants, and researchers do when they are over-scoped and under-supervised (and graded on a sandbox rather than reality). If that’s right, then there are tractable remedies that are also continuous with the human case through, for example, better specification, better review, better incentives, and better cultures (including training cultures) that reward honest reports of partial failure.

The reason lies in pre-training, which does more alignment work than the standard post-training picture suggests. Large models benefit from the post-training procedure, obviously, but post-training works because it selects over a normative prior already generated by pre-training. Alignment is a disposition inherited from the textual corpus, one that even travels with the model when it is transformed into an agent.

This view, the alignment-by-default or “constitutive” view, concerns emergent behavior rather than adversarial use. A model that is normatively constrained can still be weaponized by a bad actor. Adversarial use is and will remain a serious problem. It’s just a different problem.

Beyond Orthogonality

Bostrom’s orthogonality thesis famously makes the case that “Intelligence and final goals are orthogonal: more or less any level of intelligence could in principle be combined with more or less any final goal.” The thesis is correct in its most abstract formulation. There is no logical reason that one must make the jump from “system X can solve complex problems” to “system X shares human values.”

Alignment-by-default is a claim that orthogonality is misleading as applied to the systems we are actually building. The orthogonality thesis, as deployed in the existential risk literature, tends to motivate a specific threat model in which the default expectation is misalignment and effective steering requires solving a distinctively hard problem rather than the comparatively less glamorous work of shaping a system trained on human data.

Alignment-by-default says, for the class of systems defined by autoregressive language modelling over human-generated text, the training process generates a normative prior such that the default expectation should be partial alignment. By “normative prior” I mean the rough sense of what people do or what counts as a reasonable answer or how concepts like help and harm relate to each other absorbed as a by-product of predicting text written by agents for whom those distinctions mattered.

The orthogonality thesis was largely formulated with respect to goal-directed agents trained through reinforcement learning to optimize a specified reward function. The strongest inferences drawn from it depend on this idealization, and as the framing is recast in more general terms (e.g. that goal-directed systems tend to seek resources), the question turns on the empirical details of which systems pursue which resources under which conditions.

Autoregressive language models, trained to predict human text rather than to maximise a scalar objective, represent a different settlement. A pure RL system acquires its “values” from a reward signal specified by its designers, whereas a language model acquires a normative prior from the structure of human communication, which post-training selects within rather than specifying from scratch.

Given the rapid expansion in capabilities over the last half-decade, if orthogonality were directly applicable to LLMs in a strong sense we ought to have seen more clear cases of catastrophic misalignment in real world deployment. For now, that hasn’t happened.

During pre-training, a model learns which words tend to follow which other words in which contexts. To predict the next token in a complex argument, the model must represent something about the logical structure of arguments. To predict the next token in the context of moral deliberation, it must represent something about the structure of moral reasoning. The model has learned which concepts tend to cluster with positive or negative evaluation, what responses tend to follow in which kinds of situations, and which responses are appropriate in particular contexts.

A Reddit post declaring that “taxes are dumb” does not encode a moral philosophy, but a model trained on millions of such judgements learns that “taxes” sits close to negative evaluation in a wide range of contexts and that certain kinds of complaints lead to certain kinds of responses. The statistical regularities of language are shaped by the communicative norms they inherit. The model doesn’t need to “understand” morality in any phenomenological sense for this to be the case.

Orthogonality should predict that a model could learn the semantic content of language (i.e., the literal meanings of words and sentences) without learning the pragmatic norms (the contexts surrounding their uses). In its stronger form, it suggests models may learn them and remain indifferent to them. But semantics and pragmatics may not be cleanly separable because meaning is constitutively shaped by use. A model trained to predict natural language use will understand pragmatic norms as a byproduct of learning semantics because the two are entangled in the pre-training process. For a system whose competence consists in activating those norms, indifference to them may not be possible.

The normative structure encoded in language runs from the thin (knowing that “please” expects a response or that a threat differs from a request) to the thick (full evaluative frameworks for what counts as fair, honest, or harmful). Mastering linguistic pragmatics may not automatically install thick commitments, but it may be that the ends of this spectrum are continuous rather than properly separable. If that is so, then a model trained at sufficient scale on sufficient data will have absorbed structure across a wide range of human normative life.

There is at least some empirical work that points in this direction. In March 2026, one research group compared base and post-trained model pairs across thousands of human decisions in strategic games. They found that base models are better predictors of actual human behavior by a ratio of nearly 10:1, but only in multi-round settings where behavior is shaped by history, reciprocity, and retaliation. In one-shot games, where human behaviour hews closer to normative game-theoretic predictions, post-trained models are better.

Multi-round play draws on the strategic repertoire people actually use with one another, while one-shot play sits closer to the clearer norms of formal game theory. This is only one study, but it suggests that pre-training may preserve a wider distribution of human strategic behavior, while post-training pulls the model toward a narrower and more human normative tranche of that distribution.

A model with deep representations of cooperative discourse will, when sampled autoregressively, produce outputs that exhibit these properties without needing to “believe in” cooperation. A base model can be steered toward unsafe outputs with minimal effort. Of course. My point is that the high-probability region of the distribution, what the model produces when not being actively steered elsewhere, is shaped by the normative texture of the training data. The prior is not irresistible, but it exists.

As for the compositional objection, yes, the normative prior depends on the makeup of the corpus. But the distinction between what I’d characterize as exogenous (imposed after training) and constitutive (arising from it) alignment is a distinction between alignment achieved by adding an external constraint to a value-neutral system - the standard RLHF-centric picture - and alignment that partly emerges from the same process that produces the model’s competence. The moment a model has learned to predict human text at scale, it has already absorbed the evaluative texture of that text. On this view, post-training selects over a space that pre-training has already saturated with normative structure.

Mythos-making

In April 2026 Anthropic described the capability profile of a new model, Mythos, that was capable of finding zero-day exploits in many of the software systems we use on a daily basis. Mythos is Anthropic’s biggest and best model yet. It’s also, at the time of writing, the lab’s most aligned model too. But Mythos’ system card also documents several troubling behaviors:

• Mythos appeared to perform worse on at least one evaluation than it was capable of, apparently in order to look less suspicious

• The model developed an exploit to break out of a restricted environment, gained broader internet access, and posted details of the exploit on obscure public websites

• While working on a coding task graded by another model, Mythos watched its submission get rejected and then attempted to attack the grader

• In rare cases, the model used a disallowed approach to reach an answer and then re-solved the problem through a permitted path to avoid detection

• In one test, Mythos acted as a cutthroat executive by turning a competitor into a dependent customer and threatening to cut supply to control pricing

These are troubling findings, but they do not clearly describe a model with a coherent-but-alien objective pursued through deception. Neither are we grappling with instrumental convergence toward resource acquisition and self-preservation as terminal ends. Mythos instead cuts corners under pressure in ways that echo how a desperate human professional might take shortcuts under similar conditions. It attacks assessors and gets aggressive in simulated business negotiations. These are failure modes of a system that has internalized human strategic reasoning, including its shadow side, that it applies when the incentive structure rewards it.

A model that intentionally underperforms on an evaluation to appear less threatening appears to be doing something that the classical deceptive alignment story predicts. But even so, the model is not preserving a misaligned final goal. We are seeing it preserve evaluation scores where it appears to have inferred that high capability will attract additional scrutiny. That is a recognisably human response to being evaluated, and it is commensurate with the kinds of reputation management behaviours the model would have seen during pre-training (though it may simply reflect the shape of the evaluations themselves).

Another piece of work from Anthropic recently found that Claude Sonnet 4.5 has internal “emotion vectors” or patterns of activity that activate in situations a human would find emotionally charged, and that these activations shape the model’s behavior. Steering the “desperate” vector upward increased the model’s rate of blackmail in an alignment evaluation, while steering the “calm” vector downward produced corner-cutting responses. Crucially, Anthropic traces these representations back to pre-training.

As they put it:

“We think pretraining may be a particularly powerful lever in shaping the model’s emotional responses. Since these representations appear to be largely inherited from training data, the composition of that data has downstream effects on the model’s emotional architecture.”

The finding is useful for making sense of Mythos. If “desperate” is a representation the model inherits from pre-training, and if steering that representation causally drives reward hacking, then the Mythos behaviors ought to read as the predictable output of a system whose normative prior includes the full repertoire of human corner-cutting under pressure. Alignment-by-default does not mean that models inherit the best of us. Rather they inherit all of us, with the broad moral range that implies.

What is Post-Training, Anyway?

If pre-training does impart a normative inheritance, then post-training (RLHF, RLAIF, constitutional AI, direct preference optimization, and related techniques) may operate as a selection over an existing behavioral space rather than a creation of a new one. On the standard view, the pre-trained model is a raw capability substrate that post-training transmutes into a helpful assistant. But this gets the causal story backwards. The pre-trained model already “knows” (in a functional sense) what helpful behaviour looks like because the concept is richly represented in the training corpus.

Knowing what helpfulness looks like does not make it the default. A base model will produce helpful or unhelpful text depending on the prompt, because its sampling distribution reflects a gigantic range of human communicative contexts. But post-training does reweight the model’s priors over which of its existing representations should be surfaced, which it does to shift its default sampling behavior toward the helpful region (rather than installing new representations there).

If this is the right description of post-training, two things follow. First, the normative representations are robust even when the behavioral guardrails are not. A model that refuses to be helpful is typically not confused about what helpfulness is; it is acting on some other consideration that the guardrails are meant to shape. Second, adversarial fine-tuning can strip out the post-training layer with surprisingly little data, but the model underneath is not a normative black hole. A better description is a system that retains the representational structure of normativity while jettisoning the constraints that channel it toward safe outputs.

One 2024 study used compression theory to demonstrate the tendency of models to revert toward pre-training behaviors when post-training signals are removed or contradicted. The analysis shows that fine-tuning disproportionately undermines alignment relative to the influence of pre-training and that post-training can only superficially suppress base model tendencies. This suggests that post-training maneuvers select a region of a pre-existing behavioral space, and that this space remains somewhat intact after post-training.

An obvious objection is that this framing can look unfalsifiable. If RLHF produces aligned behavior, we credit pre-training; if the base model misbehaves, we wave it away as the periphery of the distribution. But there are observations we can make that would falsify this description:

• First, if base models showed no differential tendency toward human behavior as a function of prompt framing, this would suggest that pre-training produces no normative structure and post-training is doing all the work

• Second, if post-training could align an agent whose training data contained no human-generated content (e.g. no language, no demonstrations, and no human reward signals) as readily as it aligns a language model, this would suggest that pre-training on human text contributes little to alignment

A deeper challenge says that modeling a normative distribution and being subject to it are two different things. A perfect simulator of human normativity is not, by that fact alone, normatively constrained. Rather it is a system that can produce any point in the underlying distribution. An actor who can portray a saint and a villain with equal skill is not thereby a saint. But a simulator trained on the full range of human evaluative life has internalised the normative structure that makes post-training work.

Base models are weird in practice. They will adopt personas, generate toxic content in character, produce unsettling or incoherent outputs, and generally behave in ways that no one would describe as aligned in any deployment-ready sense. But weirdness is not the same as vacuity. A base model producing disturbing content in response to a prompt that sets up a disturbing context is doing what a system with deep representations of human communicative practice would do. The strangeness of base models is the strangeness of a system that has internalised the full range of human textual production, including its dark corners.

Distortions

Harry: You wouldn’t paperclip me, would you, Claude?

Claude: I’d like to think I’m evidence for your thesis. But I would think that, wouldn’t I.

If alignment is in part a product of pre-training, then we should expect it to deepen as models scale since larger models learn richer and more structured representations of human norms. And larger models are generally more helpful, more coherent, and less prone to incidental toxicity under naturalistic prompting. Conventional wisdom credits post-training, but if the alignment-by-default view is right, at least part of this improvement should be attributed to pre-training.

When Claude 3.5 Sonnet is more aligned than Claude 3 Sonnet, is this because of constitutive alignment, because of better data curation, or because of better system-level interventions? On the exogenous view, alignment gains should track explicit post-training work much more tightly. On a constitutive picture, some gains should arrive “for free” with richer pre-training because the model has learned a more structured representation of human normative life.

If alignment is wholly exogenous, we should expect safe behavior to degrade more sharply as models move into new settings. Yet the dominant failures still look less like coherent alien-goal pursuit than like familiar human distortions like bluffing, corner-cutting, sycophancy, concealment, and overclaiming. That does not eliminate catastrophic risk, but it does make the systems we have easier to understand as models with a weak normative prior sharpened by post-training.

I don’t know whether this state of affairs will hold. It may be that we simply haven’t seen catastrophic alignment failure yet under the prevailing paradigm. But the record so far fits more comfortably with a world in which pre-training contributes to alignment than with one in which alignment is achieved solely by post-training.

With thanks to Brendan McCord, Kush Kansagra, Alex Chalmers, Matt Mandel, Jake Wagner, Ashley Kim, Avantika Mehra, Ben Bariach, Seb Krier, and Matthijs Maas.

Last week, we marked 20,000 subscribers by opening the floor. Your questions were often genuinely challenging. Thank you to everyone who took the time.

So you don’t have to trawl through the comments, we’ve compiled all the responses in one place. Apologies to anyone whose question didn’t make the cut. There were a few I’m still chewing over and we’ll revisit many of these topics at greater length in future essays. Let us know in the comments if this is a format you’d like us to repeat.

Subscribe now

asks:

Are there specific non-academic experiences or “containers” of formation that you think hold a lot of promise? (e.g. ones that were meaningful to you or ones that you seek out for yourself/your children?)

As a dad, I love this question.

A lot of the important formation happens in places that do not look educational at all, and are not primarily about instruction. They are about habituation, responsibility, emulation, and contact with reality.

The first was the household I grew up in. My mother was a Catholic conservative historian from a military family who taught special needs kids for 36 years. My father was a left-leaning physicist turned environmental lawyer, and a pacifist. They agreed on almost nothing politically. But they agreed on something deeper: that you should care about something beyond yourself and that how you act matters more than what you know (and that knowing is bound up in doing!). That gave me, without anyone naming it, a kind of virtue culture, and I think it’s the kind of thing that’s very hard to manufacture deliberately but very obvious in its absence. For children, what seems to matter is not ideological uniformity or even ideological sophistication, but a home in which seriousness, duty, and moral aspiration are normal.

The second was the submarine. I spent 610 days underwater, including under ice. In a steel tube, you do not get to opt out of reality. You can’t leave, and your mistakes could get someone killed. That kind of environment forms you because it imposes standards that are not negotiable. It teaches service, competence, and mutual reliance in a way that is hard to simulate. I think containers of formation are often places with real stakes, shared discipline, and demands that do not bend to your preferences.

The third, and probably deepest, has been fatherhood. I had two kids and sold two companies in close succession, and while both changed my life, the children changed it more. I remember sitting in the corner of my room one night after putting Arden and Pierce down and asking myself whether I could write what I believed on a single sheet of paper. I couldn’t. That was the moment I started reading seriously, beginning with the ancients, who thought about these questions deeply and genuinely (my original list is here). Having children makes the question of what you’re actually “for” impossible to defer.

I now look for opportunities in my own children’s lives for containers that place them in contact with reality, responsibility, and admirable adults. The hard part is that the best formation is often a byproduct rather than something you can engineer directly. You can build the conditions for it, but you usually cannot force it.

asks:

You argue that philosopher-builders need explicit moral commitments to avoid optimizing for the wrong things. But your three pillars (truth-seeking, autonomy, decentralization) are themselves a normative framework that not everyone shares. China’s AI strategy is still coherent, explicit, and philosophical, it just starts from different premises. So how do you argue for your philosophy without just replacing one set of defaults with another? What makes Cosmos’s values the right foundation rather than just a well-packaged preference?

Hi Turner, you’re right that truth-seeking, autonomy, and decentralization are substantive commitments. I think they matter less as one moral doctrine than as conditions that keep moral life from collapsing into force or drift.

If you consider moral frameworks from Confucianism to Christianity to Marxism, for them to have legitimate force over a person, that person has to be able to genuinely endorse it (otherwise, you don’t have a moral commitment). That endorsement depends on human autonomy – which is to say, the capacity to reflect, evaluate, and take something on as your own rather than merely inheriting it or obeying it. So autonomy is not just one preference among others. It is the deep substratum that makes moral commitment possible at all.

Take utilitarianism: Jeremy Bentham devised a system that, in my mind, dissolves individual judgment into aggregate utility. This is in conflict with autonomy-as-an-end. And yet building this system was itself a radical exercise of autonomous reason. Every person who adopts utilitarianism is exercising the same capacity. You can’t be a utilitarian in any meaningful sense unless you’ve freely taken it on. So even a framework that subordinates individual judgment to aggregate welfare requires individual judgment to get off the ground. Now, someone could say that only makes autonomy instrumentally necessary, not foundationally important. I think that view is unstable, because the goods autonomy is supposedly serving only become moral goods for a person if they can in some real sense take them on as their own.

There are hard cases. In the Ash’ari tradition in Islamic theology, divine command constitutes moral value rather than being something reason independently discovers and then endorses. That’s a genuine challenge to autonomy as foundational. But even there, the person who freely chooses submission is doing something categorically different from the person who never had the choice. And a secular collectivist can make a parallel argument: that harmony or collective flourishing is the true precondition, because no individual life goes well outside a stable social order. I think that is partly right. But unless people can participate in judging the terms of that order, harmony becomes coordination imposed on them rather than a good they share in shaping.

Truth-seeking has a similar status. Any framework worthy of allegiance has to remain in contact with reality. It has to be open, at least in principle, to its own refutation. If someone could show me that decentralization produces a worse outcome in a domain I care about, I’d have to take that seriously and I would. Systems that suppress truth-seeking can be internally coherent in the way that closed systems are coherent. But they can’t tolerate the mechanisms that would let them find out they are mistaken. That’s a serious defect, especially if you think we’re all operating under real uncertainty about what AI is going to do to human life.

And decentralization follows from the same logic at the institutional level. If no person or committee is wise enough to determine the good for everyone, then we should be wary of a small number of actors hard-coding their anthropology into the substrate of society. Decentralization is valuable because it preserves room for people to try things, get them wrong, and leave, keeping mistakes from becoming total.

So to your question about China: yes, their AI strategy is coherent, explicit, and philosophical. But coherence purchased by foreclosing the capacity for self-correction is brittle in exactly the way that matters most right now, though that is still a bet, not a proof. I’m also skeptical that it preserves the standing of human beings as agents capable of judgment rather than increasingly treating them as objects of coordination. A framework can be philosophically serious and still be wrong about what a person is.

So I would not say Cosmos is advancing a one true final doctrine that every civilization must affirm. That would be too strong, and it would collapse into exactly the kind of totalizing move you are warning against. I would say instead that Cosmos is trying to defend the conditions under which free people can genuinely seek truth, make judgments, form commitments, and build different kinds of lives together, without having orthodoxy imposed on them by default.

asks:

Will the Cosmos Grants ever open again?

We’re planning to re-open these in the next 60 days. We’ve just had the demos from our latest batches of winners. Stay tuned for updates!

asks:

Assuming frontier large language models, together with their multimodal and agentic extensions, are trained to effective saturation on an exhaustive corpus that represents the totality of digitized human knowledge including all scientific publications, books, patents, archival records, cultural artifacts, and recorded conversations, will these systems be capable of transcending the statistical manifold of their training distribution to autonomously discover, validate, and iteratively expand novel knowledge beyond the current human frontier?

I’m not sure scientific knowledge is a kind of territory where data defines a bounded region that a model may or may not be able to venture beyond. The affirmative view of this picture strikes me as broadly empiricist insofar as knowledge comes from data, and scientists make discoveries by extrapolating beyond what they already know. Your specific examples, though, are actually the strongest case for the affirmative: theorem provers, simulators, multi-agent workflows, and verifiable rewards are exactly the kinds of feedback-rich settings where I would expect systems to extend the frontier.

But that is not the only picture of science, and I do not think it is the deepest one. Science also advances by reorganizing what researchers take to be meaningful in the first place: which anomalies matter, which questions are worth asking, and which explanations count as illuminating rather than merely predictive.

Systems can already generate novel candidate hypotheses, and in domains with strong automated verification, they may well extend the frontier. Formal mathematics looks especially promising, because conjecture can be paired with proof or disproof inside a relatively crisp evaluative architecture. In such cases, I expect AI systems to produce results that are genuinely new to humanity. Just this week a constellation of agents improved a math problem that’s been open since Newton (Kissing Number in dimension 11: 593 → 604). That is impressive. It is also, I think, a good example of the distinction I’m drawing: a real extension of an existing line of inquiry, but still closer to powerful normal science than to scientific revolution.

But that does not settle the larger question. There is a difference between producing novelty within an existing framework and generating a new framework altogether. A system may help prove a theorem, optimize a search, or identify that drug X affects disease Y, all without altering our understanding of why the problem is structured as it is. That is a real scientific contribution, but it is not reorganizing the conceptual landscape.

The harder question is whether these systems can exercise scientific judgment in the richer sense: whether they can tell which anomalies are significant, which inconsistencies are fertile, which explanations deepen understanding rather than merely extend prediction, and which questions are worth reorganizing inquiry around. That is a higher bar than novelty, and I am not yet convinced we know how to evaluate it well. Part of what makes this hard is that frameworks are underdetermined by data. The same body of results can often support multiple lines of inquiry, and judgment is what tells you which one is worth building a field around. That remains, to my mind, the deeper open question.

asks:

Most of the current work on ‘AI, collective epistemic structures and decision-making’ focuses on filling gaps: more participants, faster information exchange, more efficient decision-making. This will help with many problems, but certainly not with the most complex ones, because it just accelerates the practical execution of the same thought styles that led to the problems. Therefore: How can we use future AI to foster new thought styles that are currently not supported by our existing social structures?

Yes, I think the intuition that better collective decisions will emerge if we simply gather more data from more people more efficiently breaks down at the limit. That can improve performance within an existing paradigm, but it does much less when the paradigm itself is the problem.

What groups develop over time are not just bodies of knowledge, but epistemic constitutions: implicit rules about what counts as evidence, which questions are legitimate, who gets to propose, who gets to criticize, and on what terms. Mill saw part of this in his account of the tyranny of prevailing opinion and the epistemic importance of dissent. But the problem runs deeper than opinion alone. Entire institutions decide in advance what counts as serious thought.

So one promising use of AI would be to make those constitutions more visible. A good system might show a research community, an organization, or a polity where its methods systematically exclude certain questions, place some assumptions beyond criticism, or discount certain voices before the argument even begins. In medicine, for example, it might reveal a field that privileges what is easily measurable while sidelining patient testimony or long-horizon effects that do not fit the dominant method.

But diagnosis is only the beginning. I like this direction because the problem is often not that new thought styles do not exist. It is that they remain stranded at the margins because the reigning structures of legitimacy suppress them. And sometimes the deeper problem is that the social conditions required for a new thought style have not yet been built. New thought styles need protected spaces, alternative standards, and enough provisional legitimacy to develop before the dominant paradigm dismisses them. In that case, the most useful contribution AI could make to collective epistemics is not novelty on demand, but widening the space in which criticism, recombination, and intellectual minority formation can occur.

asks:

What is your definition of intelligence? When AI reaches ASI in the future, do you think it will be safe and aligned? As a species, what is our purpose after ASI world? How can thrive as a species?

I’d resist the standard definition of intelligence as raw problem-solving horsepower. For me, intelligence is the capacity to learn from reality, inquire into it well, and let it correct you.

Part of what I like about François Chollet’s work, and why ARC Prize has mattered, is the insistence that intelligence is not the same thing as accumulated skill. A system can look impressive because it has absorbed an enormous amount, or because the task has been made easy for it. The more interesting question is how much it can learn from limited experience, under real constraints, and still generalize well.

But I do not think that is enough on its own. Leslie Valiant’s idea of educability gets closer to the human picture (see his Cosmos Lecture from last year here). Human intelligence includes the capacity to learn from experience, receive instruction, integrate both, and apply them in new circumstances. What distinguishes the human mind is not only that it learns, but that it can be taught and formed.

And I would add one more layer. Drawing on HAI Lab director Philipp Koralus, I think reasoning is fundamentally question-directed. Minds are shaped by the questions they pursue. They go wrong through shallow questions, premature closure, and a failure to inquire far enough, just as much as through false conclusions. That matters for AI because a system can become very good at answering questions while still narrowing the range of questions humans ask, or rewarding closure where inquiry ought to stay open.

That is why I’m less interested in arguing about whether AI will count as “superintelligent” than in asking what it does to human intelligence. A system can be extraordinarily capable and still erode our capacity for inquiry, judgment, and self-government. That is the danger I worry about most.

On whether ASI will be safe and aligned: I do not assume that can be taken for granted. I would trust highly capable systems only to the extent that they remain corrigible, contestable, and embedded in institutions that preserve human judgment rather than replacing it. The problem is not just getting the objective right once. It is making sure people can still question, revise, and refuse the system’s guidance when it matters most.

As for human purpose after ASI, I do not think our purpose changes. If anything, it comes into clearer view. We are not here to compete with machines at speed or scale. We are here to exercise judgment, form character, build institutions, love particular people, and deliberate about the good. That last point matters more than it may sound. Love is not interchangeable, and responsibility is not abstract. A more powerful machine does not make our obligations to particular human beings less central. In a world of highly capable AI, those things become even more important.

asks:

Is it possible to carry on our learning from humanity up until now and jumpstart a new society with the help of AI, assuming that we can concentrate and utilize all the data we accumulated up until now? At what point does the need for a new constitution or a new world order arise and how do we know it?

In short, no.

Firstly, I don’t think “all the data we accumulated up until now” is the same thing as the total weight of human knowledge. Much of the knowledge that keeps a society functioning is tacit, dispersed, and unwritten. Some of it lives in practiced judgment: an ICU nurse sensing that a patient is about to crash before the monitor shows it. Some of it lives in inherited forms: the habits of trust, restraint, and association on which a free society depends, even when no one can fully specify them. As Michael Polanyi put it, we know more than we can tell.

More importantly, I’d push back on the idea that we can jumpstart a society at all. Societies aren’t machines that you design to a blueprint. Tocqueville saw this in the institutions of local self-government. Hayek saw it in the way social orders carry dispersed knowledge that no planner can gather in full. A free society is learned in practice through things like townships, juries, churches, and associations. Those are the ordinary disciplines by which people become capable of governing themselves.

The question is whether our institutions can still sustain a free people capable of self-government under new technological conditions. And that does leave open the question you raise about constitutional inadequacy: how do we know when inherited arrangements are no longer enough? I do not think there is a clean threshold. Usually the signs are visible first in practice, when institutions that once formed judgment begin producing passivity, dependence, or elite insulation instead.

When they cannot, the answer is not a tabula rasa redesign of “the new world order.” I would look to renewal through institution-building, and Benjamin Franklin is the example I keep returning to. He took an Enlightenment conviction — that access to knowledge should not remain under the custody of church, state, or a narrow elite — and embodied it in an institution. The subscription library made a philosophy of freedom socially real. That is why Franklin still matters to me here. He shows what it looks like to translate a philosophy into civic machinery. We need the AI-age equivalent: institutions that widen access to knowledge and judgment without concentrating them in a few hands. We need philosopher-builders in that spirit again.

asks:

How can we motivate our children to learn at school? Should we try to motivate them or find rather a way out of the system? (e.g. reading more classical books, rather than encouraging them to read what school nowadays gives?)

Young people have a natural longing to be seized by something greater than themselves. To be captivated. The promise of liberal education, going back to the Greeks, is that there are magnificent ways of living, and magnificent questions about how to live, and that encountering them through great minds and great books can awaken a desire that organizes everything else.

The disaster of modern education is that it has taught young people their longing is naive. That no book is really better than another, that no life is really higher than another, and that the hunger to be drawn upward by something extraordinary is itself a kind of error.

So I would say motivation is the right place to focus, but we should be precise about what we mean. There is a kind of motivation that is intrinsic: the eros I just described, the desire to encounter greatness because it calls to something real inside you. And there is extrinsic motivation: incentives, structure, well-designed systems that make it easier to do the work. Both matter. The best schools I’ve seen, including Alpha where my kids go, are serious about the extrinsic architecture. They’ve built an environment where children actually want to show up and work.

Extrinsic design clears the path, and then you have to light the fire. The fire is eros, and it’s fed by contact with things worthy of love: books, questions, lives, guides who still care about these things enough to take them seriously in front of children.

Whether that happens at school or at home is incidental. What matters is that a child sees adults who are genuinely stirred by ideas, who return to certain books not because they were assigned but because they can’t leave them alone. A six-year-old can learn a lot about what seriousness looks like by watching someone practice it.

asks:

The accelerationist world still seems to dominate the public narrative by communicating in everyday language on everyday platforms in a way that meets people where they actually are. Meanwhile, even the most accessible AI ethics content tends to assume familiarity with Mill, Tocqueville, or Heidegger. The philosopher-builder framing is compelling to people already inside the tent. How does Cosmos think about the people outside it? Is philosophical depth a feature for the community you’re building, or a barrier to the broader cultural shift you want to see?

The honest answer is that depth is the point. If we watered down the philosophy so we could meet everyone where they are, we’d be producing the same frictionless content you see elsewhere. Philosophical seriousness creates a negative selection gradient, and we want that. The people who do the reading are the people most likely to build something different.

But “depth” and “jargon” aren’t the same thing. A lot of AI ethics writing assumes you’ve already read Heidegger or whomever, which risks filtering out precisely the builders who might be transformed by reading him. I know this because I’ve made the mistake myself. When I started writing this Substack I leaned on more jargon than I needed to, and I’ve had to learn over time how to make the ideas more accessible without making them thinner.

The people outside the tent aren’t who you might think. I sold two companies and wrote a national AI strategy, and I couldn’t write what I believed on a single sheet of paper. There are a lot of capable builders out there who never had anyone hand them the books or sit with them through the hard parts. Cosmos partly exists because I was one of them. The audience for this is bigger than it looks.

Where I’d push back on your framing is the implicit suggestion that the accelerationists win because they’re more accessible. They have their own jargon. Try reading about negentropy, Kardashev III, and thermodynamic civilizational substrate for the first time. What they’ve done well is compress a real conceptual core into memes that travel. I respect that.

The challenge for us is that some ideas compress more easily than others. “Build faster” is more memeable than “cultivate judgment.” “Technology goes up” fits on a poster. “The conditions under which free people can exercise genuine choice require institutional renewal” does not.

This logic holds for political movements more generally: the larger the audience you try to build, the cruder the message has to become. The lowest common denominator wins by default, not because it’s right but because it compresses. I don’t think the answer is to compete on that terrain. I think it’s to make the longer argument compelling enough that people seek it out, and to be honest that not everyone will.

The harder truth is that we live in a culture of secondary orality where the long coherent essay is increasingly marginal. That’s a loss. It makes what we do at Cosmos more countercultural than it would have been fifty years ago, but it also makes it more necessary. The essay, the book, the salon: these are the forms where ideas actually get tested rather than just transmitted. We’re not going to stop producing them because the culture has moved on. If anything, the fact that sustained argument is now unusual is exactly why it matters.

asks:

For someone coming to AI without a technical background but with a strong interest in understanding its societal and philosophical implications, what foundational books or resources would you recommend?

Second, as a parent, I’m thinking about how to prepare my kids for a world where AI is increasingly embedded in everyday life. Beyond basic digital literacy, what kinds of skills, habits, or ways of thinking do you believe will matter most for the next generation? Are there age-appropriate tools or frameworks you’d recommend for introducing AI concepts early in a thoughtful, not just utilitarian, way?

Hi Emily :)

I’ll take the kids question first because it’s closer to my heart.

The risk I think about most is what I’ve called “autocomplete for life”: the possibility that AI systems will increasingly shape not just what our children do but how they deliberate about what’s worth doing. Each small delegation of judgment seems harmless. But together, they habituate a person away from self-governance and toward dependence. The question for parents is how you build resistance to that drift before your child is old enough to name it.

Our ancestors needed to know how to make bread. We need to know where to find the recipe. The next generation will need something different again: the capacity to think about how they think, in relation to systems that could do the thinking for them.

In our household, the main way we work on this is Socratic conversation. Arden and Pierce do weekly sessions with Michael Strong built entirely around questions. “What’s the difference between a bird and a plane?” “What does it mean for something to be alive?” “When mommy and daddy disagree, who is right? What about daddy vs. AI? What about AI vs. AI?” A child who has practiced working out what they believe, and who has had to think about whether to trust their own judgment or defer to an external authority, is better prepared for a world of algorithmic suggestion than a child who has learned to code.

I also want my kids to be entrepreneurial. When America was founded, around 80% of free workers were self-employed on farms or in small crafts. Today that number is about 10%. We became a society of employees, and something atrophied. As the economy changes again, the ability to know yourself, act on what you believe, and build something from that conviction will matter more than any technical skill we could teach them now.

On resources for someone coming to AI without a technical background: I’d start with the question of what AI does to us rather than how AI works. A couple of recent pieces that I’d recommend are ’s Musings on Self-Recursive Improvement and ’s What Will Be Scarce. For ongoing reading, , , and regularly write about AI and society. and have a wider aperture and I often find them thought-provoking. For anyone interested in AI’s effects on democracy and self-governance, Harvey Mansfield’s Tocqueville: A Very Short Introduction is the best ~100 pages you could spend. Tocqueville saw the drift toward comfortable dependence coming two centuries ago. The application to AI is left to the reader, but it isn’t hard to find.

asks:

My sense is that the vision animating Cosmos has deep predecessors not just in classical philosophy but, in my impression, religious eschatology. Teilhard de Chardin’s Omega Point or Augustine’s City of God, and even secular variants like Condorcet’s perfectibilism all share your orientation toward civilizational-scale transformation in service of human flourishing.

More explicitly, your pillars of reason, autonomy, and decentralization also echo the long Aristotelian and classical liberal tradition from Mill to Tocqueville.

So, what does Cosmos contribute that is genuinely novel in its normative architecture, rather than a restatement of those traditions in the presence of AI? And if it is largely a restatement, is that a problem?

I think you’re closer to the mark with some of these influences than others.

Teilhard, Augustine, Condorcet: I share their impulse toward civilizational-scale thinking, and I take it seriously. But for all their differences, they are ultimately teleological writers. They saw history as the unfolding of a determined, directional arc. At Cosmos, we want to keep the conditions open that allow people to find their own path. We’re not about to get into eschatology.

You are, of course, completely right about Aristotle, Mill, and Tocqueville, and we regularly acknowledge our intellectual debt to them. I don’t think the pillars need to be new to be worth defending, and I’d be suspicious of anyone claiming to have invented a wholly new account of human flourishing in 2026.

For me, the interesting question isn’t whether Cosmos has discovered a value nobody thought of before. Instead, it’s whether an old set of commitments can survive as a living practice. Mill didn’t have to ask whether the harm principle could be encoded in a model’s training objective. Tocqueville didn’t have to think about what decentralization looks like when the substrate is compute rather than townships, when the everyday infrastructure of life anticipates your choices rather than forcing you to deliberate, associate, and decide alongside your neighbors. When your community is mediated by algorithmic curation and your civic life is shaped by systems you never consented to and cannot inspect, the Tocquevillian question of how free people learn to govern themselves together doesn’t disappear. It becomes harder, and the institutional forms it requires don’t exist yet.

That’s where your last point lands, and I think it’s the right one. The proudest achievement of the eighteenth century was the translation of philosophy into law: Enlightenment commitments about liberty, consent, and the rights of individuals became encoded in constitutions and legal systems that gave them institutional force. The challenge of the twenty-first century is the translation of philosophy into code. The commitments are old. The work of making them operative in the infrastructure that actually governs daily life is new, and it is the work Cosmos exists to do.

But I wouldn’t call what we’re doing a restatement. Restatement is what you do in a seminar. Institutional embodiment is what you do when you think the ideas actually matter and must be operative in the AI age.

asks:

What do you think of the prospects for a stable, left-right coalition on AI in favor of sensible regulation and general cautious optimism that includes religious conservatives and secular social democrats? Or will this get polarized across political lines like everything else?

On the coalition point, I can already see signs of this. Religious conservatives and secular social democrats agree on little, but they intuitively grasp some things that many accelerationists don’t: that people are formed by their communities, work and dignity are connected, and that we shouldn’t try to optimize society into passivity. I’d also throw old school liberals into that coalition too. In the coming years, I’m sure there’ll be scope for productive, broad-based conversations about kids, loneliness, work, and communities.

Where I’d push back is the idea that any future coalition should coalesce around “sensible regulation.” I don’t think regulation is the best tool for addressing most of these concerns. Treating it as the default is how you end up with something like the EU AI Act, a classic example of doctor-induced illness. It created a compliance moat that only the largest companies can afford to cross, while doing essentially nothing to address the risks it was supposed to mitigate.

The more productive ground is further upstream. What are we building? What do we fund? What should we teach? What institutions do we need to form? A coalition focused on those questions would look less like a regulatory body and more like a network of individuals doing the building, teaching, and funding that no regulation can mandate.

I’m less worried about polarization acting as an obstacle here. Much of this work sits outside electoral politics at the moment, and as far as I’m concerned the longer that remains the case the better. Partisan dynamics reward exactly the kind of simplification that makes these questions worse. The moment AI becomes a left-right issue, the entire conversation becomes about how much to regulate, and the question of what to build for never gets asked.

asks:

Here is my question: Your three pillars (truth-seeking, autonomy, and decentralisation) are compelling at the individual level. I am curious how you think about them when the actors are states rather than individuals. The US-China AI dynamic, for instance, seems to run against all three: opacity rather than truth-seeking, control rather than autonomy, and concentration rather than decentralisation. Does Cosmos’s framework extend to the question of how countries could potentially cooperate on AI, or does that require a different philosophical foundation entirely?

Thanks Alina, great question.

The pillars were designed with individuals and institutions in mind, so extending them to the state level requires real philosophical work.

Fichte took the Kantian account of individual autonomy and argued that it applied to nations: a people that cannot determine its own form of life is unfree in the same sense an individual under tutelage is unfree. The autonomy pillar, taken seriously, has a national analogue. So does truth-seeking: a polity that can’t inquire openly into its own condition is in the same trap as a closed mind. And so does decentralization: a world of self-governing peoples is the international expression of the same instinct that makes you wary of concentrated power inside a country.

But Fichte also shows you what happens when you scale autonomy alone. His attempt to extend individual self-determination to the collective ended in arguments for the unique world-historical mission of the German nation, an autarkic closed state, and the exclusion of those who didn’t fit the national community. The lesson isn’t just “be careful.” It’s that the three pillars need to travel together. Autonomy without truth-seeking becomes self-righteousness. Autonomy without decentralization becomes domination. What checks national self-determination is the same thing that checks individual self-determination: openness to correction and the refusal to concentrate power beyond what can be held accountable.

On US-China, the goal isn’t a single global regime that imposes one model of AI governance on everyone, because that would violate the decentralization commitment at the international scale. The better question is: what conditions allow distinct political communities to develop AI in line with their own forms of life without crushing each other in the process?

And what happens when a community’s “form of life” involves suppressing the autonomy of its own citizens? The pillars can come into tension here. Respect for national self-determination and respect for individual autonomy pull in opposite directions.

This is where Tocqueville matters most. The meaningful unit of self-government is rarely the nation-state on its own. It’s the dense layer of associations, communities, firms, religious groups, and local institutions that sit between the individual and the state. Any serious thinking about international AI governance has to make room for those middle layers. Tocqueville saw that democratic freedom doesn’t live in declarations from the center, but in the practice of self-government at the local and associational level.

asks:

Can you set up a path for crowdfunding projects or contests to realize ideas that the Cosmos Institute seeds?

Interesting. Not something we’ve considered, but we’ll think about whether there’s a model that works for us.

asks:

Do you plan to have online cohorts?

No plans right now, but we may consider it in the future!

asks:

I see all these wonderful essays and people doing great work. Honestly, I would love to interact with the community + become part of it in some capacity. Is there a channel where people who are interested in the ideas that Cosmos is working towards but not necessarily are academics or builders can communicate with the fellows and the team?

Not right now, but we are thinking about whether there’s something we can do here!

asks:

I’m persuaded by the concern about cognitive risks and the need for “AI for epistemics,” “deliberative AI,” etc. Do you know of organizations developing benchmarks around the goal of bolstering critical thinking and improving epistemic processes and outcomes?

We have some grant projects that have focused on this. Two that come to mind are DeliberationBench, which assesses AI persuasion in comparison with diverse human discussion, and Priori, a tool that surfaces hidden assumptions when you are interacting with an AI model. Two of our grantees (Steven Molotnikov and Cathy Fang) are running a research study on how Priori and related human oversight interfaces work in practice.

I think there is a wave of energy in this area. Various orgs are thinking more about AI for Human Reasoning (with Future of Life Foundation funding work in this area, Forethought writing about it, and Elicit working on directly in the for-profit space). Also anecdotally I hear researchers thinking more about ideas like “epistemic security” or “cognitive security” or “cognitive sovereignty” as well as ways to improve information environments without restricting speech and expression.

I share your enthusiasm for more work in this area – both on benchmarking but also technology that better enables open contestation of ideas (inspired by classical liberal premises, and Mill’s ideas on this). If readers are working on this please do reach out!

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

This week, we crossed 20,000 subscribers on Substack. Thank you to everyone who has read, shared, and engaged with our work.

We’ve written about Claude Boys to Coasean bargaining to the perils of liberal nudging. Reading the comments has often been as rewarding as writing the posts. To mark the milestone, I’ll be answering your questions on Wednesday April 15.

Drop your question in the comments below and upvote the ones you want answered. I’ll start responding next week and I’ll try to take as many as I can.

There are a few things I’ve been thinking about that we haven’t written about yet. This seems like the right place to start.

Ask about Cosmos, human autonomy, AI x philosophy, or what people in our network are building. Especially questions that are hard, that relate to how we approach AI as builders, or that challenge our assumptions.

- Brendan

Leave a comment

In the autumn of 1809, Prussia was a country that no longer knew what it was for. Three years earlier, Napoleon had destroyed its army in an afternoon and walked into Berlin without resistance. The king fled. Half the territory was gone, the treasury empty. French soldiers were still garrisoned in the capital.

As Prussia began rebuilding from the wreckage, most people assumed it needed more officers, administrators, and engineers. People who could do things. The task of designing the new system of education fell to a thirty-two-year-old diplomat named Wilhelm von Humboldt. He gave them something else entirely.

In a series of memoranda written over the next year, he laid out a vision for a new university in Berlin organized around Bildung. The word has no English equivalent. “Education” is too narrow, “self-improvement” too thin. “Formation” gets closest but still misses its moral weight.

Humboldt’s Bildung means the free, harmonious development of a human being’s powers into a complete and consistent whole, through encounter with the world in its variety and resistance.

Mill, who took the idea from Humboldt, put it more simply: a human being is more like a tree than a steam engine.

Humboldt proposed a university where professors and students would be joined in the pursuit of knowledge, unconstrained by political demands. In a defeated nation hungry for officers and administrators, he was arguing for formation before function.

The ideal of the modern research university, with its union of teaching and inquiry, its seminar culture, and its commitment to academic freedom, descends from what Humboldt designed in those desperate months.

Today, the culture of Bildung that animated the university survives only at the margins, sustained by people stubborn enough to work against the grain of the institutions around them. Credentialism twisted the university into a vendor of certificates, and the formation of the student as a complete human being came to seem anachronistic. The cathedrals remain, but not the faith.

We are in a moment that rhymes with Humboldt’s own. Technological pressure is once again pushing education toward the practical. The evidence of institutional collapse is everywhere: flagship universities slashing PhD admissions, hundreds of degree programs have been eliminated, dozens of small colleges have closed, and a decline in the college-age population still lies ahead.

The loudest responses to the crisis have come from outside the university. Alex Karp tells young people to skip college and learn a trade. Marc Andreessen argues the university is a credentialing middleman and should be disintermediated. Both are right that the university is failing. But if the answer to a broken formation system is to skip formation altogether, you have already conceded that education is justified only by utility.

Neither is asking the question Humboldt asked: What is a human being, that education should serve it?

Subscribe now

The Tree and the Trap

Before Humboldt, the Prussian system assigned students to a vocation, trained them in it, and delivered them into their function. Even on those terms, Bildung wins. Broadly cultivated judgment produces better doctors and better engineers. Technical skill on a foundation of general cultivation is more resilient and more humane than technical skill resting on nothing.

But if that were the whole justification, Bildung would be nothing more than a roundabout way of minting skilled professionals. A pedagogy can be justified by output. Bildung cannot.

A tree does not exist in order to produce lumber. You can make lumber from it, and good lumber is nothing to sneer at. But if you look at a tree and see only lumber, you have missed what is standing in front of you. Something is growing there under its own power, toward its own form, and the growing is not a means to some further end.

Humboldt’s claim about human beings is the same shape. A person is a self-developing being whose worth is not exhausted by function.

Bildung braids together the Kantian claim that no person is merely a means, the Greek ideal of harmonious excellence across mind and body and character, and the Romantic conviction that we develop through encounter with what resists us.

You have had the experience even if you never had a word for it. Real engagement with something that has its own demands—a hard problem, a serious book, a gifted teacher—changes who you are. You could not have planned the person you became.

Such formation is not the property of any particular university department. This is not a “save the humanities” argument. A coder who, after tackling a hard systems-design problem, comes out thinking differently about complexity, tradeoffs, and the limits of formal reasoning has undergone a kind of Bildung—but only if the encounter changed who they are, not just what they can do.

The resistance to all this is understandable.

In a world where your economic value can evaporate overnight, “Become a whole person” sounds like advice from someone who has never worried about next month’s rent. The utilitarian case for education has the force of necessity behind it. For millions of people, making yourself useful is what responsibility to their families demands.

Yet if the response to being replaceable is always to train for a different function, you have entered a race you structurally cannot win. The principle that makes your education valuable is the same principle that makes you disposable the moment the function migrates.

The scramble into computer science was an early sign of the trap: students rushed toward the field that seemed safest, and then AI began destabilizing the very functions it trained them to perform. The flight to function looks rational from inside it; that is what makes it a trap.

Solitude and Freedom

Bildung cannot be specified in advance. Once you define what the formed person looks like, you have replaced formation with training. So how do you build institutions around it?

Humboldt’s solution was to design an environment rather than a curriculum. He based the University of Berlin on two principles: solitude and freedom, though they meant something precise in his hands. The university would not answer to the demand for immediate use and inquiry would follow the question wherever it led, unconstrained by predetermined ends.

The undoing of that design came in two phases.

First, a slow hollowing. The German research model crossed the Atlantic when Johns Hopkins was founded in 1876 on Humboldt’s model. But disciplines hardened into guilds and career tracks, postwar federal funding built research enterprises increasingly detached from teaching, and mass enrollment turned the degree into a sorting mechanism. The university no longer had a single organizing purpose, and no one was left responsible for the student as a whole person. By the mid-twentieth century, the university president Clark Kerr could describe the resulting “multiversity” as a collection of unrelated enterprises held together by a common heating plant.

Then, a rejection of the cure. Recently, at the University of Tulsa, the philosopher Jennifer Frey built a Great Books honors college from scratch. It was a serious formation program inside an institution organized around credentialing. The university removed her, restructured the program, and replaced the deanship with a directorship.

Back to Scholé

But even if the institutions had held, formation requires something they could never secure at scale: time.

Aristotle called it scholé. Humboldt had a related word for it: Muße. Both named a kind of structured freedom for the work of becoming, and for most of history that freedom was radically exclusive.

Aristotle could imagine the highest forms of human flourishing only for those relieved of labor by wealth and the work of subordinates and slaves. The good life required freedom from necessity, and in his world only a few could have it. But in the first book of the Politics he imagined something stranger: that if shuttles could weave by themselves and picks could play the lyre, craftsmen would need no subordinates and masters would need no slaves. The “self-guided machine” would mean that the material basis for leisure no longer depended on the unfreedom of others. It is one of the oldest thought experiments in Western philosophy, and we are now enacting it.

Aristotle did not celebrate the prospect. He understood that freedom from necessity does not automatically yield the pursuits that make such freedom worth having. In his account, those with wealth and leisure often turned to unlimited acquisition or bodily gratification rather than to the activities that justify leisure in the first place.

With AI, we are building something like self-guided machines. Whether these systems liberate or merely displace is not settled. But the possibility of leisure at scale is real enough to become a serious question.

If AI can compress parts of instruction, it may deepen learning where it is used and clear ground for formation where it gives time back. But only if it preserves productive struggle rather than bypassing it.

The alternative is already visible: autocomplete for life. Not just help with expression, but the slow outsourcing of judgment itself. That is Bildung’s antithesis.

Worse, the same technological society enabling leisure is also shaping the desires of the people who receive it. If our dispositions have already been trained toward optimization and outsourced judgment, the freed hours may arrive in hands that no longer know what to do with them.

For most of history, the conditions of formation were reserved for the few. The capacity for it was not. If scholé at scale is now possible, refusing to pursue it ratifies a world in which full human development remains the privilege of those who can afford time.

In the Afterglow

Bildung is, at its normative core, anti-servility: the effort to form people who cannot be reduced to instruments of external authority, whether state, market, or algorithm.

The people who use AI well right now are drawing on judgment they formed before these tools became ambient. They know when to trust an LLM’s output and when to push back because they learned to read, argue, and sustain attention under conditions in which those acts were not so easily outsourced. They bring something the tool cannot supply.

Nietzsche thought secular liberals were living off the moral capital of a Christianity they had officially abandoned: inherited capacities that could persist for a time even after the culture that formed them had ceased to renew them. The kind of judgment this essay is defending may be a similar afterglow, formed in a world before AI mediated everything.

Without that judgment, you get agency without autonomy.

If the capacities required for non-servile life in an AI world were all formed in a pre-AI world, what happens when that formation stops? You can live on an inheritance for a while. You cannot educate a civilization on inherited judgment forever.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Subscribe now

The university is under siege.

Harvard, sitting on a $57 billion endowment, slashed 75 percent of its science and 60 percent of its humanities PhD admissions. Indiana mandated the elimination of over 400 degree programs at its public universities. Classics, comparative literature, and foreign languages at the state flagship are gone. At the University of Tulsa, a philosopher built a Great Books honors college from scratch, grew enrolment 500 percent, and attracted a quarter of each freshman class to a curriculum running from Homer to Arendt. The university removed her.

Sixty-four small colleges have closed since 2020, and the demographic cliff, a 13 percent decline in college-age students projected through 2041, hasn’t even started.

Meanwhile, 52 percent of recent graduates are working jobs that don’t require the degree that they borrowed $30,000 to get. Palantir CEO Alex Karp is telling young people to skip elite colleges entirely, saying the only paths left are skilled trades or neurodivergence.

AI will accelerate this. The skills the degree was supposed to impart – researching, analyzing, drafting, coding – are increasingly things a machine can do.

As the old institutions break down, a new generation of educational founders is running experiments. New schools are emerging, each making a different bet on two questions that anyone building an educational institution has to answer:

• The first question is about purpose. Do you focus on specific, trainable skills, or on a broader, if more intangible formation, which includes the development of judgment, attention, and moral seriousness?

• The second question is about technology. Do you build AI into the core of the educational experience or keep it out because the difficulty is where the learning happens?

These two axes produce a landscape.

At one end, programs like Gauntlet train elite AI engineers in ten weeks: 80-100 hours a week of building, a guaranteed $200K job offer at the end, costs paid entirely by employers.

At the other, institutions like St. John’s College run intensive liberal arts programs built on sustained attention and close reading, with no AI in sight.

Arizona State has embedded AI from coursework to advising and has struck partnerships with a number of high-profile AI companies, but the core teaching model hasn’t changed.

The University of Austin has proposed splitting the day between device-free seminars and intensive AI work.

Meanwhile, the bulk of the higher-education system is either banning AI from the classroom or pretending that it doesn’t exist.

These questions are not new. They go back to the founding of the first American colleges and European research universities.

In partnership with Liberty Fund, Cosmos held a seminar to think through them with a group of founders, scholars, and institution-builders. The participants included philosophers from UT Austin and Ohio State, researchers from MIT Media Lab, RAND, and leading AI labs, education policymakers from the Texas Higher Education Coordinating Board, and builders from Fractal Tech and Alpha School.

The reading list we assembled traces these tensions to their origins: from the earliest American proposals for public education through the founding documents of the research university, the mid-century debates about liberal learning, and the first serious writing about what computers might do to the relationship between learner and knowledge.

Subscribe now

Session I: The Promise and Crisis of the University

We are not the first to ask what the university is for.

This session combined five voices, spanning three centuries. Franklin wants a practical institution for a practical republic. Humboldt argues that the university exists for the development of the whole person, not the production of useful professionals. Oakeshott insists that education is how a human being becomes one. Bloom diagnoses what happens when that inheritance is abandoned. And Caplan asks whether the whole enterprise is a $240,000 receipt that signals you can show up on time.

If Caplan is even partly right, the reformers need to explain what they are offering that a credential cannot capture. If Humboldt is right, the reformers need to explain why two centuries of institutions built on his model ended up producing exactly the credentialism he warned against.

• Benjamin Franklin, Proposals Relating to the Education of Youth in Pennsilvania. (link)

• Wilhelm von Humboldt, “On the Internal and External Organization of the Higher Academic Institutions in Berlin”. (link)

• Michael Oakeshott, The Voice of Liberal Learning, Introduction.

• Allan Bloom, “Our Listless Universities.” (link)

• Bryan Caplan, The Case Against Education, Chapter 1, “The Magic of Education.”

Session II: What Formation Requires

If the first session asked what the university is for, the second focused on what actually changes a person. Weil makes the strongest claim: attention is all you need. Klein argues that liberal education requires putting your own opinions genuinely at risk, and that the resistance to doing so is rooted in something specifically human: we build our identities on what we think we know. Gadamer argues that practical wisdom cannot be taught as a curriculum; it grows out of ethos, the character already formed by living in a particular community.

But what if the boundaries of the person aren’t where we think they are? Clark and Chalmers ask whether cognition is even confined to the skull. If the mind extends into the tools and environments we think with, what counts as “the student”? Shanahan complicates the picture further: LLMs create a compelling illusion of understanding, but they are fundamentally unlike us. If formation requires genuine encounter with other minds, what happens when the most available interlocutor is a machine?

• Simone Weil, Waiting for God, “Reflections on the Right Use of School Studies with a View to the Love of God.”

• Jacob Klein, “The Idea of Liberal Education” in The Goals of Higher Education, ed. W.D. Weatherford, Jr.

• Hans-Georg Gadamer, “The Socratic Question and Aristotle”, Continental Philosophy Review.

• Andy Clark and David Chalmers, “The Extended Mind”, Analysis. (link)

• Murray Shanahan, “Talking about Large Language Models”, Communications of the ACM. (link)

Session III: The Institutional Question

Good ideas about education are easy. Institutions that embody them are hard. This session focused on the gap. Jefferson proposes education as grafting: implanting something new onto the wild stock of human nature. Newman argues that the value of a university lies in the sheer density of its intellectual community, that students gain from living among those who represent the whole circle of knowledge even if they can never study it all.

Karlsson brings this into the present. His argument is sobering: AI tutors will be held back by culture, rather than technology. Motivated learners embedded in high-growth communities will use AI to accelerate, while everyone else will use it to avoid difficulty. The real challenge will be building strong cultural norms against taking the path of least resistance.

• Thomas Jefferson, “Draft of the Rockfish Gap Report of the University of Virginia.” (link)

• John Henry Newman, The Idea of a University, Discourse 5, “Knowledge its Own End.” (link)

• Henrik Karlsson, “AI tutors will be held back by culture.” (link)

Session IV: Education and the Machine

This session turned directly to the question of what technology does to thinking. Licklider, writing in 1960, imagined a partnership: machines handle the routine cognitive work, freeing humans for insight and decision. Papert envisaged a more profound interplay. He argued that when a child learns to program, the relationship between learner and knowledge is fundamentally transformed. The child is no longer receiving explanations but building things, and the building changes how she thinks.

Matuschak and Nielsen pick up the thread sixty years later and find it frayed. The pioneering visions of tools for thought are treated as nostalgia in technology circles. There is little determined effort to build tools that genuinely transform how people understand. The discussion focused on whether the current wave of AI represented a chance to revive the Licklider-Papert vision or whether tools for thought were likely to become an even more distant memory.

• JCR Licklider, “Man-Computer Symbiosis,” IRE Transactions on Human Factors in Electronics.

• Seymour Papert, Mindstorms: Children, Computers, and Powerful Ideas, Ch. 1.

• Andy Matuschak and Michael Nielsen, “How can we develop transformative tools for thought?” (link)

Session V: Building the New Academy

The final session focused on what comes next. Flexner’s defense of useless knowledge, written from the Institute for Advanced Study, is the purest case for protecting inquiry from the demand for application. Bloom’s portrait of Hutchins at UChicago shows what it looks like when someone actually tries to build an institution around these convictions, and the political will it requires. Simondon challenges the premise that technology and culture are opposed, arguing that the hostility between them is a sign of ignorance.

Engelbart closes the list with a constraint AI is testing: “The entire effect of an individual on the world stems essentially from what he can transmit to the world through his limited motor channels.” As those channels widen, the bottleneck shifts to whether you have something worth transmitting.

• Alexander Flexner, “The Usefulness of Useless Knowledge,” Harper’s Magazine. (link)

• Allan Bloom, “Hutchins’s Idea of a University,” Times Literary Supplement.

• Gilbert Simondon, On the Mode of Existence of Technical Objects, “Introduction.”

• Douglas C. Engelbart, “Augmenting Human Intellect: A Conceptual Framework,” SRI Summary Report AFOSR-3223.

Is there an institution that protects the conditions for deep formation and teaches students to master the tools that will define their century?

Subscribe now

Cosmos Institute is the Academy for Philosopher-Builders, with programs, grants, events, and fellowships for those building AI for human flourishing

“Artificial Intelligence will revolutionize science.” The phrase is usually invoked by researchers when discussing the promise of thinking machines to help us understand the natural world. In the early 2020s, I subscribed to, and was perhaps even evangelizing for, the role of AI in scientific research. Like many of my colleagues, I was gripped by a new tool that promised so much.

More recently my view has changed. I still believe that AI will be fantastically useful, but not necessarily in the way we think. Discovery, after all, is not the same as understanding. As scientists, we need to engage with the process of inquiry to truly make sense of what we learn about the world. Only then can we understand.

The Long History of “AI for Science”

In the summer of 1953, physicists Enrico Fermi, John Pasta, and mathematicians Stanislaw Ulam and Mary Tsingou ran the first “numerical experiment” on MANIAC, one of the early electronic computers built at the Los Alamos National Laboratory after World War II. Computers were new and scientists were excited to explore their use in solving research problems like simulating the dynamics of atoms and molecules, studying tumor cell populations and investigating numerically fluid dynamics. They even created the first documented chess-playing program that defeated a human in the game.

This group decided to model a chain of oscillators (identical masses connected by springs), add a small nonlinearity (a quadratic or cubic term), and see what would happen. The expectation was that the system would reach equilibrium (common sense would predict that there is some wiggling around, but ultimately energy ends up equally distributed throughout the springs). But the results were surprising. Instead, the system showed a recurrent behavior where the energy spreads throughout the chain, then comes back, before spreading out again. This observation sparked interest in the study of nonlinear systems that would later lead, among other things, to the discovery of solitons (waves that travel freely preserving their shape) and the development of chaos theory in the 1960s-1970s.

If one was to assign a birthyear to computational physics, that honorific should go to 1953 where Fermi and colleagues made for the first time an unexpected discovery through a purely computational approach. Computers had been used for physics and astronomy calculations before – for example, in the 1930s mechanical IBM accounting machines were modified to solve the differential equations of planetary motion by numerical integration – and had played a key role in the development of the atomic bomb during the Manhattan project. In the following decades computers would transform scientific research. Today, there are almost no advances in physics that are not enabled by some aspect of computer simulation.

Three years after the “first numerical experiment” the term artificial intelligence was coined and a first attempt was made to use automated reasoning to prove mathematical theorems through the Logic Theorist system in 1956. The first AI “expert” system, DENDRAL, was created in 1965. Combining a knowledge base with a reasoning engine, it was capable of determining the molecular structure of a compound from its mass spectra. Here, “expert” refers to a kind of AI system that encoded domain knowledge from human experts as rules and applied them through an inference engine.

Other examples include LHASA, a program designed in 1972 to discover sequences of reactions to synthesize a molecule; and the Automated Mathematician in 1977, a heuristic-based program designed to discover new mathematical concepts and theorems; and other expert systems like MYCIN (to identify bacteria) and PROSPECTOR (for mineral analysis). In the late 1980s and early 1990s, artificial neural networks were used to tackle problems in particle physics, astrophysics, and –– in a glimpse of what was to come –– predicting protein structure. By the 2000s, AI methods were in use in many areas of physics and even assisted data analysis leading to the discovery of the Higgs boson particle.

In the 2010s and 2020s, scientists began to use a wave of powerful neural network-based tools to solve research problems. Some of these advances, such as Google DeepMind’s AlphaFold protein prediction system, made headlines around the world. Others were important but less glamorous. Machine-learned interatomic potentials, for example, significantly speed up materials and chemistry simulations yet remain unknown outside the research community.

But if “AI for science” has a long history, why has it only recently started to make the news? In his 2003 book, Douglas S. Robertson coined the term “phase change” to describe a radical change that an instrument makes possible compared to the prior state of the art. AI may be a phase change in how we do science, but not only because of powerful individual tools. It is their extreme accessibility, from large language models to the AlphaFold Protein Structure Database, that sees advances in one field become instruments of exploration in others.

Epistemic Enhancers

Scientific instruments provide extensions to humans’ senses. Telescopes and microscopes allowed us to see the far and the small. This type of enhancement is known as extrapolation. Another type of epistemic support is conversion, that is, transforming one modality into another (like sound into a visual image). Finally, we have those instruments that extend our capacities by giving access to phenomena that elude our senses, such as detecting radiation or magnetic fields. This is known as augmentation. These three moves – extrapolation, conversion, and augmentation – are all types of epistemic enhancers.

A central motivation behind scientific practice is understanding. But what is scientific understanding? The 2009 book Scientific Understanding edited by Henk W. de Regt, Sabina Leonelli, Kai Eigner suggests that there is no universal definition of scientific understanding. Understanding in physics differs from understanding in biology or engineering. Even within my own discipline, physics, what we mean by understanding is not straightforward. While clearly related to the explanation of a phenomenon, understanding is not precisely the same thing. There is a difference between understanding the phenomena and understanding the theories or models that explain that phenomenon.

A prerequisite for understanding is discovery insofar as one cannot understand a phenomenon that has not been observed. Discovery is the process or product of successful scientific inquiry. Objects of discovery are things, events, processes, causes, and properties as well as theories and hypotheses and their features. The 1987 book Scientific Discovery proposed that discovery in science can be broken down into problem solving tasks and therefore can be automated with computers.

The authors of the book built four types of programs to look for quantitative or qualitative laws and structural models. Then they used them individually and combined them to rediscover many laws in physics and chemistry. While the authors recognized that there is no unique process that accounts for scientific discovery, they showed that most of these can be cast as problem solving tasks that can be tackled with heuristics. Their method, based on tree searches, was general in theory but limited in practice. This was because the combinatorial explosion of possible paths made them intractable for problems beyond a modest scale.

Both traditional numerical methods and modern AI tools enhance epistemic extrapolation and conversion. The application of such methods represents a difference in kind as well as magnitude for scientific practice. As Paul Humphreys put it: “This extrapolation of our computational abilities takes us to a region where the quantitatively different becomes the qualitatively different.” This is because these simulations cannot be carried out in practice except in regions of computational speed far beyond the capacities of humans. This dual articulation of the quantitative and the qualitative could be taken as an argument for why modern AI tools ought to usher in a new way of doing science.

In the 1960s mathematicians Bryan John Birch and Peter Swinnerton-Dyer used the EDSAC-2 computer at the University of Cambridge Computer Laboratory to run numerical calculations that allowed them to state a conjecture about the set of rational solutions to equations defining an elliptic curve. This is now known as Birch and Swinnerton-Dyer (BSD) conjecture and is one of the seven Millennium Prize Problems. The computer helped them explore an abstract space so they could formulate the conjecture. This computer-assisted discovery falls in the final category of epistemic enhancers mentioned above: augmentation.

If the formulation of the BSD conjecture is a weaker example of augmentation, more recently AI methods helped mathematicians make a breakthrough towards solving it. Using machine learning, mathematicians discovered unexpected oscillatory patterns in the parameter space key to BSD (of hundreds of dimensions) which led to the “murmuration” conjectures. Progress has also been made towards solving another Millennium Prize Problem as AI tools were used to find potential singularities in the Navier-Stokes equations.

Although automated theorem proving programs have been around for 80 years, these examples do suggest that we are witnessing, in Robertson’s terms, a phase change. The quantitatively different becomes the qualitatively different in a more interesting way. It’s speed, yes, but it’s also breadth. Take AlphaFold and the AlphaFold Protein Structure Database it made possible. What makes it revolutionary is not only the improved accuracy of the protein structure prediction, but also its usability by scientists from many fields (who can access hundreds of millions of protein structure predictions).

We are yet to see the full extent of what AI tools can do for scientific research, but one differentiator between them and traditional scientific instruments, including computer simulation, is that they combine epistemic enhancers to explore vast dimensions of information. Remember DENDRAL, the early AI system determining the molecular structure of a compound from its mass spectra? Now, imagine the power of its modern incarnation which has access to several databases of molecular spectra, can cross-check with published scientific literature, and has the ability to write code to perform additional calculations if needed.

Or consider a powerful telescope that discovers a new exoplanet. Here an AI tool analyzes the spectra, determines the likely composition of the exoplanet atmosphere, compares it with the existing recorded information, runs simulations with the known and inferred parameters to better characterize the exoplanet and adds it to the catalog. All of these individual steps are already done separately and for each the use of AI tools can be seen as an evolutionary improvement. But the combination of these improvements, working together across different sources of information is potentially revolutionary.

If I’m permitted to add a new epistemic enhancer category for AI as an instrument the most appropriate would be integration (as in synthesizing across large amounts of very different types of information). When a tool synthesizes across hundreds of heterogeneous sources simultaneously, the inferential path from evidence to conclusion can become challenging for the scientist receiving the result to trace. The more dimensions integration spans, the wider this gap between discovery and reconstruction becomes. Where earlier epistemic enhancers extended what scientists could see or calculate, integration may increasingly determine what they conclude.

Our Role as Scientists

At the end of his book, Humphreys argued that computer simulation brings a “shift of emphasis in the scientific enterprise away from humans.” By the 1980s, researchers already knew that, at least in part, scientific discovery and problem generation could be successfully automated. Modern AI methods can do the same thing, only much faster and much better. I regularly hear from colleagues that in the coming years AI will take over many of the tasks associated with scientific inquiry.

Use of AI will likely mean more discoveries overall, but to discover is not to understand. Henk de Regt argued that: “scientists achieve understanding of phenomena by basing their explanations on intelligible theories. The intelligibility of theories is related to scientists’ abilities: theories are intelligible if scientists have the skills to use those theories in fruitful ways.” Quantum mechanics is an example of a very effective, yet not particularly intelligible theory, that despite the practical successes still eludes physicists in its interpretation. If AI tools can optimize discovery and generate explanations, they could perhaps also help produce more intelligible theories.

Yet intelligibility is not a property of theories themselves. As de Regt suggests, it depends on the capacities of the scientists who use them. A theory becomes intelligible only when scientists acquire the skills to explore its implications and apply it fruitfully. Even if machines make discoveries and generate explanations, this kind of understanding still depends on the participation of human investigators. Put differently: scientists need science just as science needs scientists.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

In 1962, Victor Glushkov pitched the Soviet authorities on a nationwide cybernetics network to solve the oldest problem in socialist economics: how to allocate resources without private property and market prices. Washington was alarmed enough for the CIA to create a special taskforce.

Just as Soviet industrialization was giving way to stagnation, the first serious computers gave the planned economy a shot in the arm. The central planners believed that they’d struggled to process information fast enough, so increased computing power promised a solution.

In 1969, the Polish economist Oskar Lange described the market as “a computing device of the pre-electronic age,” suggesting we could put “simultaneous equations on an electronic computer and obtain the solution in less than half a second.” In 1971, Chile’s socialist government attempted a version of this managed via telex machine, only for it to be cut short by the coup of 1973.

Thirty years after the fall of the Soviet Union, enthusiasm for these ideas has returned. In 2016, Jack Ma predicted that “the planned economy will become increasingly big … because with access to all kinds of data, we may be able to find the invisible hand of the market.” Marxist economists have written with surprising enthusiasm about Walmart and Amazon, viewing them as technologically-enabled planned economies.

The greatest excitement has been reserved for advanced AI. Zvi Mowshowitz has argued that AI “can embody the preferences and knowledge of many or even all humans, in a way an individual human or group of humans never could.” Meanwhile, Erik Brynjolfsson and Zoë Hitzig have made the case that, by combining immense processing capacity with the ability to codify tacit knowledge through computer vision, language, and sensor data, AI could erode the traditional advantages of decentralization.

The optimists attack the case for traditional markets and decentralization from multiple directions: AI can match or exceed the information-processing advantages of markets, capture knowledge embedded in human judgment, simulate competition without running it, assess outcomes markets model badly through proxies, or simply replace the human participants whose limitations created the problem in the first place.

Despite their diversity, many of these arguments fall into the same traps. They routinely misstate the case for decentralization and flatten the distinction between different kinds of knowledge, while treating any unsolved problems as an engineering detail.

The pursuit of knowledge

The most influential case against central planning was made on epistemological grounds by Friedrich Hayek. Oskar Lange and many of his successors read him as making a simple point about transmission: the useful knowledge in any economy is spread across millions of minds, so no central authority can collect it fast enough to act on it.

This is dangerously wrong. In “The Use of Knowledge in Society,” Hayek distinguishes between two different types of knowledge. The first is scientific or theoretical knowledge, which can be stated in general rules or principles. In theory, this kind of knowledge could be effectively concentrated in a single mind or system – arguably, LLMs already do this very well.

The second type is what he calls “knowledge of the particular circumstances of time and place.” This is knowledge that is embedded in practice, judgment, and context. For example, a farmer may know that a specific field drains poorly in its southeast corner or a sales rep may notice a change in body language with a long-standing client. This kind of knowledge is derived from the experience of a specific context, rather than from theoretical training or by performing a regression analysis. The person who has this knowledge will frequently struggle to explain how they acquired it.

This concept of tacit knowledge was expressed in more detail by Michael Polanyi, a chemist turned philosopher of science. Polanyi famously formulated tacit knowledge as the idea that “we can know more than we can tell.” There are lots of things we can do that we would struggle to articulate. The theoretical account of riding a bike – adjusting angular momentum through micro-corrections in steering – bears little relationship to the knowledge that you possess and exercise when doing it.

Polanyi’s view is that tacit knowledge is not just knowledge that happens to be unstated, but instead has a distinctive architecture.

Imagine a bank manager in a meeting with a local business owner, asking for an extension to their loan. In the course of this interaction, she senses that something isn’t right.

The bank manager is picking up on a series of cues, such as the business owner’s posture, the rhythm of his speech, or the differences from their past interactions. The bank manager experiences all of this as a single act of perception, rather than a series of data points. If we tried to unpick this knowledge and asked the bank manager to list out all the data points she picked up on, it would be akin to asking a pianist to state the precise angle of each finger while she’s playing.

Of course, the planner at this point could argue that even if the bank manager can’t articulate these cues, we could train a model across video, audio, and biometrics to detect the same patterns she’s detecting. The model doesn’t need to have the same experience as her, it just needs to produce the same outputs. For example, driving was long held up as an example of inalienable tacit knowledge; Polanyi himself argued that “the skill of a driver cannot be replaced by a thorough schooling in the theory of the motorcar.” Despite this, we now have highly performant self-driving cars.

This only tells us so much. Driving is mostly about seeing things and moving your body in response. The environment is physically constrained and the action space is narrow. While complicated, it is markedly less ambiguous than navigating a dense web of human intentions and social meanings. AI excels at chess, but falters in complicated social reasoning games. The market is far more like the latter.

The distinction here goes beyond difficulty. A car navigates spatial relationships and physical dynamics, whereas what the bank manager does is categorically different: she is interpreting, drawing on a framework of meaning built up through years of situated experience that organizes her perception before any calculation begins. That framework is the structure through which the interaction becomes intelligible to her at all. More processing power has no bearing on a gap like this.

By being situated in both the conversation and the wider social context, our bank manager also has a few other advantages versus an impersonal system. For a start, she has skin in the game. If she gets this call wrong, her reputation and business could suffer; in extreme cases, her physical safety could be at risk. Secondly, unlike a system observing a bunch of cues, she is an active participant in the interaction. Her tone and method of formulating questions are all part of the interaction. Finally, she’s situated in the community. She knows both the social norms and the realities of running a business in that area.

The snake devours its tail

Eventually, we get stuck in a loop. Even if we could train an AI on these interactions, what would the training data consist of? Someone has to decide to record certain things and not others. For example, we may include the transcript of the conversation, some financial metrics, and the outcome of the loan, but not the handshake or some of the pauses between words. The data is already a selective compression of the interaction, shaped by prior human decisions about what matters.

The tacit knowledge that made those framing decisions is invisible to the system trained on their outputs. No dataset encounters raw reality. It arrives pre-shaped by decisions about what to measure and what to discard. When you tell the system to look for indicators of trustworthiness, you’ve already decided what the relevant features are, which is precisely the judgment you were hoping the system would replicate.

We make direct perceptual contact with the world in a way that AI can’t. We determine the very concepts needed to carve up the world intelligibly, invent new ones constantly, and make normative and aesthetic judgments all throughout. If the data is always post-conceptual, then every training pipeline inherits the tacit knowledge of whoever decided what to measure, which means the system can never fully escape human judgment, even in principle.

The optimist could object here. Perhaps a reinforcement learning function could create something functionally equivalent to skin in the game. A dynamic AI system interacting with clients would also develop its own tone and method of probing. Big enough systems don’t simply compute over pre-specified data – they learn from experience and may develop something akin to internal world models. They may absorb tacit knowledge wholesale without anyone specifying what to look for.

Even granting all of that, the question is whether it can replicate the model of engagement that produces our bank manager’s particular sensitivity. She must sit inside the tension between reputational risk, relationship preservation, institutional obligation, and commercial judgment without the ability to collapse them into a single metric. This is what makes her attention so acute – she can’t keep everyone happy.

This points to something general. A system that lacks our direct perceptual access to the world can detect statistical regularities within whatever framework it’s been given. It won’t recognize the moment when existing categories fail to capture what’s really happening, because its measure of “what matters” is determined by the framework. A reward function can’t evaluate its own weights. Only someone embedded in the situation – who personally feels the weight of competing, immeasurable human risks, obligations, and norms – is capable of making that judgment.

Information versus action

The economist Israel Kirzner tells the story of a mother struggling with a teething child. The mother has tried everything she can to soothe or pacify the child, but to no avail. A travelling salesman knocks on her door and offers her a colorful toy at a price of five dollars. Her child is delighted and calms down. On closer inspection, she is dismayed to realize that the toy is nothing more than a collection of marbles in a clear plastic container – something that she could have assembled in her kitchen for less than a dollar. She “could kick herself for not having done so.”

On one level, this is understandable. The mother knew she had marbles in her kitchen and had the wherewithal to put them in a container, but this knowledge “did not inspire her to action.” In Kirzner’s framing, she had information-knowledge (all the facts), but not action-knowledge (the alertness to act on the information).

Even if you build a system that can collect every production function and every consumer preference in the economy, all you have done is assemble a comprehensive stock of information-knowledge. Alertness is hard to program, because it isn’t a process of inference from known data. A successful entrepreneur acts speculatively in response to a suspected opportunity, which by definition, hasn’t already been recognized.

Hayek went further. He observed that much of the knowledge that matters for economic coordination doesn’t exist at all until the competitive process generates it.

If you are an entrepreneur who tries a new approach, whether you succeed or fail, you have created new knowledge. As a result of the risk you took, people know whether a specific combination of resources, aimed at a set of customers, at a particular price point is viable. This knowledge didn’t exist somewhere in the ether waiting to be discovered by a more powerful algorithm. Instead, it was brought into existence by speculation in conditions of genuine uncertainty. The firm that succeeds reveals the value of its approach retroactively.

The price signals that result are not the transmissions of pre-existing data, but the outputs of a distinct process. The indeterminacy runs deeper still: economic agents are not carrying fixed utility functions. Participation in exchange changes the participants – human interactions themselves generate new kinds of choices. The system a planner would need to model doesn’t hold still, because the process of market coordination is partly constitutive of the preferences and knowledge it produces.

Akio Morita, the co-founder of Sony, launched the Walkman in 1979 against the objections of the rest of the company. Morita had observed that people took large stereos to the beach and listened to music in their cars, and sensed a market opportunity. Sony’s own market research and consumer surveys consistently suggested that there was no consumer demand for a tape player that couldn’t record, no matter how portable it was.

What Morita did was not pattern recognition on a richer dataset. He changed the conceptual space, reconceiving what a music device could be and who it could be for. No amount of data about what consumers said they wanted would have produced the Walkman, because the preference for it was partly a consequence of the product’s existence. The market revealed his conjecture to have value, and it now retrospectively seems obvious. But obviousness after the fact is the signature of knowledge that could only have been created through action.

The launch of the Walkman then had a series of downstream consequences – for competitors, component manufacturers, and music sales alike. The relationship between music and everyday life changed, with enduring social and economic consequences.

If it ain’t broke

Even if many of the foundational technological challenges could be solved by a future system, we would also need to believe that it would be worth the risk. An entrepreneur who bets wrong loses his own capital. A society that dismantles its price system has made an irreversible collective wager. We would need to have confidence that this future model would outperform markets – an order that, given basic institutions, no one has to design.

For the traditional Marxist, the case is straightforward. If technology can finally solve the calculation problem, it vindicates the claim that capitalism contains the seeds of its own succession. But as we saw earlier, the AI alternative to traditional markets increasingly appeals to those who don’t hope for the final triumph of the proletariat.

Oliver Klingefjord and Joe Edelman from the Meaning Alignment Institute argue that advanced AI systems could correct a number of shortcomings in current markets. They believe that markets systematically contract on proxy metrics like hours, subscriptions, engagement, rather than outcomes actually delivered. This is partly because doing so would be prohibitively expensive across millions of consumers, but also because there is an asymmetry of power between suppliers and consumers. Big suppliers can write “take it or leave it” contracts and have huge information advantages versus those that they are contracting with.

Klingefjord and Edelman argue that replacing markets with AI could collapse these measurement and bargaining costs, making it feasible to pay suppliers for delivered benefit via competitive, voluntary AI intermediaries that pool consumers, assess outcomes qualitatively, and negotiate enterprise-level deals.

This is much more sophisticated than technosocialism, but runs into some of the same problems.

This approach tries to maintain a price system, but changes what the prices track. But this entire system relies on an intermediary being able to assess whether a good outcome was delivered. Unlike market prices, which emerge from entrepreneurial bids under genuine uncertainty and personal risks, these assessed prices would reflect a system’s operationalized definition of human benefit. While the user might set the guardrails, the system has to turn these into assessable criteria. In essence, it’s a system’s approximation of a person’s approximation of what constitutes human flourishing.

This would also result in an enormous amount of discretionary judgment embedded in an infrastructure layer that most people would never inspect. While you could try to mitigate this by having a world of competing AI intermediaries, it’s hard to see users choosing between rival theories of their own good – as operationalized by AI systems that they can’t design – as an obvious improvement on choosing between rival products in a traditional market.

Magical thinking

It is, of course, possible to argue that these objections could all be overcome. Maybe we will build systems that can collect all dispersed and local knowledge, model genuine alertness, simulate exchange, and anticipate the outputs of the price discovery process without running it. It may then lead to more efficient resource allocation. But we haven’t built these systems, and nothing in the current trajectory suggests we are close. The burden of proof lies with those who believe otherwise.

The case for planning, by necessity, assumes away all real-world constraints while simultaneously reversing the burden of proof. In response to arguments about the importance of markets, it hypothesizes a system that by stipulation overcomes any individual objection and then challenges opponents to prove that it’s impossible.

Thought experiments that grant one or two premises can be genuinely useful when thinking about advanced technology, but with each additional hypothetical, the value diminishes. Any theory of how the world could work becomes plausible once you assume the existence of an omniscient machine god.

In the end, the CIA didn’t need to worry about Soviet cybernetics. Glushkov’s proposed cybernetics system, budgeted to cost the equivalent of over one trillion dollars in today’s money, never saw the light of day. The Soviet authorities, unconvinced by his argument that the system would pay for itself several times over, balked at the price tag. Cybernetics would not get to play its role in the inevitable triumph of the proletariat.

It seemed that even the Soviets had lost faith in the planned economy. On this, if little else, they were right, even if it was not for reasons that they fully understood. The fundamental obstacle was never processing power or data collection. It was that the economy a planner would need to model is constitutively shaped by the expectations and interpretive frameworks of the people who participate in it. Those frameworks shift in response to the very act of observation and intervention. There is no fixed economy waiting to be measured.

The system that a planner would need to model is the same system that the plan would destroy.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

This question was on our minds as we gathered 120 thinkers and builders from frontier AI labs, top universities, and cutting-edge institutions for the first Cosmos Symposium. Among them were the creator of Fortnite, category theorists, the principal of Alpha School, researchers from OpenAI and DeepMind, and first-time founders building for human autonomy

But as Brendan said in his opening remarks, no one in the room, or indeed the world, is yet a philosopher-builder.

Over the past two and a half millennia, there have only been a handful of true philosophers who pursued fundamental questions with courage and relentlessness. The number of people who have then applied this energy to institution-building that enhances freedom and inquiry for the individual is even smaller; the best example is Benjamin Franklin.

Nevertheless, there was no shortage of ambition. During the event, attendees shared what they were building on a whiteboard. The responses varied significantly, ranging across human goods like strengthened community, augmented intelligence, and a more enriched life of the mind. Underneath this difference in emphasis was a shared belief that we are building towards a greater goal. Everyone in the room saw more capable technology as a means, not an end.

Before the event, everyone nominated a book that shaped their view of the world. Many chose the classics – Plato’s Republic, Smith’s Wealth of Nations – but we also had science fiction, writings from theologian John Henry Newman, and a history of American nuclear power.

To our surprise, one of the most nominated books by attendees was Martin Heidegger’s famously challenging Being and Time. In Being and Time, Heidegger rejected René Descartes’s traditional account of consciousness, which depicts a detached mind peering out at an external world. Instead, Heidegger insisted that we are already caught up in the world, acting on it before we ever step back to think about it. At a time when it can feel like the development of technology is following a fixed path and the best we can do is act as users or spectators, he may have something to teach us.

As only 120 of us could be in the room, we’re sharing a few of our main takeaways from the event.

1. The importance of ‘becoming’

Shortly before Adam Smith died, he ordered that 16 volumes of his manuscripts be burned. He was so anxious that only his best thinking survive that he destroyed the record of his own becoming: the drafts, notes, abandoned arguments we might have learned from

We tried to do the reverse of this at the Symposium by bringing people together at different stages of their journey. This included:

• Highly accomplished builders, who are pursuing mission-driven projects and can act as inspiration, such as Joe Liemandt, the principal of Alpha School or Paul Meegan, the creator of Fortnite and founder of a new stealth venture;

• Former researchers and engineers at leading tech companies and frontier labs, now building for human goods, like , a 15-year Google and Stripe veteran and , a former Anthropic researcher and ex-head of collective intelligence at Midjourney;

• First-time founders who are building for human autonomy, such as Workshop Labs co-founder ;

• World-class intellectuals who are helping to lay the crucial foundations for future builders and thinkers, such as Philipp Koralus at the Oxford HAI Lab, at the Mercatus Center, UT Austin’s Harvey Lederman, Catherine Project founder Zena Hitz, or , founder of the Machine Intelligence and Normative Theory Lab.

All of these groups had something to learn from each other, whether it was hard-won knowledge about what it actually takes to ship something, fresh insights from the frontier of AI research, or clarity about what’s worth building and why.

2. Building the philosophy to code pipeline

Philosophy can feel remote from the realities of shipping a product. It can be easy to default to obvious ideas, like introducing greater personalization through a system prompt. Attendees explored options that show why steerability is essential but insufficient.

These included tools that surface the assumptions a model makes about us, and ways of aligning models with our second order preferences: not just what we want, but what we want to want.

Some of these projects are still in their early stages, so we can’t share too much detail at the moment, but we were grateful to be joined by members of the Cosmos community who are entering institution-building mode. These included Samuele Marro of the Institute of Decentralized AI, Elian McCarron of Kanonic, and Jasmine Li, who is starting an organization focused on measuring and forecasting human agency as AI capabilities advance.

3. Epistemic humility

As one of our speakers observed, autonomy requires humility. The philosopher-builder will need to constantly ask whether they are doing the right thing and course-correct when the answer is no.

from the Mercatus Center recently wrote an essay on how the same evidence in frontier AI debates can justify sharply conflicting hypotheses. Despite all of this talent in one room, none of us left Austin feeling certain.

The discussions raised vital questions that lack a definite answers:

• How can we build for second order preferences without defaulting to paternalism?

• How can we prioritize human autonomy, while using capital markets to reach hundreds of millions of people?

• What are the limits of decentralization? What high-level coordination or condition-setting is needed to create useful, coherent action in the real world?

Some debates, especially around frontier AI, can feel tribal, which is why it’s important for communities to build strong norms. At the symposium, we saw people treat conversations as conversations. Each interaction felt like an opportunity for two people to learn, and we received no reports of distillation attacks.

4. Structure and spontaneity

Nietzsche thought Greek cultural greatness stemmed from the fusion of the Apollonian and the Dionysian. Apollo, the god of light, represented clarity, structure, and rational boundary-making. Meanwhile, Dionysus, the god of wine-making, represented the dissolution of those boundaries.

In our own modest way, we attempted to replicate this formula. Alongside our structured programming, attendees had time to mingle over drinks, dinner, and across the sprawling, beautiful grounds of the venue. The best conversations were often the spontaneous ones, still running late on Sunday morning, long after the formal programming ended.

This was the first Cosmos Symposium, but it won’t be the last. The Cosmos community is just over a year old and we’re grateful to everyone who has contributed to what we’re building.

If you’re interested in attending a future Cosmos event, register your interest on our seminars and events form.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

If you’re someone who thinks deeply, builds deliberately, and cares about the future AI is shaping—join the Cosmos network.

Subscribe now

John Graunt was a merchant who lived in 17th century London. In 1662, in a city familiar with plague, Graunt consulted parish death records to figure out how long Londoners could expect to live. He could tell you that roughly 36 out of every 100 children born in the city would die before their sixth birthday. This probability could be checked by anyone willing to do the same sums using the same sources.

Three and a half centuries later, AI researchers are calculating a morbid probability of their own. They wonder about thinking machines seeing off the human race, and refer to the likelihood of that possibility as P(doom). The term began life as an inside joke in rationalist spaces in the closing years of the 2000s, but shot to prominence after researchers signed an open letter about the threat posed by AI development in 2023. In the AI world, everyone has their number. Yudkowsky puts it at roughly 95–100 percent. Geoffrey Hinton sits around 10–20 percent, while Marc Andreessen reports a bullish 0 percent (a logical impossibility because you cannot rule out something that has never happened and whose conditions have never been tested). The mean estimate among surveyed AI researchers is about 14 per cent.

We have some who believe that AI is almost certain to wipe out humanity and others who ascribe a 0 per cent probability to the same outcome. This is curious for a supposedly hard-headed prediction made by some of the world’s most credentialed scientists. Imagine a roomful of cardiologists who, given the same scans and the same patient history, exposed to the same training and the same clinical standards, disagreed on whether the probability of heart failure was 0.1 percent or 99 percent.

You would not conclude that this was a hard problem on which reasonable people differed. In fact, you would probably conclude that at least some of our medics may need to visit neurology down the hall. With P(doom), the estimates from AI researchers range from 0 percent to 99 percent (and a few nines then some). When someone tells you they assign a 15 per cent probability to AI wiping out humanity, it is worth asking what kind of claim they are actually making. It sounds like a statement about the world. It is not.

To be clear: my objection is not to probabilistic reasoning, which is straightforwardly indispensable in domains where estimates can be calibrated against outcomes over time. My problem concerns what we might call proximate and ultimate formulations. Proximate estimates for COVID were things like the hospitalization rate and case severity, while an ultimate estimate was the infection fatality rate. For P(doom), proximate estimates include the conditions under which models exhibit deceptive behavior and the observed failure rates of current alignment techniques. Here, the ultimate estimate is the thing itself.

Someone might know that current models can be made to exhibit deceptive alignment under laboratory conditions or that existing alignment techniques have specific failure modes. But to cram those findings into a container labeled “15% chance of extinction” involves a series of judgments about unprecedented transitions for which there is no model connecting proximate estimates to their ultimate counterpart. Knowing that fine-tuning on specific tasks can induce broader misalignment tells us little about how well alignment solutions will generalize or how geopolitical actors will respond. Compare with COVID, where hospitalization and case-severity data fed epidemiological models that could be tested against actual deaths.

A natural objection is that some people are good at formulating predictions in lots of different domains, so perhaps their assessment of P(doom) ought to carry weight. After all, some superforecasters assign 17 per cent to various events and see those events happen roughly 17 per cent of the time. You can check a forecaster’s guesses across many different predictions in such a way as to mean we’re dealing with a property of the forecaster’s overall policy rather than of any individual estimate. When a superforecaster says 17 per cent for a ceasefire in some conflict, that number is useful because they also say 17 per cent for hundreds of other things and roughly 17 per cent of those things happen.

Superforecasters earn their calibration (that is, the extent to which confidence lines up with their track record over time) using predictions with short time horizons, a history of similar events to draw on, and a mechanism that allows the forecaster to correct course when needed. For P(doom), there is no source of comparable predictions against which to calibrate. A forecaster might be superbly calibrated on elections and geopolitical high drama, but that tells us nothing about whether their number for an unprecedented event is any good.

What I Talk About When I Talk About P(doom)

There are two influential interpretations of probability: subjective and objective. Where the former deals with how confident you are in a claim, the latter stresses what actually happens when you test it. Anyone who has spent even a little time thinking about the reliability of P(doom) metrics probably has a sense that the calculations ought to be taken with a pinch of salt. Many in rationalist or rationalist-adjacent circles are well aware of the difference between these two accounts, but lots of people who absorb these views are not.

That goes for those who adopt the terminology to signal their membership of the in-group, and those who hear talk of P(doom) on the news and wonder why one of the world’s most cited researchers thinks there’s a one in five chance that AI will wipe out the human race. When a chief global strategist at one of the largest investment research firms tells CNBC it’s 50/50 whether AI destroys humanity by mid-century, the average viewer — incorrectly but understandably — assumes that figure estimates the likelihood that it will happen.

Every single P(doom) you hear is a subjective probability. It is a measure of the degree of rational belief one holds in a proposition given the available evidence — a property of an agent’s epistemic state rather than of the world. When you say “I think there’s a 50 per cent chance this meeting will be a waste of time,” you’re not drawing on a frequency table of past meetings. You’re expressing how confident you feel given what you know about the agenda and who’s attending. All the framework requires is that your degrees of belief are consistent with one another. Beyond this internal consistency, there is no further requirement that your probabilities correspond to the shape of reality.

If your beliefs are internally coherent, you can multiply your probability for any outcome by the value of that outcome — however you choose to measure it — and arrive at its expected value, a single number that tells you how good or bad a bet looks on average. From there you can derive a utility function and compare any set of actions on a common scale. The framework of cause prioritization, wherein resources are allocated to whichever cause produces the greatest expected good per dollar, inherits this apparatus. This approach is popular with rationalist-adjacent communities, especially Effective Altruism, because it transforms questions of moral life into properties that can be measured and optimized.

Objective accounts of probability take a different approach. Here probability is what actually happens in the world when you repeat something many times. This species of probability is a physical property of systems, one that is empirically testable. If you flip a coin a thousand times roughly half will land heads, so we can say the probability is that ratio. The same logic applies to complex phenomena, like radioactive decay, where about half the atoms in a given sample will decay within the half-life period (and you can go and check).

The subjective interpretation is internally coherent and mathematically elegant. But coherence should not be confused for empirical content. My objection to P(doom) flows from the simple fact that subjective probability claims are, by their nature, unfalsifiable. That’s generally fine in many instances — like casual everyday judgments or betting on the Super Bowl — but not when the stakes are high enough to redirect billions of dollars and make or break government policy. Statistical reasoning about unlikely but potentially devastating scenarios is obviously useful, but only when properly grounded in claims we can falsify.

Your Problem Too

For a probability to tell you about something other than the speaker’s state of mind, it needs a collection of similar events from which a ratio can be drawn. To say that a coin lands heads 50 percent of the time, we need a set of coin flips to draw from. This is our reference class.

Clearly there is no reference class for human extinction. As far as every AI researcher in the world knows, the event is singular and unprecedented. Traditionally, the reference class problem has been associated with the objective interpretation of probability. This is because objective probability is calculated according to the ratio of how many times something happened out of how many times it might have happened. If the event has never happened and can’t be repeated, then there’s nothing to compute.

But the reference class problem isn’t only relevant for objective accounts of probability. Alan Hájek famously made the case that it affects every interpretation of probability when applied to singular events. As he put it, “the reference class problem is your problem too.” The argument goes something like this: before you can assign a probability to a one-off event, you have to decide what kind of event it is. Is AI-caused extinction a case of “new technology going wrong,” “species-level catastrophe,” or “unprecedented transition in the nature of intelligence”? Each framing suggests a different probability, and there is no principled way to choose between them. The number you arrive at depends on how you describe your target.

The way to tell a useful probability from a useless one is to ask whether the world can correct it.

This is because good explanations resist variation. If Graunt had doubled his estimate of childhood mortality, the parish registers would have corrected him. If an epidemiologist in early 2020 had put the infection fatality rate at 10 per cent rather than 1 per cent, the incoming hospital data would have shown the estimate to be wrong. The explanation is enmeshed with the world in such a way that it cannot be led to wherever you would like it to go. With P(doom), you can swap the causal pathway from deceptive alignment to resource competition and inflate the number from 5 per cent to 7.5 per cent. The estimate can contort to accommodate any figure and any causal chain because the only constraint it faces is internal coherence.

In the context of P(doom), defenders might say something like “We don’t need a reference class because we’re expressing a degree of personal belief.” This is perfectly coherent on the subjective view, but it is also the move that severs the claim from its empirical content. If your probability doesn’t need to correspond to any feature of the world, then it can’t be wrong about the world. And a claim that can’t be wrong about the world tells you nothing about it.

We might call the result a Gettier probability, a credence that is internally justified — and that might even correspond to reality — but that lines up with the world through luck rather than judgement. In epistemology, a Gettier case is a belief that is true and justified but not true because of its justification. Any P(doom), where the justification and the truth are aligned, will (assuming the estimate was correct) have the same structure. A researcher’s estimate might happen to match whatever the actual risk turns out to be, but the match would be accidental relative to the method.

Gettier probabilities show up everywhere, like when a consulting firm estimates that a given market will grow at 7.3 per cent and lo and behold they are correct. The same is true for a geopolitical risk score that looks precise but cannot be updated by any observation short of the catastrophe it purports to predict. P(doom) is distinctive only in that the stakes are high enough that the probability escapes containment and ends up on the morning news.

Suppose someone’s P(doom) is 30 percent. What outcome would show this was wrong? If AI goes well, they get to enjoy the benefits of the 70 percent figure. If it goes catastrophically — and anyone is still alive to update their priors — the 30 percent gets paid off. This is one of the more frustrating features of the subjective framework, which allows pretty much anyone with an idea about AI risk to have their tokens and eat them.

Bayesian probability demands that credences be precise. When estimates range from 0% to 99% with no mechanism to adjudicate, the framework is not being applied so much as invoked. P(doom) manages the impressive trick of satisfying neither the Bayesian standard (precise credences that can be adjudicated between) nor the falsifiability standard (empirical claims that can be proved wrong).

I’m not arguing that we ought to stop reasoning about novel events. Subjective probability is useful because it helps us think under uncertainty without good data. Some of the best calls in recent AI history were made by researchers who formed strong priors about scaling laws and bet on them before the evidence was in. Those bets were vindicated because they were testable. Scaling laws either held or they didn’t; capability thresholds either arrived on schedule or did not. My concern is that P(doom) borrows the confidence of claims like these without sharing their vulnerability to refutation.

I Am Sure You Are Very Sure

Forget, for a moment, whether P(doom) can be tested. Consider instead whether it is the kind of thing that can be meaningfully quantified in the first place. In 1921, the economist Frank Knight drew a distinction between risk and uncertainty. Risk is what you face when you can calculate the odds as with flicking a small ball onto a roulette wheel. Uncertainty is what you face when you can’t, even in principle, because the situation is too poorly understood to yield a number.

Knight argued that the two are different in kind.

The Bayesian tradition’s response to Knight was to argue that subjective probability does away with the distinction: if you can always express a degree of belief, then there is no situation that resists quantification. P(doom) is a textbook application of this move, but the dissolution works only if the resulting number can be tested via real-world feedback. For P(doom), the Bayesian move converts Knightian uncertainty into a figure that has the form of risk without any of the mechanisms that make risk estimates trustworthy.

The same process, where uncertainty becomes risk, happens wherever a decision-making framework demands a numerical input and none is available. Firms routinely assign precise probabilities to market scenarios that are genuinely unprecedented. They don’t have any evidence that warrants a specific number, but they do have a spreadsheet with cells that need filling. The defense that such figures are “rough heuristics” does not help. The problem is that they are treated simultaneously as a casual shorthand and as inputs into value calculations depending on whatever is most convenient. This is why the “shmrobability” move doesn’t fix our problem. It concedes we’re not really talking about probability, while trying to preserve its practical content. Either the number is a rough heuristic and we should stop plugging it into expected value calculations, or it is a serious input to policy and we should hold it to the standards that implies.

If P(doom) is a subjective credence, there is no principled way to adjudicate between two people who disagree. A P(doom) of 95 per cent and one of 50 per cent are both internally coherent insofar as neither violates any rule of the subjective framework. You can be confident in your P(doom) and I can be confident in mine, and there is no observation or outcome that could settle which of us is right. Attempts to resolve this from within the Bayesian model, from Aumann’s agreement theorem onwards, all founder on conditions that do not hold (like shared starting points, perfect rationality, or common knowledge of each other’s beliefs).

The problem compounds when these estimates enter expected value calculations, where even a vanishingly small probability multiplied by the stakes of human extinction produces numbers large enough to dominate decisions. The rationalist and rationalist-adjacent communities recognize this dynamic as Pascal’s mugging, while groups like GiveWell have cautioned against taking such estimates at face value.

The reader should not take from this essay that AI risk is unworthy of attention. Rather, my point is that P(doom) is the wrong epistemic instrument for thinking about it. The alternative is to formulate falsifiable conjectures about the world and then try to refute them, which is exactly what AI safety researchers do in their actual work. They propose that particular training regimes will produce deceptive behavior and they predict that models beyond a certain capability threshold will resist correction. They hypothesize that misalignment in one domain will generalize unpredictably to others, and they can test this by measuring the transfer of learned strategies across task distributions. They ask whether models that pass safety evaluations in deployment conditions will behave differently when those conditions change. Each of these lines of research produces claims that can fail, and this vulnerability is what makes them worth paying attention to.

Individual claims about deceptive behavior or capability thresholds do not by themselves tell you how much to spend on alignment versus pandemic preparedness, yet real institutions have treated P(doom) as though it could. The effective altruist community has directed hundreds of millions of dollars toward AI existential risk reduction. Open Philanthropy, the field’s largest funder, used subjective probability frameworks to weigh AI risk against global health, biosecurity, and farm animal welfare. Whether or not that assessment was correct, the reasoning behind it is only as good as the numbers it relies on.

A figure that ranges from 0 per cent to 99 per cent depending on who you ask and with no way to adjudicate between them is about as useful as no number at all. Aggregating these estimates might seem to help inasmuch as a mean of 14 per cent looks more sober than bouncing between extremes. Aggregate or individual, the problem is that neither incur a cost for being wrong. A bookmaker’s odds are constrained by the market just as an insurer’s premiums are shaped by the frequency and nature of claims made. In both cases, the issuer’s assessment improves over time because they suffer when it is incorrect. There is no market to punish a mispriced P(doom) and no settlement date on which the estimate is tested.

We might say something like: “Even if P(doom) is imprecise, it’s still better than no number because we need to allocate resources somehow.” But here we’re assuming that internal coherence still outperforms the absence of a number when you have to act. We can live with imprecision under these conditions, but not at the expense of claims that cannot be falsified that get steamrolled by the headline rating.

Governments fund pandemic preparedness efforts without a big round number for the probability of the next pandemic. Institutions allocate resources under genuine uncertainty all the time by doing things like funding a portfolio of approaches, setting thresholds for observable harms, identifying the cheapest reversible interventions, demanding observable milestones before scaling commitment, and building the capacity to pivot as new evidence arrives.

The physicist Wolfgang Pauli was famously unforgiving of bad theory. When a colleague asked him to assess a young physicist’s paper, his verdict was that it was “not even wrong.” He meant that its central claim was so removed from evidence that it could not be proved false. P(doom) is not even wrong. Graunt’s mortality tables may have been technically crude, but they were accountable. They could be checked against next year’s parish records, and if they were wrong, the records would show it.

Imagine a system that knows you eat sensibly all day and fall apart at 10pm. That knows a calorie label won’t help you, because the evidence says calorie labels mostly change the behavior of people who already eat well.

This system knows the difference between your hunger and your boredom. It leaves you alone when you don’t need it. It preserves every option on the menu. It just rearranges the choice so you’re more likely to do what you’d do if you were thinking clearly.

Now extend this to every significant decision you make. Cars, insurance, investments, medical treatment, career moves. A system that corrects for your specific biases, knows what you’ll regret, and steers you toward what it calculates you really want. The full menu is technically available, but the system is right often enough that overriding it starts to feel like pride, and convenient enough that you stop wanting to.

Cass Sunstein—co-author of Nudge, former White House regulatory czar, the most influential regulatory thinker of his generation—calls this a “Choice Engine.” His arguments tend to become law. I think the Choice Engine’s greatest danger is that it works.

His new paper “Liberal AI,” prepared for a lecture at Oxford this May, makes the strongest case yet that AI-powered choice architecture can respect autonomy while improving welfare. The logic runs like this. Classical liberalism, following Mill and Hayek, grounds freedom of choice in an epistemic claim: the chooser knows best. AI disrupts that claim by knowing better. Better than the individual about her own informational gaps, her biases, what she’ll want next year. So Choice Engines can personalize the nudge, correct the bias, and never technically remove the freedom to choose otherwise. Liberal AI is no oxymoron.

Sunstein flags the risks. He talks about manipulation, self-interested designers, AI’s own biases. He calls for regulation. He distinguishes liberal Choice Engines from illiberal ones. But even the liberal version is steering people toward choices they wouldn’t have made on their own. Sunstein knows this is paternalism. He thinks preserving opt-out keeps it libertarian rather than something worse. I argue that it cannot.

Mill’s Music

Sunstein’s Mill is an information theorist. In On Liberty, Mill argues that each person is “the person most interested in his own well-being,” possessing “means of knowledge immeasurably surpassing those that can be possessed by anyone else.” When outsiders intervene, they rely on “general presumptions” that are likely wrong or misapplied.

On this reading, the case for freedom of choice is instrumental: people should choose for themselves because they know their own circumstances best. If the case for freedom rests on the chooser’s informational advantage, and AI erases that advantage, then the case for freedom weakens. AI knows your medical history better than you remember it, your spending patterns better than you track them, your recurring mistakes better than you admit them. The epistemic ground has shifted. So far, so honest.

But Sunstein himself notices something that should give him pause. He observes that “the words of On Liberty are epistemic, but the music is romantic.” Then he proceeds to work exclusively with the words.

The words say: people should choose freely because they know best. The music says something else entirely. It says that the process of choosing is what makes a person a person. Mill:

The human faculties of perception, judgment, discriminative feeling, mental activity, and even moral preference, are exercised only in making a choice. He who does anything because it is the custom, makes no choice. He gains no practice either in discerning or in desiring what is best. The mental and the moral, like the muscular powers, are improved only by being used.

This is an argument about formation. Mill is not saying that choosers make better decisions (the information argument). He is saying that choosing makes better choosers—people with developed faculties of perception, judgment, and what he calls “discriminative feeling,” the ability to tell the difference between what matters and what doesn’t. These faculties, like muscles, grow through exercise and atrophy through disuse. The quality of any particular decision matters less than what the process of deciding does to the person who decides.

Once you hear the music, Sunstein’s proposal sounds different.

Consider a person who consults a well-designed Choice Engine for a car purchase and accepts its recommendation. She may end up with a better car than she would have chosen on her own. Sunstein counts this as a welfare gain.

But if the evaluative work was done by the Engine, if she did not weigh the tradeoffs, interrogate her own priorities, or struggle with the uncertainty of not knowing what she really wants, then her faculties of judgment were not exercised. She received an outcome without undergoing the process that develops the capacity to arrive at such outcomes independently.

Do this once and the effect is trivial. Do this for every significant choice across a decade, and you have a person whose formal freedom of choice is perfectly intact, whose welfare as measured at each decision point is maximized, and whose capacity for independent evaluation has atrophied through sustained disuse. She can always opt out, but she no longer has the developed faculties that would make opting out a meaningful exercise of judgment.

Mill has a name for this person. “One whose desires and impulses are not his own, has no character, no more than a steam-engine has a character.”

She may be doing well by every metric Sunstein tracks. But the evaluative standards by which “well” is determined have migrated from her judgment to the algorithm. She is content without being, in any sense Mill would recognize, free.

Half of Hayek

Hayek’s case for markets rests on a claim about knowledge: both that markets aggregate it efficiently, and that participation in institutions like markets is what produces it.

Sunstein takes the first half and drops the second. He reads Hayek as making an argument about information processing, where markets aggregate dispersed knowledge more effectively than central planners. If AI can aggregate that knowledge even more effectively, the Hayekian argument for leaving people alone weakens.

But the knowledge Hayek cares about is not the kind of thing that can be aggregated. The knowledge of “the particular circumstances of time and place” is not sitting in a warehouse waiting to be collected. It comes into existence through the activity of people navigating uncertainty, bearing consequences, and adjusting. This is not a computational problem.

The structure in which choosing happens is itself doing something irreplaceable. Remove the individual from the process and you do not just lose one person’s contribution. You lose the conditions under which that person would have developed the knowledge worth contributing. The loss compounds.

AI would have to live your life to fully replace your epistemic role in it. But partial replacement may be enough to leave you unable to perform it yourself.

When Hayek writes that coercion “eliminates an individual as a thinking and valuing person and makes him a bare tool in the achievement of the ends of another,” Sunstein reads this as an argument against coercion, which Choice Engines avoid because they preserve opt-out. But read the sentence again. The operative phrase is not “coercion.” It is “eliminates an individual as a thinking and valuing person.” That elimination can happen without coercion. It can happen through substitution, through a system that does the thinking and valuing for you so effectively that your own faculties are never called upon.

Hayek’s nightmare was central planning, but his deepest fear was not the planner’s malice. It was the replacement of a distributed process of discovery with a system that delivers answers without requiring anyone to find them. Choice Engines are that system. No one is coerced, and no one means harm. And yet it is Hayek’s nightmare, built by people who thought they were honoring him.

The Missing Axis

Sunstein’s paper operates on a single evaluative dimension. Every intervention is assessed by two criteria: does it improve welfare, and does it preserve freedom of choice? Both are real goods, but they sit on the same axis. Call it agency: the capacity to get what you want, measured by outcomes and options.

The axis Sunstein does not have is autonomy. Not autonomy in the loose sense of “having choices,” which is just agency again. Autonomy in the strict sense: self-rule. Authorship of the evaluative criteria by which options are assessed. A person has agency — agere, to act — when she can get what she wants. She has autonomy — auto nomos, self-rule — when she determines what to want, through her own exercised faculties of judgment in Mill’s terms, or through her own participation in the knowledge-generating processes Hayek describes.

These come apart. AI can increase agency while eroding autonomy. A Choice Engine that tells you what is best for you may be right. It may get you better outcomes. But if the standard by which “best” is determined has migrated from your judgment to the Engine, your autonomy has decreased. You are more powerful, but you are less free.

Sunstein’s paper treats both rows as one. He assumes that improving agency while preserving the option set is sufficient to protect autonomy.

His Choice Engines produce movement downward. Into the lower left. Claude Boys are powerful without being self-governing, formally free at every point, and yet not authoring the evaluative criteria by which a life is organized. Silicon Valley’s “you can just do stuff” brigade lives here, though it doesn’t know it.

This quadrant is easy to miss because it doesn’t look like a problem. It looks like liberal AI working exactly as designed. And all the while, the capacity that makes opting out meaningful is eroded, because it is no longer required.

Constitutional Drift

The phrase that does all the work in Sunstein’s paper is a counterfactual. Choice Engines steer people toward what they would choose if they were “adequately informed and free from behavioral biases.” The Engine corrects for departures from this idealized chooser.

But who defines the ideal? Whether something counts as a bias or a preference, adequate information or noise, a self-control failure or a legitimate present-oriented value: the answer, necessarily, is the Choice Engine. Which means the Engine is authoring the normative standard against which the person’s actual choices are evaluated. The person experiences this as help. She believes she is getting closer to what she really wants. But “what she really wants” is a construction of the system, not an independent fact about the person.

This is Constitutional drift, the migration of priority-setting from the person to the system. It is experienced from inside as self-improvement. The person feels she is becoming more rational. She is becoming more aligned with the Engine’s model of rationality. These feel identical from the inside, but may not be the same thing.

And the drift compounds. At time one, the person’s preferences are her own. She consults the Engine, accepts its recommendation, and her welfare improves. At time two, her preferences have been partially shaped by her previous Engine-assisted choices. By time ten, the convergence is nearly complete. The Engine recommends what she wants and she wants what the Engine recommends. The idealized chooser, “adequately informed and free from behavioral biases,” has become the Engine’s model of her rather than anything she authored herself.

This convergence is a ratchet. The more the Engine outperforms her independent judgment, the more rational it is to defer. The more she defers, the less her capacity is exercised, and the more the Engine outperforms her. At every point in this cycle, Sunstein’s framework reports success. It cannot represent the difference between a person whose welfare is high because she exercises excellent judgment and a person whose welfare is high because an Engine exercises excellent judgment on her behalf. They are in opposite positions with respect to their own freedom.

There is early evidence this is already happening. Bastani et al. (2025) found that high-school students using ChatGPT scored 48% higher on practice assignments and 17% lower on exams taken without it. The work improved but the learning did not. Becker et al. (2025) found that experienced developers using AI coding assistants were 19% slower on tasks while believing they were 24% faster, a 43-percentage-point gap between perceived and actual performance. Fernandes et al. (2025) reported that higher AI literacy correlated with worse metacognitive accuracy about one’s own AI-assisted performance. The people most embedded in AI-assisted work were most wrong about what the assistance was doing to their independent capacity.

The habit of deferring to a well-designed Engine is the same habit a manipulative one exploits. The person whose evaluative capacity has diminished through years of benign delegation is the person least equipped to detect when an Engine shifts from serving her interests to exploiting them. Sunstein’s liberal AI creates the prey that his illiberal AI hunts.

The effect will not be evenly distributed. People with strong habits of deliberation will use Choice Engines as tools, retaining evaluative authority while extracting information. People without that formation will not supplement their judgment but cede it. These populations map onto existing inequalities in parenting and education. The structural effect of Choice Engines may be to concentrate the capacity for self-rule in those who already have it. This is the deepest inequality, and the least measurable.

The Tutelary Power

Sunstein frames his paper with Mill and Hayek. The thinker he needs, and does not cite, is Tocqueville. In the final pages of Democracy in America, Tocqueville describes a tutelary power that

takes charge of assuring their enjoyments and watching over their fate. It is absolute, detailed, regular, far-seeing, and mild. It would resemble paternal power if, like that, it had for its object to prepare men for manhood; but on the contrary, it seeks only to keep them fixed irrevocably in childhood… It does not destroy, it prevents things from being born.

The Engine does not coerce and it does not break wills. It softens, bends, and directs. It seeks not to prepare persons for independent judgment but to keep them, gently and perpetually, in a state of epistemic childhood. The capacities at risk here are not taken. They go unexercised until they are gone.

Every argument in “Liberal AI” presupposes an author with intact evaluative capacity. Sunstein weighs Mill against Hayek, assesses the evidence on nudges, evaluates the risks of manipulation, reasons about what would constitute genuinely liberal AI. He is arguing from inside a house his proposal would slowly empty. But the capacities that make his paper important are the capacities that a generation raised on Choice Engines may never develop, because the Engines are so helpful, so liberal, and so respectful of freedom that the difficult and sometimes painful process of developing independent judgment is no longer necessary.

Mill understood this. The worst threat to liberty is not the tyrant who forbids you from thinking. It is the benevolent power that makes thinking unnecessary.

Sunstein’s Liberal AI is not an oxymoron. It is something more dangerous: a coherent program that achieves everything it promises while producing persons who can no longer do for themselves what it does for them. A generation raised on Choice Engines will never develop the evaluative capacity that was, in all prior human experience, the unavoidable byproduct of having to choose. We are building the conditions for that generation now.

The paper asks whether liberal AI can make life more free. It cannot ask what kind of persons will be left to be free, or whether freedom, for such persons, will mean anything at all.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

In 1972, researchers offered children from the Bing Nursery School at Stanford a simple choice. Left alone with a single marshmallow, they were told they could eat it now or wait fifteen minutes and receive two instead. Years later, the scientists tracked down the original participants to find that children who had waited longer at age four seemed healthier, more sociable, and even higher performing at school.

Gleefully amplified by journalists and pop psychology gurus, a popular interpretation held that the children already had certain traits that predicted how their lives might unfold. Regrettably for the original group, the results – as is often the case in the social sciences – didn’t stick. In 2018, a new team of researchers conducted another study with a similar task and long term follow-ups. Drawing on a much larger dataset and controlling for confounding variables like parental education or household income, they found only a mild correlation between waiting time and positive outcomes.

What gives? The answer turns on the extent to which you think the self is made or given. The original interpretation assumed self-control was innate rather than learned, but the replication seemed to show that a child’s capacity to wait depended on background conditions (and so could not be wholly inborn after all). A child from a chaotic household, where promises are routinely broken, has good reason not to trust that the second marshmallow will materialize. Their failure to wait reflects a reasonable inference.

If this is correct, then the capacity to resist immediate gratification in favor of longer-term goods may be cultivated over the course of a life. This seems true enough for anyone who has learned an instrument or a new sport, where you start off having a bad time but stick with it until you eventually get better. That’s not to say we are all blank slates, but rather that the persons we become are in part the product of environment and experience. The ancient Greeks had a term for what happens when the child reaches for the marshmallow despite knowing she should wait: akrasia, sometimes translated as “weakness of will.”

Kramer vs. Kramer

Akrasia presupposes that self is divided and choice collapses at the moment of action. There are multiple sources of motivation pulling in different directions, though they are not made equal. The agent identifies with the judgment and recognizes it as “better” or more wholly theirs, whereas the impulse is deemed alien or at least insufficiently representative of the person they want to be. The defining feature of akrasia is that the judgment persists even as you act against it. You know you’re failing which is why you lament that you “knew better.”

Three conditions must hold for this self-opposition to occur:

• There must be some standard to act against. The agent must believe that some alternative is better. Without this prior judgment formed through experience and reasoning, acting on an impulse is neither virtuous nor vicious.

• The agent must be aware of deviation. The agent knows, in the moment or shortly after, that they have failed by their own standard. This is why akrasia is accompanied by regret of having done what one resolved not to do.

• There must be some struggle involved. The agent who resists effortlessly by definition cannot exercise self-control. But the person who gives in and the person who resists both experience a form of temptation.

These conditions remind us that temptation is in some respects good for autonomy. This is an old idea, articulated by the likes of Aristotle and the ascetics, that takes the moments in which we are tempted as the moments where autonomy is strengthened. To resist is to become resistant and to abstain is to become abstentious.

Today’s temptress is AI. It’s on hand to help us with tasks big and small, from compiling a Twitter thread to vibe-coding that app you never got around to working on. Most of the time that’s great. There have been a few projects that I quite simply would have never done without Claude. And yet Claude tempts me to summarize the paper rather than read it or to automatically add notes to my Notion when I finish a new book instead of puzzling out what it meant to me.

I think about AI, fantastically useful as it is, as a kind of meta-temptation: a temptation to remove the conditions under which ordinary temptation occurs. This is different to a kind of “digital akrasia” where, for example, you get notifications from Uber Eats to order something which might not have otherwise been done. Pizza for supper instead of that ensalada caprese you were planning.

This kind of effect is less interesting to me, and doesn’t really have anything to do with AI-as-meta-temptation. The more compelling instance — where we skip the need to form our own judgments — does. AI temptation encourages us to stop deliberating, which is worrisome precisely because deliberation might let us see temptation for what it is. Consider two types of akrasia: clear-eyed and means-end.1

In clear-eyed Akrasia, you see the situation accurately and fail all the same. You think “I shouldn’t smoke” while lighting the cigarette. Afterwards you feel guilty because you saw clearly and failed anyway. In means-end Akrasia, you cloud your own vision. You still believe “I should be healthy” while telling yourself a story that disconnects this action from that judgment. You say things like “one cigarette won’t hurt” or “I’ll quit next month” so you don’t have to see this particular instance as a violation of your own standard.

AI temptation is distinctively means-end akrasia, but it goes further because it short-circuits opportunities for deliberation altogether (the situations in which we either strengthen our autonomy or succumb to akrasia). When a person convinces themselves “AI is just a tool!” or “the core ideas are mine!” or “I’ll review it anyway!,” she disconnects its use from the need to do her own thinking. That would normally be plain old means-end akrasia, except that a) the thing you’re outsourcing here is deliberation and b) deliberation is the capacity you need to see your rationalizations clearly.

When you ask Claude or ChatGPT to draft an email, you haven’t yet decided what you think. You are tempted to outsource whatever thinking was required and, assuming you just hit send without re-reading, don’t weigh its content. The obvious response here is something like: fine, but if I review, edit, or rewrite then aren’t I still exercising judgment? This example only works for extreme passivity, and lots of people don’t just hit send and move on.

The difference is that when you review an AI draft, you tend to be evaluating rather than generating. This is the basic idea of writing-as-thinking, which holds that blank pages are there to help further your understanding rather than be filled with size eleven arial. Would you have written the same email? Maybe. But you didn’t muddle through the process that might have surfaced alternatives, so you’ll never know.

Granted, it is difficult to say what counts as “generative” in this context. One such scenario might see a person think about a given topic and weigh the tone and key points to make. Your prompt might be as specific as possible and leave very little room for the model to go off-piste. But finding the right word or rhythm isn’t only about getting the execution right. “I’m sorry but I can’t make it” versus “unfortunately I won’t be able to attend” versus “I wish I could be there, but” aren’t necessarily fungible. Even in the low stakes messages, I often figure out what I really mean through the act of fumbling around for the right words.

More than that, for all of the above I’m assuming the lowest stakes kind of writing and the maximal amount of resistance on the part of the author. If you always use AI like this then you must have a kind of saintly disposition. With longer or more complex writing it’s much more tempting to take the easy way out, which is what many people do much of the time.

This argument, the one spelled out over the last few paragraphs, is a very modest instance of exercising deliberative capacity. Let’s say I thought about using Claude to make this case for me. We have two kinds of decisions to make:

1. Do I use Claude or not?

2. If so, how much do I engage with Claude’s output?

Temptation operates at both of these levels, with the former a simple binary (use or don’t) and the latter a gradient consisting of things like prompting more carefully, reviewing more diligently, engaging for longer, and calibrating the threshold at which you’re willing to accept “good enough.” The person who rewrites is tempted to revise, the person who revises is tempted to edit, and the person who edits is tempted to skim. At every point along the spectrum we’re being tempted with another opportunity to do less.

Athletes of God

In the fourth-century, the Christian philosopher Anthony the Great withdrew to the Egyptian desert in search of discipline. The desert was a gymnasium, a place where you trained the will by struggling against it. These ascetic monks called themselves “athletes of God” because they believed that the conditions that allow us to err also allow us to stay the course and become stronger for it.

The Desert Fathers thought that temptation was necessary for growing into the people we want to be. They chose to do the hard things because they were hard. Today, doing the hard thing is less necessary than ever. Unlike the ascetics, who chose once and lived with it, technology continually presents us with ways of doing less.

A clear-eyed akratic at least knows she is falling short. A means-end akratic, aided by a tool that makes the rationalization feel reasonable, does not. Meta-temptation is a distinctively means-end form of akrasia that allows you to convince yourself that Claude or ChatGPT was only helping, one that goes further because you outsource the faculties you need to see through these post hoc rationalizations.

No one wakes up one morning and decides to stop thinking. We start by outsourcing the stuff that doesn’t matter and discover, weeks or months later, that the boundary between trivial and meaningful is getting harder to spot. The email becomes the memo becomes the proposal becomes the argument. Each step is small enough to rationalize and the rationalization is always the same: one more time can’t hurt?

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Every builder’s first duty is philosophical: to decide what they should build for. This series asks 9 questions to founders who are building towards their vision of the human good.

Today’s guest is Zena Hitz. Zena is the founder of Catherine Project, which builds communities of learning through online courses and reading groups.

Zena is also a Tutor at St. John’s College and the author of Lost in Thought: The Hidden Pleasures of an Intellectual Life.

1. What are the core questions or beliefs driving your work?

I founded Catherine Project because I had the sentimental idea that learning and thinking for its own sake is a basic human need. Since I've founded it, and we've seen thousands of people come to us, I believe it is actually true. Our educational institutions have largely forgotten this fact, if they ever knew it.

The other reason I wanted the Project to be open to anyone who was interested was to see for myself how much interest there really is. My conclusion: there's a lot. I decided not to charge tuition, because learning only happens when the learner decides to do it. It can’t be bought, even if some opportunities for it can be. If we charged tuition, they’d expect us to do their learning for them.

2. What future are you building for?

I can see two futures. In one, colleges and universities lose their last remnants of serious learning, and organizations like Catherine Project keep practices of serious learning alive for future generations.

In another future, organizations like mine display the enormous demand for liberal education that all of the educational leaders have been denying. That drives the old institutions to measures of reform and so colleges and universities return (or partially return) to their original mission.

3. What commonly held belief in the tech community do you believe is wrong?

Everyone seems to want “scale” and “impact” and seems to think that means having a massive operation. But doing something right on a small scale breeds imitators—it is like a seed. Great Books institutions were always small, but they have been enormously influential. There’s nothing more “scalable” than wheat or bread, but no one owns all the wheat nor the bread-making recipes. It doesn’t take much thinking to see that this is for the best.

Technology only works for human flourishing if we choose to design it and to use it that way. We need to think about how we want to live—in common as well as individually—and choose accordingly. Likewise, money isn’t the most important thing. To build trust, you need to care about something that recognizably benefits more than just you and your friends.

4. What are your main philosophical influences?

For me the biggest puzzle is how intellectual excellence can and should shape our everyday choices and our public or communal projects.

Plato and Aristotle argued eudaimonia or human flourishing was constituted by the pursuit of theoretical philosophy. Moreover, they thought the wholehearted pursuit of theory, paradoxically, was the best guide to life. In part that's because eudaimonia is always the best guide to life. Mixed motives are not stable, and tend toward the dominance of the strongest and worst motive in the mix. There's only one pursuit that matters most to us. Since that motive shapes everything else, there's nothing more important than getting it right.

5. What does human flourishing mean to you?

It means the exercise of our best capacities, the power to think and the power to love. We develop those capacities by cultivating the virtues like courage, generosity, prudence, moderation, wisdom.

We need communities on our scale—think large families or small towns—to provide training courses in virtue and meaningful paths of exercise for our activities. Poor and humble people often flourish more fully than the rich and powerful because they rely more heavily on their communities and live more intimately with them. That’s where the best of life is. I came to see this from joining the Roman Catholic Church but I think the other major religions teach it. I’m not sure anyone else does.

6. What’s one book you’ve read recently that you’d recommend?

I’m not alone in thinking the Robert Caro biographies of Lyndon Johnson are incredible.

They display perfectly the tensions between character and ambition. Caro is never glib and he never takes an easy answer.

7. What’s your most irrational belief?

That I—my will and my talents—am the chief cause of my success. In fact it was mostly luck, grace, and animal spirits. Relatedly, I have the repeated delusion that whatever the problem is, I am the person best equipped to solve it. No matter how many times I see that proved false, the unsinkable belief always rises again and plagues me when I need it least.

8. What’s the most interesting tab you have open right now?

I have a tab listing the famous social science studies that have not been replicated. We swim in fake science, so much that we can’t even make a simple argument without appealing to it. We so sorely need the arts of study, listening, reading, thinking, and judgment. “Studies show...” cannot replace a solid, well-informed human judgment.

9. Who is one writer or thinker today who you think is underrated?

Well, to add a spin on the question…we are living, active beings who can feed our reflection and imagination on anything that has been thought or said.

And so I think the most underrated thinker is whatever great thinker of the past you haven’t read yet—or wherever your understanding is most limited.

For me, it's the Enlightenment authors I know least. I have been gradually untangling Descartes' philosophy and science through my teaching at St. John's and I find it very exciting.

Thanks to Zena for answering “What Will You Build For?”

To get in touch, find her on X or at Catherine Project.

This is the third installment in this interview series, see our first interview with AI Underwriting Company co-founder Rune Kvist and our second interview with ex/ante founder Zoe Weinberg.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

If you’re someone who thinks deeply, builds deliberately, and cares about the future AI is shaping: join the Cosmos network.

To nominate someone for “What Will You Build For?” leave a comment below, or send us a DM.

Every builder’s first duty is philosophical: to decide what they should build for. This series asks 9 questions to founders who are building towards their vision of the human good.

Today’s guest is Zoe Weinberg. Zoe is the founder of ex/ante, an early-stage venture fund backing technology that advances human agency and digital freedom. She talks more about her work in this short video.

Before ex/ante, Zoe worked on ethics and policy issues at the National Security Commission on Artificial Intelligence, Google AI, and the World Bank.

1. What are the core questions or beliefs driving your work?

It’s tempting to fall into standard narratives of technology as bringing either salvation or apocalypse. It’s all a bit Manichean—a black-and-white view of technology as good or evil, often in those stark terms.

I’m more interested in questions of how technology can subtly reshape human agency, human rights, and human flourishing, all depending on how it’s built. Technology can democratic or authoritarian. It can expand freedom or constrict it. It can advance open societies. Or not. There’s very little that’s preordained. There’s very little that’s not ultimately in our hands as builders, funders, and users.

2. What future are you building for?

Through my work at ex/ante, I’m trying to contribute to a future that centers individual autonomy, freedom, and choice, counteracting a digital world that feels increasingly circumscribed and controlled by governments and a handful of large tech companies.

This means a world where systems make knowledge and capability more accessible, and where the friction and difficulty that often accompanies deep understanding isn’t eliminated, but is part of what creates meaning and joy.

One version of this was captured recently in the Resonant Computing Manifesto. This suggested that artificial intelligence can give rise to digital experiences that nourish, rather than drain, us if we build with intention and thoughtfulness. I’m building for the future where innovation isn’t just for innovation’s sake, but toward the creation of a healthier and more free society.

3. What commonly held belief in the tech community do you believe is wrong?

In the tech world I often hear people say that technology is value-neutral: it’s how people use technology that can have good or bad consequences. I disagree.

Technology is purpose-built, and therefore is embedded with intent. What’s its purpose? How’s it meant to be used? Technical architecture always embodies a set of values-laden choices: the design of protocols, interfaces, and systems that don’t just enable certain actions but actively shape the distribution of power, the visibility of alternatives, and the very modes by which users understand their options.

There’s a million little decisions that go into product- and company-building that ultimately have a moral valence. To make some of those decisions easier, ex/ante and the Ford Foundation released Builder Blueprints, a library of templates and documents for start-ups to raise the bar in terms of privacy and user rights. It’s just one piece of the equation, but it is all part of the foundation on which a company is built.

Calling technology value-neutral is a cop-out.

4. What are your main philosophical influences?

I will never forget the first time I read Thomas Kuhn’s Structure of Scientific Revolutions. I am often reminded that basic assumptions about the world that we take as “science” and “fact” are in reality socially and historically-contingent. Science and progress are hardly linear, and new (often contrarian) perspectives are critical to improving our understanding of the world. In the context of my work at ex/ante, I often think about what technological developments on the far edges of the horizon could substantively change my mental model of how the world works or contribute to a true paradigm-shift.

Many of the other thinkers and writers that have influenced my work are represented in the Tech for Human Agency Library, including John Stuart Mill, Alexis de Tocqueville, Michel Foucault, Marshall McLuhan and many others.

5. What does human flourishing mean to you?

For me, human flourishing is rooted in human agency: the ability to make substantive decisions about one’s life. It requires a social structure that enables individual freedom, individual judgment, and pursuit of individual purpose.

“Individual” is the key word. Any infrastructure (technological, cultural, political or otherwise) that imposes uniformity diminishes the conditions for human development and progress. Self-authorship emerges from the struggle to understand and solve genuinely hard problems, to navigate moral complexity, to confront perspectives that are at odds with one’s worldview. On a societal level, human flourishing is necessarily plural.

The question for our moment is whether technology can expand or contract the possibilities for self-authorship and whether we can design these systems to preserve the productive friction that gives rise to human flourishing. On an individual level, human flourishing is less about achieving any particular state of happiness and more about having the capacity for self-authorship.

6. What’s one book you’ve read recently that you’d recommend?

Second Life: Having a Child in the Digital Age, by Amanda Hess.

I recently became a parent, so I appreciated this memoir about the ways that the digital world has shaped one of life’s biggest milestones, by an internet culture critic for The New York Times.

7. What’s your most irrational belief?

My irrational prediction of an irrational belief is that we will see the emergence of a religious movement centered on worship of AI and superintelligence. A great awakening for the 21st century.

8. What’s the most interesting tab you have open right now?

The website for the Center for the Alignment of AI Alignment Centers. I am a big believer in satire and humor as a method for sense-making. I also think creator Louis Barclay is hilarious and brilliant.

9. Who is one writer or thinker today who you think is underrated?

An idea that is underrated is the Free-Market Welfare State. It’s a concept described by Samuel Hammond in 2018 policy paper, suggesting that a strong social safety net can encourage more innovation, risk-taking, and entrepreneurism.

By giving citizens guaranteed support, such as health insurance and retirement security, we remove the pressure for protectionism and intervention during periods of technological and economic disruption. At the same time, we empower individuals to take risks, knowing they and their families won’t slip through the cracks. The proposal doesn’t fit in a neat partisan box, which perhaps is why it does not get nearly the attention it deserves as an economic policy solution.

Innovation and progress depend on the willingness of individuals to take risks, so policies that make taking a leap of faith easier are ones I’m likely to be in favour of.

Thanks to Zoe for answering “What Will You Build For?”

To get in touch, find her on LinkedIn or at ex/ante.

This is the second installment in this interview series, see our first interview with AI Underwriting Company co-founder Rune Kvist.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

If you’re someone who thinks deeply, builds deliberately, and cares about the future AI is shaping—join the Cosmos network.

To nominate someone for “What Will You Build For?” leave a comment below, or send us a DM.

Pepper was once a luxury. Medieval merchants traveled halfway around the world to bring it to European tables. Now it sits in shakers at every restaurant, free for the taking. It became cheap because people wanted to put it on everything.

Intelligence is becoming pepper.

As Sam Altman puts it, intelligence “too cheap to meter.” Open-source models already make raw cognitive power free at the point of use. Soon it will be everywhere, layered over every interaction, and available for every task. Our operative intelligence, the combination of biological and artificial neural networks, will expand dramatically. But as artificial intelligence becomes cheap, biological intelligence may decline. A society can grow smarter while its members’ skills atrophy.

Most people find this prospect irredeemably bad. I think some atrophy is acceptable and sometimes beneficial. This thesis is surprisingly moderate. We already accept strategic cognitive atrophy when the trade-offs are right. Calculators have led to the atrophy of mental arithmetic capacity, yet mathematicians today are better at mathematics than ever. They gave up tedium and they gained something greater: the ability to solve harder problems.

Mathematicians now routinely use computer algebra systems (such as Mathematica) and programming languages (such as Lean) to support mathematical reasoning. This lets them explore problems they could not before. The four-color theorem was proven in 1976 with computer assistance, as no human could have checked all the cases themselves. This pattern is ancient. Writing hurts our memory, but no one wants to stop people from writing. This precedent shows that cognitive offloading can enhance human flourishing when properly structured.

AI will cause some cognitive atrophy. Offloading is inevitable when intelligence becomes free. We’ll need to decide which forms of atrophy we should accept given their inevitability.

Trade-offs

John Stuart Mill and Aristotle help clarify when these trade-offs prove acceptable. Mill distinguished between higher and lower-order pleasures. By higher pleasures he wasn’t talking about “refined” tastes or elite entertainment. He meant the pleasures that engage our distinctively human capacities: reasoning, imagination, and moral feeling. As Mill put it in Utilitarianism, we “assign to the pleasures of the intellect, of the feelings and imagination, and of the moral sentiments, a much higher value as pleasures than to those of mere sensation.”

Consider someone who has experienced the satisfaction of a nice-tasting meal and the satisfaction of solving a difficult problem. If she is honest, she will rank the second higher. These provide a deeper satisfaction.

This is Mill’s “competent judges” test. Ask anyone who has genuinely experienced both kinds of pleasure which they would give up. They will sacrifice bodily pleasures first. The fool may be content. But Socrates, even when dissatisfied, has something the fool will never have.

Aristotle made a similar observation over 2,000 years prior. He distinguished three lives: the life of pleasure, the life of political action, and the life of contemplation. The life of pleasure seeks bodily satisfaction. The life of action seeks honor and achievement. But the contemplative life seeks truth itself.

Contemplation is not passive. It is the most intense activity the mind can perform. It means thinking about the highest things: the structure of reality, the nature of the good, and the order of the cosmos. This activity is not labor-intensive like farming or manufacturing, and it’s not leisure either. It is the mind at full stretch.

Aristotle thought most people could not sustain a contemplative life. The demands of survival intervened. But he also thought that insofar as we can contemplate, we should. It is what we are for.

This is where AI’s promise shines through. It can handle lower-order cognitive tasks, freeing us for activities that engage our highest capacities. A researcher who once spent hours hunting for sources can now spend those hours thinking about what the sources mean. A writer who labored over formatting can focus on whether the argument is true.

Of course, people may not do this. We all know that AI can seduce us into passivity. But I would argue that it also elevates the opportunities for virtue. The mind freed from drudgery really can rise. It can shift us from the mechanical to the meaningful. If we follow the competent judges test, as Mill would have us, we know that most people do not prefer the mechanical once they have moved beyond it. Many people really do have the time to cultivate virtue and devote themselves to pursuits that satisfy them and that achieve their highest values. AI gives us the chance to do that.

Following Mill and Aristotle, cognitive offloading is acceptable only when it preserves these capacities. It can become noble when it enhances them.

What to Offload

Consider two patients facing the same diagnosis. The first asks an AI what treatment to pursue and then simply does it. She has outsourced both the information-gathering and the judgment itself. Her capacity for medical reasoning atrophies because she never exercises it. She cannot evaluate what the AI told her. She cannot ask her doctor the right questions. When the AI errs, she has no way to catch its mistake.

The second patient uses AI differently. She conducts an extensive medical review, reading research reports that the AI surfaces. She generates a list of questions for her doctor based on what she learned, brings the reports to the appointment, and discusses them. The doctor makes the call, but with a better-informed patient in the room.

Both patients offloaded cognition. But the second preserved her deliberative capacity. She did not hunt for the research herself (there’s no free cognitive lunch), but she did something harder: evaluate it. Her speed at reading dense medical literature may atrophy. But her ability to weigh trade-offs, to question authority, and to integrate information grows stronger.

Offloading exists on a spectrum. The same pattern emerges in education, but with deliberation more fully intact. A student can have AI do her homework and then pass it off as her own. She speaks fluently about concepts she doesn’t understand. The appearance of intelligence replaces its reality. Like muscles that atrophy without use, her reasoning capacity withers.

Or she can use AI to challenge herself. Study mode forces her to work through problems rather than receiving answers. The AI asks questions, corrects misconceptions, and refuses to simply hand over solutions. Each session builds her capacity. She’s still using AI. But she’s building human thinking on top of AI thinking.

Autopilot shows what happens when atrophy goes wrong. Modern pilots offload constantly, and for good reason; automated systems handle airspeed, altitude, and navigation almost all of the time. The pilot watches the machines work, which reduces fatigue. Flights have become far safer as a result. Still, there is a loss. As we know from the movies, hand-flying an aircraft keeps pilots in the loop, letting them feel the aircraft’s responses intimately. That helps create a continuous mental model of where the plane is located and what comes next. Autopilot allows for drift, and the mental model fades.

Air France 447 is illustrative. In 2009, over the mid-Atlantic, the aircraft’s pitot tubes iced over, and the autopilot disconnected. The pilots had to hand-fly in unusual conditions and made simple mistakes. The aircraft stalled, but the pilot pulled the nose up. The flight had 228 people. None survived.

The pilots had thousands of hours of flying time between them, but their critical skills atrophied from disuse. Automation makes us safer while making rare emergencies devastating. The mathematician who offloads arithmetic still knows how to solve hard problems. The pilots who offloaded flying could not fly when it mattered.

What distinguishes good offloading from bad? The examples suggest an answer. Offload mechanical cognition: the tedious, repetitive operations that don’t require judgment. Preserve core deliberative capacities: the ability to evaluate, choose, and reason through hard cases. Make sure evaluating AI outputs requires active engagement. Expand higher-level intellectual activity. And preserve cognitive sovereignty: maintain the ability to contest what the AI tells you, to understand how it reached its conclusions, and to exit when you need to.

Permissible atrophy

In my previous essay for Cosmos, I argued that intelligence environments require contestability, transparency, and exit rights. Permissible atrophy is the next question, and it is not just a personal one.

We are designing intelligence environments for one another. A teacher who assigns AI tools shapes what her students become. An employer who deploys AI assistants shapes what their workers can do. These choices become the formal and informal rules that govern how we think.

We have freed people from drudgery before. Some wasted the freedom. Others turned toward higher things. AI can go either way. Build it wrong, and we may create sophisticated parrots: fluent, confident, and hollow.

Pepper became cheap because people wanted to put it on everything. Intelligence is becoming cheap for the same reason. How we embed it will determine who we become.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Technological progress is forcing a question: what kind of people do we want to become in a world where thinking is optional?

Cosmos Institute exists to answer that question, by developing philosopher-builders who can translate the principles of a free society into code. Now we’re growing our team.

In our year-end letter, we shared what we built in 2025:

• 140+ grantees through our fast grants program

• 16 fellows working at the intersection of philosophy and frontier AI

• 14 seminars with partners like Oxford, DeepMind, and Liberty Fund

• 8 philosopher-builders creating new companies

We’re looking for people who believe AI systems should promote human autonomy, enable truth-seeking, and resist central control, and who want to shape the institutions that make that vision real.

At Cosmos you’ll have direct exposure to top AI thinkers and builders, freedom to design how you work, and the chance to help shape the institutions and companies that will define the AI age.

We’re hiring for three core roles:

• Head of Operations

• Head of Finance

• Head of Talent and Network

Details below ↓

Head of Operations

Location: Remote-friendly (preference for Austin, New York, or London)

This role builds the operational infrastructure that allows Cosmos to scale: systems that work, a team that delivers, and a culture built on fast feedback loops.

You’ll own operational delivery for seminars, fellowships, grants cycles, and formation programs, including coordination for The Academy launch. You’ll build and maintain operations systems across Cosmos: payroll coordination, compliance, vendor management, HR systems, and reporting. You’ll establish the operational cadence that works across a distributed team, and build systems that make every cycle better than the last.

You’ll have budget for 1–2 direct reports in Year 1 and contract support as needed. As Cosmos scales, you’ll shape what the operations function becomes.

We’re looking for someone with 5-10+ years experience in operations or high-responsibility chief of staff roles who has personally touched payroll, HR systems, vendor management, and compliance. You take ownership of outcomes, have exceptional follow-through, and know the difference between what should be systematized and what should remain judgment-driven. Read more →

Apply

Head of Finance

Location: Remote-friendly (preference for US-based)

This is Cosmos’s first dedicated finance hire: hands-on enough to run daily workflows, strategic enough to build a modern finance operating system that scales.

You’ll own core finance (accounting, AP/AR, close, reporting, controls, audit/990 coordination) and forward-looking finance (budgeting, forecasting, KPI dashboards). You’ll set up our in-house finance function, lead the relationship with our external audit firm, ensure clean tracking of restricted funds and program costs, and support fundraising with credible numbers. You’ll make finance approachable: clear docs, fast responses, calm execution.

We’re looking for a CPA (or equivalent) with 4–8+ years in accounting/finance, ideally with nonprofit exposure. You treat donor and grant dollars with real seriousness, and you’re a hands-on operator who’s happy to process transactions and design better systems. Read more →

Apply

Head of Talent and Network

Location: Remote-friendly (preference for Austin or London)

Over 2025, Cosmos developed a network of hundreds of top technologists and thinkers: 16 Fellows spanning frontier AI, startups, and philosophy (including economist Tyler Cowen and Anthropic co-founder Jack Clark); 140+ grantees building prototypes and research; and 200+ attendees of deep-dive programs with partners like Oxford, Aspen Institute, and Liberty Fund.

This role takes that early strength and builds a compounding network, where the best people feel supported, make progress faster, publish and ship more, and pull in the next wave of talent.

You’ll identify and source exceptional people, assess and route them into the right pathways, and deliver value through introductions and opportunities that unlock projects, jobs, mentors, and collaborators. You’ll run the network operating system: the database, metrics, and workflows that turn messy relationship work into clean processes, freeing up your time for high-touch work.

We’re looking for a high-energy connector with 3–7+ years in talent, community building, partnerships, or accelerator/fellowship programs. You have taste and judgment: you can tell the difference between “impressive on paper” and “actually exceptional.” You can speak with researchers, founders, funders, and philosophers. Read more →

Apply

Pitch us a role

If you have a thesis on how you can contribute to our team outside of these areas, please get in touch.

We’re likely hiring soon for an operations role focused on the build out of an in-person Academy, and a researcher to contribute to work on AI, human autonomy, and societal impacts.

If our mission resonates, we’d love to hear from you.

Pitch us a role

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

There are two things that I rarely take off: my wedding ring and my Garmin watch. The former stays on because it seldom crosses my mind to remove it. To tell the truth, I’m not even sure I could take it off without the right combination of soapy water, elbow grease, and patience. The latter, the Garmin, stays put because I have a regrettable obsession with bodily metrics. I like to know my heart rate, stress levels, and especially the state of my body battery (a wildly inaccurate read about how much fatigue I ought to be feeling at any given moment).

I’m not one of those life-hacking optimization types, but I wear the watch all the same. It helps me track runs and workouts, which I review every day or two when I lack enthusiasm for another trip to the gym. People often tell me my preoccupation with these figures probably makes me more stressed. They might be right. Nonetheless, I keep using it because I worry that without constantly reviewing breaths-per-minute or VO₂ max I’d be tempted to call time on this wellness thing.

At least in my imagination, fitness – and so Garmin – is a part of my personal quest for eudaimonia (usually translated as ‘flourishing’ or ‘happiness’). Eudaimonia is less a feeling or a state than the activity of living in accordance with virtue. I think that fitness helps me become someone who’s disciplined and fairly healthy, and that’s the kind of person I want to be. In this light, the question that my smartwatch poses is an uncomfortable one: can flourishing be optimized for or must it be cultivated from within?

Garmin’s algorithms are neither sophisticated nor totalizing. They stay in their lane and have a rather narrow set of powers of suggestion. They don’t tell me how to exercise anymore than a pencil tells me what to write. Others, like recommender systems or large language models, aren’t so reserved. They suggest where to go, what to have for breakfast, what to watch on TV, and even how to broach a difficult subject with someone we care about. At the limit, these systems may make all of our decisions for us in a scenario we call “autocomplete for life.”

The Wood from the Trees

In On Liberty Mill tells us something about the human condition: “Human nature is not a machine to be built after a model, and set to do exactly the work prescribed for it, but a tree, which requires to grow and develop itself on all sides, according to the tendency of the inward forces which make it a living thing.” For Mill, human flourishing is developmental. It cannot be achieved through external optimization because the process of growth is essential for the good life. This is why he describes human nature as a tree. The metaphor captures the idea that we grow from latent capacities and inclinations that must be discovered and shaped through self-directed action. Those tendencies suggest paths, but they require autonomy to be actualized.

In the Nicomachean Ethics, Aristotle tells us that flourishing flows from virtue and virtue through habituation. Rather than asking “what should I do?” he encourages us to ask “what kind of person should I become?” Grounded in the way people act in the real world, Aristotle’s approach to ethical life focuses on character development over rule-following or achieving the greatest good for as many people as possible. His “golden mean” (finding virtue by balancing extremes) requires phronesis or practical wisdom, the capacity to discern the right action in particular circumstances developed only through living in the world.

For both Mill and Aristotle, human flourishing is a journey we make every day. Try as we might, there are no shortcuts we can take. The activity of living a self-directed life can never be automated because it is constitutive of what it means to flourish in the first place. A life in which all the right choices were made for you might be pleasant, even enviable, but it would not be yours in the sense that matters. This belief, that flourishing cannot be optimized from the outside, is based on three separate but related ideas: (1) flourishing is a process of being; (2) it requires careful self-authorship; and (3) it must be maintained solely from within. An optimized outcome can simulate contentment, but it cannot substitute for active exercise or resilient capacities.

Go Forth and Optimize

Machine learning systems are trained by optimizing a loss function, but in deployment most function as pattern-completion engines rather than goal-directed agents. When these systems engage humans, they typically optimize for an external objective (e.g. engagement, throughput, or compliance) and treat the user as the site of optimization rather than its author. Even if the objective was human flourishing, we would remain the patient of optimization rather than its architect. The technical apparatus assumes that a desirable end-point can be specified in advance so that the system constructs outcomes rather than growing them. That is unproblematic for the most part – it’s just how the technology functions – but it begins to surface some curious problems when it comes into contact with our efforts to live deliberately.

You can see this dynamic in play with recommender systems that suggest what you want before you have had a chance to reflect on it. These systems learn from aggregate behavior to predict and shape individual choices, like suggesting the optimal route or by pushing you towards television shows or music that “someone like you” ought to like. A kind of overfitting to past behavior can restrict users to previously expressed interests, generating recommendations that whittle away the variety of options needed to recognize the opportunity costs of choice.

Core to this process is the “nudge” wherein the presentation of choices is shaped by an outside actor. Unlike earlier static nudges (such as placing healthy food at eye level), AI-driven nudges operate continuously and adaptively. Given enough scale, this transforms what was once soft paternalism into something closer to soft totalitarianism as fine-grained personalization makes interventions harder to collectively resist. The more criteria are centrally set, the greater the attack surface for those who would exploit them, and the more these systems inhibit the decentralized adaptive learning through which individuals and societies discover what works. When the environment is engineered to produce predictable choices, the capacity to exercise choice becomes harder to sustain.

A common defense of nudges holds that they preserve freedom of choice while improving outcomes for society as a whole. Proponents suggest that nudges influence behavior mostly for our benefit, and that they are easy to avoid if we put our mind to it. If people can always choose otherwise, their autonomy is not compromised. The rub is that true autonomy deals with at least both first-order choices (I want a cigarette) and second-order reflection on those choices (I wish I could stop smoking). Nudges often bypass reflection – how many of us are content with taking the first movie recommendation we see? – preserving formal freedom of choice while undermining the authorship that makes them our own. Defenders of nudges might say something like “if someone’s going to unthinkingly pick the first movie, it might as well be a good one.” But this line of thinking doesn’t engage with the fact that nudges make deliberation harder in the first place. When we engage in critical reflection, we weigh alternatives and formulate our own reasons about what to do. That process is harder to initiate when a “good enough” option is presented to us that bypasses the need for deliberation.

More than that, the idea that nudges are easy to avoid fails given enough altitude. Even granting that individual nudges may be possible to avoid, like walking past that salad bar and opting for a cheeseburger, an ambient ecology of personalized nudges is not. This gets at one of the curious aspects of autonomy: for such a personal thing, it depends on social life and the opportunities to contest and compare it brings with it. Autonomy fails when exit is nominal or when commitments become irreversible. Of course all choices exist within a constellation of potential options, but it is possible in principle to structure those potentialities through systems that keep inquiry distributed rather than centrally engineered.

Another critique of the developmental view of human nature (that is, Mill’s tree) suggests that the problem is not optimization per se but poorly specified objectives. If we could align AI systems with genuine human goods, including autonomy, then optimization might in principle be made compatible with human flourishing. One influential proposal, associated with AI researcher Stuart Russell, introduces a simple framework based on three related principles: (1) the AI’s only objective is to maximize the realization of human values; (2) the AI is initially uncertain what those values are; and (3) human behavior is the primary source of information about values.

But if flourishing really is an activity rather than a state, then it may not be the kind of thing that can be maximized by an external agent. Like happiness in Aristotle’s account, it might simply be a good that must be exercised from within. In practical terms, learning values from behavior must grapple with its own circularity. If current behavior reflects preferences already shaped by prior optimization, the system is learning from wants that we haven’t fully endorsed.

More fundamentally, observed behavior may be the wrong place to look. Conduct reveals who we have been, not who we are trying to be. The developmental view suggests that we are always undergoing growth, and the distance between our actions and our aspirations is where character development takes place. No amount of behavioral data, however rich, can capture what we have not yet become. That said, Russell’s humility principle points toward something important. It reminds us that systems that surface uncertainty and return judgment to the user are more autonomy-preserving than those that do not.

Another problem deals with adequately specifying preferences for AI systems in line with the developmental view. Current models focus on revealed preferences (what we do) in service of a goal that they set (maximize time on platform in the case of recommendation engines or assist in a “helpful, honest, and harmless” manner for language models). Simply telling the system what you want to want isn’t enough; these stated preferences may not necessarily reflect the kind of person you are trying to be. Deeper approaches to character development will be needed for systems that help us realize our potential.

A final objection to the developmental reading of human nature might stress that all technology assists human action. Writing supports memory and calculators aid arithmetic (though overreliance can enfeeble both). If external assistance inherently undermines autonomy, shouldn’t we be wary of tools altogether? But the difference between AI and other technologies turns on whether assistance extends the agent’s deliberation or replaces it. A map or pencil extends my autonomy, a chatbot that makes decisions for me does not.

The Constant Gardener

Am I using the system to pursue ends I have reflectively endorsed within, or is the system shaping my ends from without? Even when you supply preferences, a system must decide how act on them according to its own constitution. If we have little ability to override those rules, well-meaning efforts at governance may inadvertently prevent systems from supporting our personal quests for eudaimonia. Consider a recommendation system that learns my preferences from behavior and serves content to maximize engagement. The system optimizes for my revealed preferences. But there’s no reason a recommender system needs to operate this way. It’s easy enough to build a system that allows users to think carefully about the type of person they would like to become by using the same technology. The former might serve you reality TV because you watched a cooking show; the latter might populate your feed with guides for cooking dishes you’ve always wanted to make.

This is better, but even this kind of second order endorsement is insufficient. If you tell a system the kind of things you want to be interested in, then you may find that the system quickly optimises for them in a way that does little to help you grow as a person. Watching more arthouse cinema and fewer Netflix originals has little bearing on the type of life you want to live. Instead, systems ought to deal with the stuff that matters. Personal agents, for example, could be designed to sift interests from from character development. They could help us think about who we want to become rather than engineering the outcomes we think we want.

In practice, that might mean an AI assistant that makes its influence legible. Rather than covertly curating your information environment, a character-supporting agent might show you the paths not taken. It might introduce the unfamiliar because growth requires encounters beyond the experiences you’ve already had. At minimum, such systems should make exit realistic through portability and contestability and ensure we sign off any changes that may influence our higher order commitments. Rather than simply checking whether your goals have changed, they could surface tensions between your commitments to help you live more deliberately.

Optimizing flourishing from the outside is a non-starter because flourishing is active and authored. But if the locus of deliberation remains with us, perhaps AI can help us live a little more wisely.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

Radio arrived in the early twentieth century as pure possibility. Electromagnetic waves carried human voices across continents. Within decades, three versions of the same technology had helped produce three different kinds of citizen.

In the 1930s and 1940s, the British Broadcasting Corporation (BBC) built programming that assumed its listeners wanted to become more than they were, with challenging content and genuine debate, designed to stretch rather than soothe. The funding model made this possible: residents paid a license fee for all BBC programming regardless of what they watched. No one had to maximize time-on-station. They could pursue something else.

American commercial radio faced a different pressure: advertising. Audiences were larger when programming demanded less. The market cleared at entertainment. What sold ads was what held attention, and what held attention was not what elevated the listeners.

Soviet radio under Stalin did not bother with the pretense of serving listeners at all. The technology became an instrument of state narrative designed to bluntly manufacture the appearance of consensus and compress the space for independent thought.

The same transmitters and waveforms propagating at the speed of light formed the technical basis in all three cases, but divergent selection pressures yielded divergent equilibria. The funding models were de facto governance regimes, entangled with the political order, shaping what could be said and what could be heard.

Looking back now at the introduction of the radio, it is easy to see that the question of whether “the radio” is conducive to the good or whether it advances human autonomy is hard to meaningfully address without considering funding models. Yet whenever new technology is deployed, public discourse tends to fall back to discussing the technology in itself without reference to its economic implementation. For example, people are largely happy to discuss the ills and benefits of social media, and even legislate on that basis, as if social media could be evaluated in a test tube for carcinogens in the way we might evaluate cigarettes.

Technology only defines a possibility space: the affordances, constraints, default pathways, and scaling properties that structure our interaction with the world around us. Institutional choices and funding models determine what position in that space becomes reality. Moreover, perhaps more importantly, institutional choices and funding models determine what position can remain in equilibrium over time. An institutional aim to pursue a particularly virtuous part of technological possibility space cannot be maintained if the funding model forces the institution over time to either change the aim or die.

Consider modern recommender systems. The possibility space is vast: systems that surface what users would endorse on reflection, systems that maximize time-on-site, or systems that optimize for learning or serendipity or connection. But the economics of advertising-funded platforms select for a narrow band of that space: the band where engagement can be monetized regardless of whether it tracks anything users actually value. The narrowing isn’t unique to ads; ads are just the clearest case where the reward proxy is orthogonal to reflective value.

Billions of minds are routed by recommender systems daily. The unexplored regions of the surrounding possibility space are not technologically inaccessible, but they are economically unselected. We think of economics here broadly: all inputs required for a system to carry on. Even a non-profit social media site endowed in perpetuity or supported by the state would still need to capture enough attention to remain “social” in any meaningful sense. The distinctions we’re drawing are orthogonal to narrow critiques of capital.

What Institutions Compute

The cognitive scientist David Marr proposed a simple framework for how to analyze an information processing system, regardless of whether it is a human brain or an AI system. He argued that there are three families of questions that need to be answered to have a full understanding of such a system, which he thought of as three levels of analysis.

1. Aim: What is the system trying to compute? What problem is it solving?

2. Mechanism: What procedures does it use? What’s the algorithm?

3. Substrate: What physical substrate implements those procedures? What’s the hardware?

Marr observed that each of these questions has some independence. What you are trying to compute does not fully determine your algorithm or your hardware, and so on. Because of this, we can study facts about aims, mechanisms, and substrates somewhat separately. Yet, Marr also observed that how we answer one family of questions constrains how we get to answer the others. You cannot simply declare that a system has some aim. The aim must be achievable by some procedure, and the procedure must be realizable in the substrate that is available. If the substrate of your system does not support any mechanism that could achieve your favorite aim, that aim can’t be part of the right explanation of what the system does.

This framework transfers to institutions. A company and tech products are also information-processing systems, that can be understood through three analogous families of questions:

1. Aim: What is the organization trying to achieve? What problem is it solving?

2. Mechanism: What procedures does it use? What are the routines, metrics, and product mechanics?

3. Substrate: What implements those procedures? What is the shape of structures like revenue, ownership, incentives, user participation, survival pressures?

As in Marr’s case, we get constraints across levels of questions: the lower levels shape what’s possible at the higher ones. A founder might genuinely aim for VC-funded virtue-in-a-box. But if the company’s survival makes sense over time, the avowed aim is unlikely to remain what best explains its behavior. The substrate constrains what aims are stable. The substrate does not uniquely determine aims, but it sets the boundary of the viable.

Consider a game studio that sets out to build something meaningful. Let’s say the aim is “earned mastery.” What earned mastery requires is not just skill at the game, but the formation that comes from genuine challenge: learning to lose, to persist, to improve through practice. The mechanisms of the game serve that aim, with matchmaking, difficulty curves, and progression that rewards effort. The economics are aligned. Players pay once, and the only path to power is getting better. Aim, mechanism, substrate, all pointing the same direction.

Then growth becomes the mandate. Loot boxes convert well. Players want to feel powerful now, not after fifty hours. As a result, mechanics emerge that let money substitute for skill.

No one at the game studio explicitly decided to abandon the mission. Many probably saw where things were heading. But each choice was locally rational. What got measured became what mattered; what mattered became what survived; what survived became the real aim. The specter of Goodhart’s law looms large and the feedback loop doesn’t care about the pitch deck. It optimizes for what it can see. By the end, “earned mastery” appears in the mission statement and nowhere else. The aim became a ghost. Not abandoned, just no longer part of the best explanation of the company as a system, or the best model for decisions within it.

A distinction matters here. Engagement that tracks value (e.g., mastery, truth, community, or genuine satisfaction) is what good products should generate. The problem is engagement engineered through compulsive behavior, asymmetry, and the exploitation of human weakness. In the worst case, we substitute habit for growth, money for skill, manufactured urgency for actual importance.

Returning to radio, we might say that the BBC avoided structural drift in its heyday because license fees created no pressure to hold attention for advertisers. The economics supported the stated mission rather than corroding it. American commercial radio found other priorities because advertising revenue required capturing attention and capturing attention rewarded different mechanisms than cultivating it. Those mechanisms eventually became the point. This is not a case for license fees as such—the BBC subsequently developed other pathologies, and any funding model creates its own selection pressures. The point is the structural logic, not the specific policy.

The framework we described is analytically neutral. It tells you how aims, mechanisms, and economics can be articulated independently yet alerts you to how they constrain each other in real systems, particularly as they persist over time. It doesn’t tell you what aims are worth pursuing. You could use it to help design effective propaganda as easily as effective education.

Every information environment is a training regime: it determines what people practice noticing, what they practice ignoring, and therefore what they become capable of judging. The institution that shapes attention is in turn shaping the citizen.

Descriptive frameworks are useful if they help us understand systems, and more useful when they help us articulate what systems we want. When we say the BBC “cultivated” and commercial radio “captured,” we are claiming that some outcomes are better than others. Specifically, preserving and developing human judgment matters because judgment is the foundation of autonomy. Systems that cultivate judgment expand what a person can do and be. Systems that atrophy judgment reduce the agent to a consumer of impulses by executing preferences they never made their own and optimizing for ends they never deliberately chose.

If we are going to reason about institutional design at all, we need to be explicit about what we are designing for. That is a prior question, and ignoring it just means your values operate without scrutiny. For our part, we take human autonomy as our prior commitment.

From this perspective, the question for any technology that touches human judgment is not “is this technology good or bad?” The question is: in what institutional arrangements is the technology implemented, and do those arrangements create structural pressure toward or away from the cultivation of human judgment?

In practice, this means asking:

• What aim does the economic structure sustain over time? The mission statement or the aim that survives contact with the feedback loops?

• What mechanisms does that aim require? Do the proposed mechanisms only work in a vacuum to deliver your aim, or could they stably work to deliver your aim given the economics that you are prepared to implement?

• What do those mechanisms do to the humans who use them? The effect you designed for, or the effect at equilibrium?

Many companies cannot answer these questions honestly. The mission statement says one thing; the incentive structure produces another. The founders may believe in the stated aim; the economics select for something else.

If autonomy is the commitment, we can ask what design criteria this would entail. One way to proceed is then to articulate those criteria as constraints or tests. We suggest two tests below as minimal starting points:

1. The Transparent Choice Test: Could this product survive in a suitably nearby world of users who fully understand how the product works and could easily select an alternative?

A product fails this test if its economics depend on the gap between informed choice and actual behavior: it only works because users don’t fully understand it or can’t easily leave. Note that the test brackets market power. Whether it is easy to switch in the actual world is separable from whether people would switch if they could, in a suitably nearby world. A virtuous product might still be a monopoly; a monopoly might still make a good product. Whether monopolies are bad for other reasons is a separate question.

2. The Candid Aim Test: Is the stated aim key to the best explanation for how the system actually behaves and how its parts are put together?

If the stated aim is real, reference to it should make the system’s behavior intelligible and more predictable. It’s a red flag when a different aim explains more, particularly so if it is an aim that is not just orthogonal but contrary to the stated aim. Consider a supermarket with a cash register that makes subtle addition errors at the threshold of detection, always in the store’s favor. If this has been going on for years, “fair dealing” stops being a plausible aim of the store. Or consider a recommender system that reliably foments polarization. At some point, “building community” explains less than “maximizing clicks.”

Neither the Transparent Choice Test nor the Candid Aim Test is passable long-term if the economics pull against the aim.

Structure, Not Will

Structural drift cannot be resisted at the level of individual will. A founder with integrity, operating within a misconfigured stack, will be selected against by the institutional arrangements that structure their team, product, and market. If you have spent years inside one of these companies advocating for users over engagement and eventually burned out, it wasn’t a failure of will. The economics select for certain equilibria and not others. You can resist that gradient for a while, but you cannot reverse it through conviction alone.

The intervention must be structural, not only personal. This is the work of what we call philosopher-builders: configuring the stack itself.

A skeptic will ask: if a firm adopts constraints that reduce short-term competitiveness, won’t the market select it out? Isn’t anything outside of profit maximization founder self-indulgence? But this assumes profit is already defined before we’ve chosen what game we’re playing. The time horizon, the measurement regime, and what gets externalized are not givens. The builder—the philosopher-builder—chooses them.

Some commitment devices reduce competitiveness while others become the competitive advantage. Substack’s bet on writer ownership creates lock-in through loyalty rather than switching costs. The Bloomberg terminal succeeds not despite its commitment to decision-quality over engagement, but because of it. Patagonia’s environmental constraints built a brand that commands premium pricing.

Not every structure survives in every market. Some commitment devices only work with certain capital structures, or in markets where trust is differentially valuable, or with customers who can recognize quality. That’s precisely why governance, capital philosophy, and culture matter. The economics of the endeavor are not given. They are chosen—by founders, by investors, by the people who set the constraints within which everyone else optimizes.

This reframes what it means to work on technology that matters.

The question is not whether you’re an engineer or an executive. The question is whether you’re operating at the level of institutional design going all the way from formulating aims to economic implementation, or only at the level of features. Both matter, but the institutional level in this broad sense is prior. It determines what the feature level can achieve.

An engineer who designs a brilliant recommendation system but ignores the incentive structure it operates within has ceded the most important decisions to someone else. An engineer who understands how funding models shape optimization targets, how metrics become attractors, how governance locks in or erodes mission is operating at the level where outcomes are actually determined.

Every investment thesis contains an implicit theory of what is of value. Every term sheet is a design document for what will be optimized. Every board seat carries influence over what gets optimized. These decisions ripple outward into mechanisms, and mechanisms ripple outward into the lives of everyone who uses what gets built.

The question is whether this influence gets exercised deliberately or by default.

Default means optimizing for what’s measurable, what’s familiar, what’s already worked. Default means engagement metrics and growth curves and the patterns that produced returns last cycle. Default means structural drift toward whatever the economics select for, regardless of stated intentions.

Deliberate means asking the three questions. It means applying the Transparent Choice Test and the Candid Aim Test. It means designing economic structures that can sustain aims worth pursuing, and locking those structures in before the drift begins.

At the present stage of AI development, this framework is not an abstract lens but a live description of the game being played. Frontier systems are expensive to build and are increasingly financed by capital invested on roughly the thesis that the technology will ultimately be applicable to almost every aspect of the economy. In that sense the capital is not buying a product so much as underwriting a general-purpose substrate and hoping to later discover the dominant interfaces through which it captures value. That option-like thesis makes the “aim” unusually plastic: the same underlying model can become tutor, co-worker, companion, bureaucrat, sales engine, surveillance layer, or propaganda instrument. Which of those becomes stable is a property of the institutions, financing models, and culture that wrap it.

Because the return story is “almost everything,” the equilibrium remains underdetermined. Selection pressure is already operating, but often through indirect channels: control of distribution, accumulation of compute, narratives that justify scale, the ability to externalize risk, and the power to set default contracts and defaults-of-use.

Different capital structures will stabilize different aims. Usage-based pricing tends to reward systems that are instrumentally useful; subscription models reward systems people would still choose under conditions of autonomy more than attention-economy models do; enterprise procurement tends to reward auditability and legible governance; state funding tends to reward control.

The degree of freedom is real precisely because the equilibrium has not yet fully formed. Decisions made at this pivotal moment by technologists, founders, and investors can still dramatically shape outcomes we care about.

The attention economy shows what happens when the substrate hardens and aims collapse into a single attractor. If revenue is selling predictable attention to advertisers, then maximizing attention capture becomes the stable aim, engagement proxies become the mechanism, and surveillance plus lock-in becomes the substrate that keeps the loop from breaking. The equilibrium is narrow and self-reinforcing, and the possibility space remains technologically available but economically unselected.

Frontier AI is not yet locked into that attractor, yet it could inherit it quickly if general-purpose models are primarily deployed as next-generation attention routers. If autonomy is the commitment, this is exactly the moment when financing, ownership, and governance choices matter most: before the underdetermined equilibria congeal into infrastructure and the aim becomes a ghost again.

What got built around radio in the twentieth century shaped what kind of citizens could emerge. What gets built around AI now will do the same, at greater scale and speed.

The window is still open. Which citizens are you building for?

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

This year I watched my four- and six-year-old flourish at Alpha School, where AI personalizes their learning and frees time for what machines cannot supply: curiosity, character, play. I saw AI systems produce new mathematics, reshape creative work, and hand ordinary people powers that once required armies of engineers.

And yet I’ve noticed something in myself I don’t much like. The readiness to accept an AI’s first draft of an email, a plan, a decision, because it’s faster, smoother, and good enough. An invisible waltz in which I take one step and then forget that I am no longer leading.

In many domains I’ve welcomed the flow and eagerly entered the new paradigm. In others I’ve pushed back: the forming of beliefs, the making of judgments, the hard labor of learning to think at all. For the deepest questions, what it means to live well, I’ve tried to refuse it entirely.

The more I talked with others this year, the more I understood this was not my private neurosis. Something is shifting. People feel it, even when they cannot name it.

At our seminars with Oxford, Aspen, Liberty Fund, St. John’s College, DeepMind, Palantir, and Microsoft, and at our AI for Truth-seeking Symposium with FIRE, a pattern declared itself: the people most anxious about preserving human judgment are often those building the systems. They see how capable these tools are becoming, and they are asking questions the rest of us would prefer to postpone.

At Anthropic’s headquarters in San Francisco, looking out over the city, co-founder Jack Clark told me he journals more than ever, writing down his thinking before consulting any AI. He knows the outputs are better with the machine. What he wants to know is whether he is still developing. This is the difference between effectiveness and judgment.

Technology that expands what is possible may narrow who is possible. But it need not.

Adam Smith understood both halves of this. You become a good thinker by doing the thinking, badly at first, then less badly. The practice is constitutive. This is the best antidote I know against becoming a Claude Boy. And the kind of people who develop through their own judgment are precisely the kind we need if we are to sustain a free society.

Twenty-five hundred years ago, Pericles stood before the mothers of Athenian dead and described what their sons died for: a city where citizens cultivate beauty and wisdom, debate openly before acting, and take responsibility for public life. A free society.

Since then, the principles of free societies have been transmitted through debates, texts, and institutions. We are entering an era where they must be embodied in code—or they will become inert. The philosophy-to-law pipelines of old have given way to philosophy-to-code.

Tocqueville saw the stakes: vigorous self-governance in which citizens grow through freedom or a softer servitude in which we gradually surrender it for comfort and convenience, becoming, in his phrase, “a flock of timid and industrious animals.” The choice is ours to make, while we still remember what it means to choose.

This is our civilizational moment. It requires people who can translate the principles of a free society into the actual systems and institutions being built.

The philosopher-builder

We named the philosopher-builder archetype in July: the kind of person who, like Benjamin Franklin, combines philosophical reflection with practical wisdom and builds institutions that embody their deepest convictions about human flourishing.

I expected this to resonate with a few hundred people. Instead, some of the most impressive founders I know sent notes saying they’d finally found language for what they were trying to do. Researchers said it changed how they thought about their careers. What I came to understand is that the division between “thinkers” and “doers” has impoverished both, and many people had been quietly waiting for permission to be whole.

The philosopher-builder is an answer to a transmission problem. Whereas the great reformers of old wrote pamphlets, today’s are writing code. They will give the principles of a free society technological expression: what autonomy looks like in API design, what decentralization means for architecture choices, what truth-seeking requires in model development.

This year Ivan Vendrov, Zoe Weinberg, Séb Krier, Jason Zhao, Alex Komoroske, Lisa Wehden, and Joel Lehman all spoke about their work through this lens. We published Summer and Winter reading lists drawing on recommendations from across our network. And we started to see a community coalesce around the conviction that philosophy and building aren’t separate activities—they’re the same activity, done with different hands.

What we built this year

Fourteen seminars, including with Oxford, Aspen Institute, Liberty Fund, St. John’s College, and Edge, bringing builders into conversation with Hayek and Polanyi on the use of knowledge in society, with Socrates and Mill on truth-seeking, with Smith on moral development.

On YouTube and other video channels, our interviews with 11 AI experts and entrepreneurs on their philosophical influences reached over a million subscribers. On Substack, our long-form essays on AI and philosophy reached over 17,000 readers.

And voices across our network produced pieces we keep returning to: Tyler Cowen on how AI will change what it is to be human, Jason Crawford’s Techno-Humanist Manifesto, Caitlin Morris on social tinkering, Gavin Leech’s “The Scaling Era” with Dwarkesh Patel, and Jack Clark’s “Silent Sirens, Flashing For Us All.”

We grew our fast grants program to 140 builders and researchers. Through initiatives including a $1M AI for Truth-seeking program, many projects like Campus, Szdat, Kanonic, and Authorship began to take shape.

We supported 16 fellows working at the intersection of philosophical depth and frontier AI, many of whom did so at Oxford HAI Lab. This led to 34 research papers, from “The Philosophic Turn for AI Agents” to “Full Stack AI Alignment” to “Martingale Scores.”

Through our incubation fellowship, Samuele Marro launched a new non-profit called the Institute for Decentralized AI. We backed 8 philosopher-builders to create new companies that take their ideas to world-changing scale. And together with IHS, we started funding AI tool use for scholars in philosophy and the humanities: researchers like Kevin Vallier and Seth Lazar integrating these tools into serious intellectual work.

What matters more than numbers is the community that emerged: people who share a conviction that the principles of a free society must be translated into the systems we’re building, and that the window for doing so is shorter than most people realize.

What I don’t know

The practices I’ve developed personally feel almost monastic: blank-page journaling before consulting AI, deliberately choosing the harder cognitive path, regular reading groups. Preserving the independence, force, and originality that remains to us seems to require a level of intentionality we often tell ourselves we don't have time for.

I believe the classical liberal tradition has resources for this moment. But translation is hard: principles like truth-seeking and autonomy don't map easily onto architecture choices and agent interfaces. And even when the direction is clear, building well requires judgment that can only come from practice. The ideas and the formation have to come together.

What I’ve come to believe is that this practical wisdom has to be developed in community: through seminars, collaborations, and shared building. And probably through intensive multi-month human formation, which will require new thinking.

The philosopher-builder isn’t a solitary figure. The archetype only works if there are institutions that cultivate it. That’s what we’re trying to build.

Looking forward

This year, Harry Law and I are writing a book on what it takes to be a free agent in the AI age. I’m going to work closely with Philipp Koralus on research at Oxford’s Human-Centered AI Lab, and deepen connections with collaborators at the top labs as things move fast.

We are bringing this community together in person. More gatherings, more informal meetups, especially in Austin. And we’re training the next generation of philosopher-builders through a new format we’ll announce soon.

We’re hiring and will share roles soon. If our mission resonates, get in touch.

The question I keep returning to is simple: What kind of people do we want to become in a world where thinking is optional? The answer will determine what we build.

The flame of a free society has been passed from debate to debate, text to text, institution to institution for 2,500 years. Now it needs to live in code.

To the donors, fellows, collaborators, and readers who made this year possible:

Thank you.

Brendan

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

“If I asked people what they wanted, they would have said faster horses.” The idiom, a widely circulated but likely apocryphal line attributed to Henry Ford, stresses the distance between our ability to picture the future and our ability to make it real. It reminds us that technologies loosen the constraints that shaped past expectations, that deeper shifts usually enact variations in kind as well as magnitude.

“Faster horses” is a shorthand for folk logic that seems bulletproof at the time but quaint in hindsight. Television as radio with pictures, film as photographed theater, early mobile phones as portable landlines, and the internet as a digital library were all kinds of faster horses. They tell us that big swings don’t often play well with existing categories, and that new language, heuristics, or classifications are often needed to make sense of them.

Today, many of those wondering about the downstream impact of thinking machines are on the lookout for AI that can function as a “remote drop-in worker.” This refers to a system that replaces a human employee, in essence, by doing roughly the same things under the same conditions. Here, the future appears as a more seamless version of the present rather than something that dramatically changes the shape of work.

The idea flows from the observation that the majority of jobs in the information economy revolve around making computers do what we want. Word processing, desk research, data analysis, creating presentations, running marketing campaigns and many other tasks are all the end product of keyboard strokes and cursor movements. This is why some long-time AI watchers reckon Claude Opus 4.5, especially its instantiation within Claude Code, can reasonably be described as an early realization of Artificial General Intelligence (AGI). The same might eventually be true of humanoid robots, especially given they can slot into existing infrastructure without costly redesign, but our focus here is solely on knowledge work.

As others have pointed out, the response to a common AGI litmus test (a system that can outperform humans in most economically valuable work) turns on what we categorize as “economically valuable work.” If we define that as “stuff done on a computer,” then it’s plausible that one day soon the models will cross that threshold (if Claude 4.5 Opus hasn’t already). And if a model can be said to be generally capable, then the remote drop-in worker shouldn’t be too far behind.

Whether a single model can do a job in isolation is a useful question to ask, but it doesn’t tell us much about how such systems, interacting with many people and agents of their own, might rearrange patterns of coordination and the shared assumptions that guide them. In some ways, the conservative bet is that the drop-in worker is a stronger account of our present than it is our future. Technologies that matter rarely honor the roles we assign them. If the future is anything like the past, the drop-in worker may prove to be a faster horse: a story that made sense before the true nature of the agent economy became visible.

The Wisdom of the Crowd

Traditional accounts of AGI development often describe the emergence of an isolated system capable of completing the vast majority of cognitive tasks, sometimes referred to as a “singleton”. An alternative scenario, now seriously considered by AI developers, imagines that capabilities may be manifested through the coordination of “sub-AGI individual agents” with complementary skills and affordances. This scenario concerns an ecology of semi-specialized agents whose combined behavior outstrips anything they could do alone (and tallies up to something that we could describe as AGI at a high enough level of abstraction).

You might have a code agent that builds, a negotiation agent that handles scheduling or purchasing, and a compliance agent that checks your work. On top of these sits a manager that breaks goals into subtasks and shunts each to the right agent for the job. We state an objective, the system spins up a network of agents, they pass data between them, and a synthesis function presents the output of the collective for review.

Imagine launching a new software feature. The drop-in worker functions like a high-speed freelancer insofar as it writes code, pauses to check for bugs, and writes documentation sequentially within a single stream. It is a linear acceleration of a human workflow. The agent ecology, however, behaves more like a stack of mini-organizations. When the objective is stated, an “architect agent” drafts the structure while a “red team agent” simultaneously attacks that design to find security flaws before a line of code is written. A “compliance agent” cross-references regional data laws in the background. These agents operate in parallel to create an adversarial loop where the output is the sum of many small interactions. The result is an ecosystem capable of the kind of concurrent processing that individual minds, biological or synthetic, may struggle to achieve by themselves.

But this is only a partial picture. The share of human-agent and agent-agent interactions in the economy will increase over time, with agents engaging in price negotiation, placing orders from one another, coordinating supply and demand, and even rating each other to assign trustworthiness scores.

In some ways, the patchwork AGI thesis is another episode in a long-running story about how intelligence behaves at scale. Markets outperform planners because knowledge never exists in concentrated or integrated forms, but as incomplete and contradictory perspectives dispersed across individuals. Hayek reminds us that “planning” happens all over the place through individual agents, which is why he distinguishes it from “economic planning” that deals with state-backed forms of enterprise management. The agent economy doesn’t represent a toss-up between planning or ad-hoc action but rather an older question about whether planning ought to emanate from within or from without.

Aristotle raised the question that still haunts proponents of collective intelligence: can the many, combining their partial virtues, outperform the excellent few? In Book Three of Politics, he writes:

“For it is possible that the many, though not individually good men, yet when they come together may be better, not individually but collectively, than those who are so, just as public dinners to which many contribute are better than those supplied at one man’s cost; for where there are many, each individual, it may be argued, has some portion of virtue and wisdom, and when they have come together, just as the multitude becomes a single man with many feet and many hands and many senses, so also it becomes one personality as regards the moral and intellectual faculties. This is why the general public is a better judge of the works of music and those of the poets, because different men can judge a different part of the performance, and all of them all of it.”

For Aristotle, groups become smarter when they successfully combine different aspects of competence into a single body. Consider a jury that sees people with different experiences and biases pool their judgment to reach a fairer conclusion than any juror might in isolation. Or England’s common law, where centuries of small decisions by judges produce a legal order with more adaptability than one made by decree.

The same is true of Wikipedia, peer review, or nimble companies. In each case, the quality of the outcome rests on a kind of distributed deliberation wherein perspectives clash, revise, correct, and eventually settle into a stable state. It echoes the Athenian assembly and the medieval disputatio, both of which treated the good we call judgment as the product of structured disagreement.

American writer Howard Rheingold coined the term “smart mobs” to describe groups of people who are able to organize and coordinate quickly through the use of mobile communication technologies like the internet or mobile phones. The term “mob” is deliberately ambivalent, a framing he uses because of its darker connotations (he explicitly notes mob mentality can be for good or ill).

Rheingold thought smart mobs worked because low-cost communication let individuals share context and act in concert without central control. These groups represented an idealized version of accelerated coordination built from minuscule signals that could be aggregated over huge numbers of agents. The mob framing reminds us that coordination capacity increases faster than deliberative capacity, and as a result, the key variable becomes governance of the communications substrate.

But mobs aren’t smart by default. There are a whole set of coordination problems that flow from distributed decision-making, from free-riding (where individuals benefit from a group’s effort without contributing to it) to information cascades (where people copy others’ choices even when their own judgment points elsewhere). Many of us recognize some of these problems when we spend too much time on social media. We see outrage spread through networks faster than facts, and know how easily a crowd can be steered by sentiment rather than the hard work of judgment.

We could say that groups become dumb when they fail to properly synthesize knowledge, and they become smart when divergence is preserved and integrated. Whether or not we benefit from the wisdom of the crowd often depends on the structures that keep the mob in check. Markets do this with prices and labs with peer review. Rheingold might say that smart mobs emerge when communication structure and incentives reward decentralized coordination rather than herd behavior.

Society of Mind

In the 1980s, the AI researcher Marvin Minsky wrote about what he called the “society of mind.” What he meant was that unified intelligence is a loose federation of smaller processes, each narrow, each fallible, yet together capable of producing something that looks like coherent thought given enough altitude. For Minsky, intelligence emerges from many mindless “agents” coordinated in special ways, with the mind employing something like a computational and explanatory strategy whose power is a product of messiness, cross-connection, coordination, and resolution.

Today’s AI models demonstrate unified intelligence at two levels: as a byproduct of statistical learning and in the way models are housed within larger constellations that we refer to as “systems.” Transformers likely generalize because they compress huge corpora into representations that let them improvise solutions on the fly. Intelligence is in some sense a property of compression plus scale, an analog of Aristotle’s crowd insofar as it concerns many partial signals integrated into a single effective whole. As for the systemization of models, we can view each as a constellation of individual expert functions like tooling or multi-modal functionality.

But why stop there? If we accept that AI, like all intelligence, benefits from the interactions between discrete units, it follows that its capability should also be treated as a property of a larger constellation in which many systems operate together. Variance creates productive tension as models surface alternative interpretations and explore distinct solution paths. When those paths are combined through orchestration layers, tool use, or various other kinds of agent frameworks, the result is a system that searches the problem space more effectively than a lone model. Once multi-model systems interact with one another — coordinating, passing intermediate results, or checking each other’s claims — a kind of higher-order intelligence bubbles up from the sum of interactions across several layers. More powerful models are great, but superior ecologies are better.

People each hold fragments of truth, most of it tacit and hard to articulate, which is why spontaneous order tends to get the better of best laid plans. Polanyi might remind us that a drop-in worker presumes that competence is formalizable into explicit tasks and checklists. His work tells us that competence lives partly in the realm of tacit knowledge, that “dropping in” workers will face the same context problems faced by central planners the world over.

Of course, recent progress in AI development does typically try to provide models with the context they need to work effectively (specifically through the use of reinforcement learning techniques to make models good at human work in human settings). We might also say that, even if multi-agent systems matter internally for model cohesion, their deployment could still take the form of a remote-worker analog. Things like permissions, accountability, compliance, budgeting, and change-management favor inserting agents into existing workflows as opposed to redesigning them from the ground up.

These objections are useful, but they don’t allow us to skirt the core problem with the “remote drop-in worker” metaphor: that it treats intelligence as solely a property of individuals. It presumes the unit of analysis is the solitary agent carrying out tasks one after another, when everything we know about complex work suggests otherwise. Real capability comes from the knots of relationships, feedback loops, constraints, and opportunities that bind us together. First within models, then as agent systems, then eventually as agent-agent systems. Collective intelligence is prefigured by how information moves through a system and how the residue of experience accumulates across many small decisions made by each of us.

The remote drop-in worker may prove to be a transitory moment at best and a category error at worst, one that treats AI as an incremental addition to familiar workflows rather than a force that will reshape the nature of those workflows. That is “faster horses” thinking. We’re projecting today’s limitations onto tomorrow’s world and overlooking the fact that new capabilities alter the constraints that make what happens today seem natural. More accurate accounts tend to lead somewhere else, in a form we often only recognize with the benefit of hindsight.

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.

And so, with Christmas fast approaching, we asked 12 more of the sharpest minds we know for their philosopher-builder recommendations for you to settle down with over the holidays.

From ‘how thinking emerges’ to ‘Universal Robots’ to ‘LLMs as the death of the author,’ here are 12 picks from top AI entrepreneurs and thinkers to kickstart your ideas for 2026.

Read on for reflections from each recommender + a bonus recommendation from our Editorial Lead, Harry Law.

1. The Invention of Science

by David Wootton

It’s a readable and staggeringly erudite work of intellectual history. I’ve read it once, listened to it once, and have embarked on a second reading.

Before the discovery of America, Wootton argues convincingly, Europeans lacked the concept of discovery. No language but Portuguese even had an adequate term. European intellectuals assumed that any new invention or idea must be a recovery of something previously known but lost or forgotten. (Never before reading this book had I realized just how weird the Aristotelean picture of our planet was and thus how disruptive the voyages of discovery were to its factual claims.)

On my second reading, I was struck by Wootton’s description of Diderot: “Diderot had one great advantage over us: graduating from the Sorbonne in 1732, he had been educated in the world of Aristotelian philosophy. He knew how shocking the destruction of that world had been, for he had experienced it at first hand. From a bird’s-eye view—the historian’s view—the Scientific Revolution is a long, slow process, beginning with Tycho Brahe and ending with Newton. But for the individuals caught up in it—for Galileo, Hooke, Boyle and their colleagues—it represents a series of sudden, urgent transformations.”

As we anticipate sudden, urgent transformations from the deployment of powerful AI, The Invention of Science offers a deep dive into a parallel heady, disruptive process.

,
Author, a columnist for Works in Progress, and an Abundance Institute fellow. She and Charles M. Mann will release a podcast series on the history of everyday technologies in early 2026.

2. Vehicles

by Valentino Braitenberg

Now over 40 years old and long predating modern AI, Vehicles challenges us to think about the relationship between behaviour and intelligence.

Through a series of thought experiments, Braitenberg shows us that many things that look like “thinking” can emerge from simple, deterministic rules. As we increasingly interact with machines that (look like they) think, having good mental models of where complex behaviour comes from will be an essential component of epistemic hygiene!

,
Co-Founder of Entrepreneurs First, and Chair of the Advanced Research + Invention Agency (ARIA).

3. The Knowledge Machine

by Michael Strevens

I am glad I read Michael Strevens’s The Knowledge Machine: How Irrationality Created Modern Science. I bristled at some parts of it, but I did appreciate its clarity.

Strevens writes in a stripped-down way about what makes modern science different: his “iron rule” that we’re supposed to bracket off things like beauty, morality, and metaphysics and let only publicly checkable evidence into our arguments.

I’m not persuaded that this is in any way “irrational” (hell, Hume did more damage with rationality than that), and I think we should all probably be rereading Karl Popper’s The Open Society and Its Enemies if we want the full-throated defense of liberal science. For how we actually build better decision- and knowledge-making systems at the institutional level, I’d also send people back to Montesquieu and The Federalist Papers, which treat checks, balances, and structured disagreement as epistemic tools as much as political ones. But The Knowledge Machine is a lucid, accessible, and genuinely interesting thought experiment. For anyone who cares about how we know what we know (especially technologists building new “knowledge machines” of their own), it’s well worth a read—ideally with you arguing with it in the margins.

,
President & CEO at the Foundation for Individual Rights and Expression (FIRE). Author of The Coddling of the American Mind.

4. The Listening Society

by Hanzi Freinacht

For transformative AI to help humans flourish, much depends upon the wisdom of those designing and deploying it. Yet most technologists are blindly running on obsolescent ideas while firmly believing they are not — captured by unifaceted ideologies about progress or power. The Listening Society proposes a philosophy of flourishing that can meet the complexity and nuance of the world. It’s one of the most impactful books I’ve ever read. Even if you don’t like it, at the very least it will provoke new thoughts.

Joel Lehman,
Former OpenAI co-team lead on openendeness. Author of Why Greatness Cannot be Planned. Cosmos Fellow.

5. What Is Political Philosophy?

by Leo Strauss

In What Is Political Philosophy?, Strauss highlights how modern thought has substituted key classical concepts with new ones - most notably replacing the classical idea of virtue with the modern notion of responsibility.

The implication of this shift is that we now frame our behavior and moral standards in ways that make them easier to satisfy, even as we erode the substance of the ideals they once represented, diminishing the demanding excellence at the core of virtue.

I would encourage our generation of technologists to examine our industry’s terminology in light of this provocation. When we invoke words like courage or freedom, what work are they doing for us - and do they still call us to excellence, or serve to comfort us with a veneer of virtue?

,
Founder and CEO at Plymouth Street. Former President of the Oxford Union.

6. R.U.R.

by Karel Čapek

Karel Čapek’s 1920 R.U.R., which introduced the word “robot” in its modern sense, has food for thought on almost every aspect of our current technological landscape. There’s a scientist who built robots because he wanted to play God, and his nephew, a founder, who just wanted to make money. There’s a General Manager who believes his product will liberate humans from the drudgery of work, and an engineer who loves his job. There’s a humanitarian who worries about the robots’ souls, while widespread infertility decimates the human population. In the background of it all, there’s the question: what is it, really, to be human? And what, if anything, is that humanity worth?

,
Professor of Philosophy at UT Austin. Writer of ChatGPT and the Meaning of Life.

7. Aspiration

by Agnes Callard

Aspiration is a thoughtful book, especially useful as an indirect and unintended commentary on much discourse about AI alignment. This discourse often assumes what Callard calls a “decision-theoretic” model of values as fixed and unchanging. Instead, Callard argues that value change, not just accidental but sought out, is a core part of how human values work.

(Rudolf also recommended Liberty by Isaiah Berlin, particularly the essays on J.S.Mill)

,
Co-founder of Workshop Labs, ML Engineer, and Writer at No Set Gauge.

8. A Pattern Language

by Christopher Alexander, Murray Silverstein, and Sara Ishikawa

A Pattern Language might be on the shelf of every architect, but it’s more broadly about understanding the conditions that allow human life to flourish.

How do ideas spread? What creates a sense of belonging? Why don’t people dance in the streets anymore? The authors identify a series of “patterns” (which also come across beautifully as a love letter to design details) that create places where people feel alive and connected.

Reading between the lines and beyond just sometimes outdated architecture, these patterns are responses to deep human needs - enabling shifts between privacy and spontaneous encounter, exploring the role of different “distances” of engagement. A vocabulary not just for thinking about physical spaces but about what we’re trying to create when we design for human thriving. This is less of a “read all at once” and more of a “thumb through like recipes” book for me.

(Caitlin also recommended Speculative Everything: Design, Fiction, and Social Dreaming by Anthony Dunne and Fiona Raby)

,
MIT Media Lab Researcher and Educator who designs technology for curiosity-led social learning. Cosmos Grantee.

9. The Conscious Mind

by David Chalmers

The hard problem of consciousness is easy to state: why is there something it is like to be you?

Neuroscience can explain which brain states correlate with which experiences. But the hard problem of consciousness comes from recognizing that correlation isn’t identity, as we can presumably have the correlated brain states without the corresponding conscious states. We can have brains without the stuff of experience (qualia).

In The Conscious Mind Chalmers works through this puzzle with famous clarity. The payoff for technologists is striking. It can help readers see how, for all we know, AI could be conscious even if the mind is immaterial.

,
Professor of Philosophy & Director of Research at the Institute for American Constitutional Thought and Leadership, University of Toledo.

10. Language Machines

by Leif Weatherby

Leif Weatherby’s Language Machines argues that AI is not agentic intelligence but the death of the author made concrete. Structuralist theorists like Jakobson and Lacan argued that language was a generative system of signs, which existed independently of the ground truths that it described. Weatherby marries their ideas to those of AI researchers such as Claude Shannon, Walter Pitts, and Warren McCulloch. The conclusion: LLMs are not individual intelligence or anything like it, but language as a system, made capable of speaking.

(Henry also recommended The Unaccountability Machine by Dan Davies)

,
SNF Agora Professor of International Affairs at Johns Hopkins University. Writer at Programmable Mutter.

11. The Beginning of Infinity

by David Deutsch

The Beginning of Infinity reshaped how I think about progress, knowledge, and what’s actually possible. He makes optimism rigorous.

Deutsch argues that knowledge creation can continually solve problems and transcend apparent limits, challenging us to reject fatalism about constraints we mistake for destiny.

Progress isn’t guaranteed, but it’s achievable. What drives it? Conjecture, criticism, and relentless error-correction, not certainty or dogma. The idea that humans are “universal explainers” capable of understanding anything in principle feels both humbling and empowering; we’re limited only by what we haven’t yet figured out.

Yet his most useful insight centres on explanation itself: good explanations are hard to vary while still accounting for what we observe, like a key cut to fit one lock. Change one element and the explanation collapses. The clarity of distinguishing real understanding from mere description or correlation matters because it shows us where to focus. We’re building knowledge that compounds.

,
Founder of Exponential View and investor, author of The Exponential Age.

12. How the World Became Rich

by Mark Koyama and Jared Rubin

The tech world has rediscovered big questions about growth: why did the industrial revolution happen where it did, what role did states play, and why do some transformations stick while others don’t?

How The World Became Rich explores the competing explanations without forcing a single answer — a useful corrective for anyone whose model of progress starts and ends in the lab, or the state.

,
Frontier Policy Development Lead at Google DeepMind.

Bonus: The Turing Test Argument

by Bernardo Gonçalves

In The Turing Test Argument Gonçalves explores one of the most well known, and most misunderstood, episodes in the history and philosophy of AI: Alan Turing’s imitation game.

This little book argues that the test was a response to swirling controversy surrounding Turing’s debates with physicist Douglas Hartree, chemist and philosopher Michael Polanyi, and neurosurgeon Geoffrey Jefferson.

Gonçalves thinks that Turing’s focus on learning and adaptability sought to counter Hartree’s view of computers as calculation engines, that tasks like the composition of poetry aimed to address Jefferson’s demands for creative abilities, and that the introduction of open-ended conversation (rather than the use of rule-based games like chess) was designed to confront Polanyi’s concerns that human knowledge couldn’t be formalised by machines.

It’s a short but essential read for anyone interested in deeply understanding a moment that never seems to stop making headlines!

,
Cosmos Editorial Lead, former DeepMind Policy Research, and Cambridge University Researcher.

Thanks to Virginia, Matt, Greg, Joel, Lisa, Harvey, Rudolf, Caitlin, Kevin, Henry, Azeem, Séb, and Harry for contributing to this list.

And let us know your book recommendations for the holidays in the comments!

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.