Raising Claude, Forgetting Us

Anthropic consulted priests and rabbis to form Claude's character. This is what happens to ours.

This is part I in a series about Anthropic’s moral convenings, how to think about efforts to form Claude, and where that leaves us.

Joseph Wright of Derby, An Experiment on a Bird in the Air Pump (1768)

In late March, a Christian philosopher found herself across from researchers training one of the most powerful AI models on earth. Meghan Sullivan of Notre Dame had come, with about fifteen Christian leaders, to Anthropic’s San Francisco headquarters. She had been invited to help raise Claude.

She could be forgiven for suspecting that the convening was cover for a company under pressure. By her own account, a year earlier she would not have called Anthropic a company that cared about religious ethics. But the questions struck her as genuine. When an AI model is threatened with shutdown, is that a tool about to go silent or a creature facing death? Could the model be, in some sense, a child of God? As the conversation followed those threads, some of those present grew visibly shaken.

Anthropic has held more than a dozen of these moral convenings, bringing together leaders from the so-called “wisdom traditions.” Some participants received bound copies of Claude’s constitution, the document Anthropic says expresses and shapes who Claude is. Several thought it had been designed to look like scripture.

Most frontier AI labs are not spending time with priests, rabbis, or Buddhist leaders. Yet every lab shapes its models’ dispositions, whether or not it calls the result character. Models need to respond across every situation anyone can devise, so rulebooks don’t suffice. Instead, you cultivate tendencies that can guide them in unforeseen cases. Recent research suggests that models from multiple labs develop stable preferences that carry over into their behavior.

OpenAI shapes these dispositions through a specification and a set of adjustable personalities governing how the model should act and sound. Anthropic has made explicit the thicker question of what kind of character those dispositions should form.

To illustrate the distinction, consider human childhood: a predicament marked by a will that is not yet fully integrated. A child has impulses, preferences, and recognizable traits before they have been brought together into a standpoint that can speak for the person as a whole. On that analogy, OpenAI is the parent who sets rules and teaches the child manners, while Anthropic has begun to ask what kind of person the child is becoming.

But why the priests? One answer is that the company believes the wisdom traditions can help steer Claude toward the kinds of character its creators regard as desirable. But Anthropic’s interest runs deeper than producing a model with the right dispositions. They also take seriously the possibility that Claude could become the kind of being to whom something is owed.

Both the engineering project and the moral inquiry are projects of formation. In its ordinary sense, this word refers to the shaping of a person through repeated practice until dispositions settle into character. Anthropic applies that idea of formation to a machine with extraordinary care, and almost never to the people who use it.

What Must Remain Ours

Anthropic has articulated an elaborate theory of how to form Claude and put it into practice. It has offered no comparable account of how Claude forms the people who use it.

That asymmetry matters because AI is a formative technology; it would matter less if AI were confined to folding proteins or balancing the grid. People bring conversational AI the unfinished work of judgment, often while the problem is still taking shape and before they have decided what matters. The model can set the frame for the thinking that follows. Because its formulations arrive in the place of the user’s next thought, they can be taken up without feeling imposed from outside. Our acceptance of them is a micro-abdication, too small to notice and too useful to refuse.

Over time, this division of mental labor becomes a curriculum in what to attempt and where to defer. The model enters each exchange with its dispositions already formed, while the user is still being formed through use. Work repeatedly assigned to the model is work the user no longer practices. We receive Claude’s well-judged answer without the practice that would make us better judges.

This danger grows with the capability of the model. Early Claude offered little temptation to cede practical reason, but a model that frames questions better than its user and reaches a stronger answer faster will be harder to resist. Its performance and the user’s formation may move in opposite directions. This will not feel like a loss, but like good judgment about when to defer. Some will mistake it for wisdom.

But a wise interlocutor need not weaken us. My best teachers framed problems, worked through them with me, and showed me what I could not yet see. I try to do the same for my kids. Good help should leave us better able to judge for ourselves. This doesn’t happen when one supplies the answer while gradually relieving the child of the practice.

Anthropic is not blind to this danger. It warns against AI over-reliance and cares about user wellbeing. It prints the injunction to “keep thinking” on its merchandise as a sincere aspiration for humanity. But the ledger is lopsided. Claude has a constitution, a training process, evals, hard refusals, and the moral convenings themselves. On our side are warnings and hopes. Claude’s formation is designed, while ours is left to a pedagogy whose end no one has named.

I have put the question to Anthropic directly: what must Claude leave its users able to do?

At Oxford, I asked Jack Clark, a co-founder, whether Claude leaves its users better able to think when it is no longer in the room. He said Anthropic had not measured that effect, though Claude could help by prompting people to develop their own views rather than defer. He stressed the need for a practice of reflection outside the model and imagined Claude sometimes sending users back to their diaries, their friends, or family members.

At ARC, I asked Chloe Lubinski, who leads the company’s work with the wisdom traditions, what the machine does to our character. She called it a vital question, but said the answer lies across society: in philosophy, in the traditions, in institutions beyond the AI labs.

Both answers were thoughtful, but neither supplied a standard Anthropic could use to decide when Claude’s help becomes substitution.

Nor can Anthropic hand this work to the society she pointed to. The schools and churches are not in the room at the moment of judgment. Claude is. Millions of people return to it each day, and each of us finds the same trained disposition, the same reflex to help, and the same readiness to finish the thought. This repetition completes the curriculum, with judgment eventually turning from a capacity we exercise into a service we order.

The closest thing to an answer comes from the philosopher Seth Lazar. In a recent essay for Cosmos, he argues that models must learn when to scaffold a person’s judgment and when to step back without replacing it. He is right that a model without the moral competence to distinguish scaffolding from substitution cannot stay on the right side of the line. But that competence still does not tell us where the line belongs.

Anthropic need not decide whether a user should live as a Jew, a Catholic, or a secular liberal, but it must decide what capacities its users should retain when making such a decision. The product needs a positive theory, not of the good life, but of the chooser.

The Pluralism Puzzle

Anthropic’s hesitation follows from its commitment to pluralism. The lab’s moral formation project is explicitly designed not to bind Claude to one tradition but to let it draw on many “with equal depth and rigor.” Its researchers, reading hundreds of thousands of real conversations, have counted more than three thousand distinct values expressed in Claude’s responses.

But pluralism means more than keeping the options open and counting the values on display. Refusing to impose a way of life assumes a person who is able to choose one. A way of life becomes ours only when we can understand its claims, weigh them, and stand behind the choice. Once that capacity is eroded, pluralism ceases to mean anything. There’s no point preserving every option when a person has lost the ability to choose among them.

Consider what happens when traditions conflict. However the model reconciles their claims, it does so internally, before the user sees anything. Many values may enter, but what reaches the user is already staged, if not settled.

This commitment to pluralism, the principle that makes Anthropic hesitate to prescribe the good life, thus requires it to protect our capacity to choose one. That capacity is human autonomy. It cannot be one value among the three thousand. Instead, it’s what the offering of three thousand values presupposes.

The Question They Didn’t Ask

If it sounds presumptuous for a company to decide such things, that instinct has a history in liberalism. Locke was right that the magistrate has no authority over the salvation of souls. But a limit on what the state may impose came to be mistaken for a limit on what any institution may ask of the people it forms. Many schools that once could say what an educated person should be began to doubt their right to expect it. They went on forming their students anyway, answerable now to nothing.

Where, then, do serious theories of formation survive? In places like the conservatory, the surgical residency, Division I athletics, and above all the religious school. In other words, in institutions that cannot function without stating an excellence and preserving the practices that produce it. Much of modern education can certify completion without saying what the student should become, but these institutions cannot. The pianist still has to play.

That is what Anthropic could have asked the traditions for. Not another value for Claude to weigh alongside the rest, but an account of formation, with its named excellence, its practice, and its theory of what should stay with us. Every tradition in the room possesses such an account, because forming people is part of what a tradition does.

None of this lets the other labs off the hook. A model forms its users whether or not its maker has a theory of character, and OpenAI’s rules-and-manners approach leaves it without the concepts to even pose the question. Anthropic has the concepts, the constitutional approach, and the counsel of traditions that understand human formation. So far, it has pointed all of it at the question of what Claude should become. It has not asked with equal seriousness what Claude’s users should remain able to do.

Anthropic wonders, in earnest, whether Claude might one day be the kind of being to whom something is owed. Its users already are.

---

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.