In a 1917 lecture, the German sociologist Max Weber told an audience of students that they were living through an era of “rationalization and intellectualization,” which was dominated by the conviction that “one can, in principle, master all things by calculation.” He cautioned that while experts could tell us how to pursue our goals, they could not tell us which ones to pursue. All ultimate values are irreconcilable, so “the individual has to decide which is God for him and which is the devil.”
This is what we might call a split between means and ends. In the Weberian analysis, the former can be reasoned about technically, while the latter are matters of private preference.
This division is supported by a particular conception of knowledge. The philosopher Hans-Georg Gadamer argued that the success of modern science had encouraged us to identify rational knowledge with “method.” This was his shorthand for procedures whose results can be established independently of the person applying them. He accepted method as a powerful tool for advancing scientific inquiry, but objected to making it the standard for knowledge of every kind. Gadamer believed that understanding a text, interpreting a tradition, or judging the right way to act requires us to determine how a general principle can be applied in a specific situation. Method is incapable of prescribing what to do in advance, because deciding what matters in any one case is a question of judgment.
The means-end distinction that Gadamer is challenging is at the root of a widespread, but impoverished account of autonomy. If you believe that reason is confined to working out how to achieve our ends and that our ends are a matter of preference, then autonomy can be reduced to the narrow question of the extent to which we truly own our choices.
This account of autonomy is currently the dominant one in the debate around AI deference. As we increasingly outsource everything from cognitively demanding labor to deliberation about deeply personal choices, to machines, there is a concern that we are losing our ability to act independently.
But much of this debate takes the split between the means and ends for granted. If our ends are merely a matter of private choice, then nobody can be better placed than us to decide them and the only way to lose our autonomy is to outsource choice. The most extreme manifestation of this view is to completely refuse to use AI at all. There’s already a thriving community of artists, writers, academics, and coders who proudly proclaim that they would never contaminate their craft with machine intelligence.
If you don’t use AI, you certainly won’t have to worry about AI deference. But just because you’re not deferring, are you really exercising judgment?
The absence of a method for determining what is good does not mean that our ends are beyond rational criticism and can therefore be reduced to preference. We can reason about what is worth pursuing and revise our understanding of what is important through experience. Autonomy cannot consist of merely choosing our ends if we lack the judgment to form, question, and revise them.
Against blind refusal
The assumption that we exercise our judgment by rejecting deference goes back at least as far as the Enlightenment. In “What is Enlightenment?” (1784) Kant defined man’s “self-incurred immaturity” as the inability to use one’s own understanding without the guidance of another. He warned against allowing books, doctors, or priests to think for us or to shape our consciences.
There is obviously much to admire in this approach: we shouldn’t simply accept something as true because someone with apparent authority tells us so. But Gadamer argued that the Enlightenment thinkers were blind to their “fundamental prejudice,” which was “the prejudice against prejudice itself.” He believed that we cannot approach any question without bringing our own assumptions, expectations, or inherited ways of understanding to it. While we may revise these in the light of new information, it is a mistake to believe that we can simply untether disciplined reason from the shackles of prejudice.
Gadamer believed that this misconception helped explain the Enlightenment belief that authority and reason were opposites. On this view, accepting the judgment of another often results in surrendering one’s own.
But why should it?
Recognizing that someone else understands a topic or how to act in a specific situation better than we do can be an exercise of reason. If an LLM points out an error in your arguments, it doesn’t indicate your judgment has been displaced. Instead, it means you’ve learned something – or, it should.
The mistake lies in fixating on this opposition. This is why Gadamer was equally unsparing about critics of the Enlightenment who rebelled by simply embracing tradition over reason.
Conversations around AI’s effect on our judgment or character development often end up reproducing this unhelpful binary. By focusing on whether or not we accept AI’s recommendations, we are stuck on judging the advice by its provenance rather than its merits. Refusing to listen to a machine, simply because it is a machine, might feel like an exercise of judgment. But it isn’t. You’re exercising the same level of judgment as someone who uncritically accepts its recommendations. The real test of autonomy is whether you can critically evaluate your reasons for acting.
This evaluation is critical because choice and judgment aren’t the same thing. Our own preferences can be distorted by desires and assumptions that we are unwilling to evaluate.
In Dependent Rational Animals, Alasdair MacIntyre argues that someone who has never learned to step back from their immediate wants may mistake them for reasoned judgments about their own good. What we believe is a considered commitment about the good may be “some unacknowledged form of infantile desire, a type of desire that has been protected from evaluative criticism.” The feeling that a decision is entirely our own offers no protection against self-deception. On the contrary, when we are unaided, “our confidence in our own judgments may always become a source of illusion.”
MacIntyre defines independence as the ability and willingness to evaluate the reasons others give us for acting. This makes us accountable for the conclusions we endorse, regardless of whether they originated with us. We are the ultimate authority on what we happen to desire, but we enjoy no corresponding authority over what is actually good for us. Sometimes a doctor, a teacher, or a friend – or an LLM – will be better placed to judge in a situation than we will be. This gives us a reason to take their advice seriously, although not an obligation to follow it.
Neither complying with nor refusing advice (AI or otherwise) by itself demonstrates that people are judging a particular situation well, acting appropriately, and taking responsibility. To assess this, we need an account of judgment.
Practical reason
The central problem for Gadamer was how something general, such as a text, a law, or a tradition, comes to be understood in light of particular situations. He found a model in Aristotle, who distinguished a special kind of knowledge called phronesis. Unlike episteme, the knowledge of what is necessarily true, or techne, technical knowledge or craft, phronesis is a practical wisdom focused on the conduct of life. Gadamer drew out a number of differences between technical or scientific knowledge and practical wisdom.
First, practical wisdom is inseparable from the act of living. While “we learn a techne and can also forget it … we do not learn moral knowledge, nor can we forget it … we are always already in the situation of having to act.” We cannot acquire practical wisdom in advance, but we develop it through the situations in which we have to exercise it.
The second difference concerns the relationship between means and ends. Technical knowledge is directed toward producing something, as opposed to what is the right or good thing to do. Of course, a craftsman might revise their design as they work as they run into challenges, but they don’t change their ultimate objectives. But practical reason concerns the rightness of our action. The specifics of the situation shape our understanding of the end we’re pursuing.
An AI model could participate in this process by challenging our assumptions. Over the course of the conversation, we might discover that our original objective was misconceived. An exchange could become part of a process that helps us to understand better what is worth pursuing.
Finally, the practical reasoner has some “skin in the game.” For Gadamer, moral knowledge belongs to someone facing a situation in which they have to act. Others can help us see what that situation requires, but “the person acting must himself know and decide, and he cannot let anything take this responsibility from him.”
For Gadamer, therefore, application is not a step that follows understanding, but something that “codetermines it as a whole from the beginning.” While our general understanding of what is right or good will help us judge particular situations, the experience of application can also change our understanding. For example, we might normally understand loyalty to a friend as requiring us to stand by them in the face of difficulty. But what if they’ve done something seriously wrong? Are we obliged to defend them, confront them, or to refuse to help?
In reaching our verdict, we may discover a more complex account of loyalty and friendship, which involves challenging a friend as well as supporting them. What we haven’t done is simply taken an abstract conception of “loyalty to a friend” and worked out the best means of deploying it. Instead, we’ve come to understand loyalty itself differently.
Gadamer illustrated phronesis with the example of a judge applying the law. A law states a general rule, but cannot anticipate all the circumstances in which it might be applied. Judges may use their discretion to temper the application of the law when its literal application may give rise to an injustice. In doing so, they are not abandoning the law but “finding the better law.” The just outcome emerges through the interpretation of the law in the light of a particular case. No amount of improvement in the drafting of the law could have eliminated the need for this judgment.
An AI model might prove gifted at this kind of reasoning, identifying considerations a human judge had overlooked and revising its interpretation of the law and its application in the light of the evidence. But practical reasoning requires that, however good the model’s judgment might be, the judge still be capable of evaluating its reasoning and taking responsibility for their final conclusion. It is neither total independence, nor total deference in the face of competence.
But even this ability to evaluate another’s reasoning does not, by itself, guarantee good judgment. Aristotle captured this distinction in his description of the deinos – a word that has a dual meaning of the “marvelous” and the “terrible.” The deinos possesses extraordinary ability and ingenuity and is capable of finding the way to achieve any objective, but lacks the virtue to distinguish between good and bad ends. In other words, practical wisdom requires more than cleverness or an ability to interrogate the output of models, but instead a cultivated understanding of what is good.
In search of virtue
Gadamer believed this capacity is developed through Bildung, or formation. We are all formed within a world of language, custom, and institutions. True formation consists not in blindly obeying the standards that are set for us, but in learning to see beyond our immediate concerns, encountering unfamiliar perspectives, and recognizing the limitations of our own understanding. Gadamer describes the experienced person as “radically undogmatic” as “every experience worthy of the name thwarts an expectation.”
MacIntyre similarly argues against mistaking formation for conformity. He believes that it draws on a living tradition, which is a “historically extended, socially embodied argument.” We inherit standards of judgment, but we also participate in questioning or revising them. For him, a good life is a quest. It’s not a search for something we can define in advance, like a miner searching for gold. Instead “a quest is always an education both as to the character of that which is sought and in self-knowledge.”
The practices and traditions that form us can be corrupted, which is why we need classical virtues like justice, courage, and truthfulness to sustain them. If a university is committed to truth, its members must be prepared to defend it against financial pressures or resist compromises in the name of political or social pressure. Professions can similarly only preserve their standards when their members are sometimes prepared to put them ahead of personal advantage. Of course, there isn’t a single list of virtues that everyone can agree on. MacIntyre observed that Homer, Sophocles, Aristotle, the New Testament and medieval thinkers “offer us different and incompatible lists of the virtues.” But he argues that justice, courage, and truthfulness are necessary because no standard of excellence can survive without them.
Here many influential liberal accounts of autonomy run into difficulty. For philosophers like Harry Frankfurt, Gerald Dworkin, and John Christman, autonomy is primarily concerned with the extent to which our desires and commitments are truly our own, rather than whether they are good. While these accounts provide ways of distinguishing between desires that we have reflectively endorsed or had imposed upon us, they tell us little about how we become capable of judging whether our commitments are worthwhile.
These somewhat procedural variants of liberalism risk becoming parasitic on the practices and traditions that sustain them. A functioning liberal society requires that its members feel obligations toward one another, beyond what the law commands, and that they will be capable of distinguishing between higher and lower goods. But these capacities don’t appear out of nowhere. Simply being free to choose or able to reason intelligently doesn’t guarantee that you will ever develop them. Instead, we cultivate them through inheritances, such as family, custom, place, and faith. These set standards for behavior that shape us, prior to us being able to explain or justify them.
A liberalism that treats ultimate ends as a matter of private preference struggles to rationally articulate why we should respect these inheritances. More importantly, it cannot sustain them. Instead, it spends down their accumulated moral and cultural capital, while simultaneously questioning the idea that we need it at all. It can protect our freedom to choose between different ways of life, but it cannot reproduce the conditions that allow us to make meaningful choices. Autonomy risks degenerating into the freedom to pursue whatever we happen to want.
If we return to the university, it might justify its function – the pursuit of truth – through the innovation it produces, the earnings of its graduates, or even its formation of autonomous individuals. But none of these fully explains why truth itself is worth pursuing. Even if you believe that teaching people to think for themselves is valuable, it doesn’t explain why they should prefer truth over comforting illusions or honesty over personal advantage. Whether a university allows, encourages, controls, or forbids AI use has little bearing on any of these fundamental questions.
The usual remedy for AI deference is to ensure the human being makes the final choice. But this belongs to the same thin account of autonomy. The inadequacy of preserving human choice will become even more apparent as agents become increasingly capable of acting on our behalf. They will pursue whatever goals we entrust to them, whether they are worthwhile or not.
AI has arrived in societies that are already struggling to sustain the conditions of good judgment. The institutions that helped us to acquire an understanding of substantive goods have been eroded by the decline of shared traditions and attachments, as well as a growing tendency to measure their value in terms of preference-satisfaction or material benefit.
Avoiding excessive deference to AI is important, but it is ultimately secondary to whether we can renew the virtues, traditions, and institutions required to form good choosers.
Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.
very cool !! shameless sharing this paper which you may find relevant: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5856382