Optimization and its Discontents
If the framework you followed brought you here…
Imagine two futures. In the first, a vast number of people lived rich, fulfilling, and varied lives. They have a range of temperaments, pursue their own aspirations, and find joy in different things. In the second, the same number of people live the same excellent life, but repeated identically, because an intelligent designer worked out which life scored higher on a wellbeing scale and made copies of it.
Most people would overwhelmingly prefer the first. But many consequentialists would view the second as at least as good, if not better. If total wellbeing is what matters, then the copies of the ideally-lived life have more of it.
William MacAskill has become troubled by this implication of wellbeing optimization. Having co-founded the effective altruist movement, which compares various courses of action in those terms, AI is starting to pose a conundrum. The total wellbeing framework recommends producing as many copies as possible of whichever digital mind scores “best” on a wellbeing metric. A future with trillions of copies of the same aggressively optimized life must logically be the best one.
In a new essay, MacAskill lays out the case for “saturationism,” a formula in which welfare is still the core driver of value, but stops accumulating once enough similar lives have been created. He highlights that “from a purely intellectual perspective, it’s probably the best idea [he’s] ever had.”
Unfortunately, it misunderstands why variety matters. More importantly, it preserves the mistake of the original framework: the assumption that the best future is something that a theorist can derive.
Experiments in Living
On one level, MacAskill is undoubtedly right. A future of varied lives is better than one of optimized carbon copies. But thinking about variety an end-in-itself gets things backwards. Rather, we should think of variety as the condition under which value itself comes into being.
We are formed by what we encounter. When we live among different kinds of people, face novel challenges, or engage in different types of work or recreation, we develop more than we would if we didn’t experience such variety. As the education pioneer Wilhelm von Humboldt wrote: “Even the most free and self-reliant of men is thwarted and hindered in his development by uniformity of position.” The development of a person’s powers requires not only freedom but a variety of situations.
John Stuart Mill, writing sixty years later, used a passage from Humboldt as the epigraph to On Liberty, just before the dedication to his wife Harriet Taylor. In On Liberty, Mill makes the case that we cannot discern what constitutes a good life solely by theorizing from the comfort of our armchairs. Mill believed that humans differ in temperament, capacities, along with what makes them flourish, and that no one can know in advance which way of life will suit them. A society that permits these “experiments in living” generates real knowledge over time about which lives are desirable.
This argument extends well from individuals to the structure of societies. The knowledge of what is worth doing, making, or wanting is not held by anyone. It is held in pieces – by artisans, traders, communities, and families – each of whom knows something about their own corner of the world that nobody else does. Institutions like the price system, the common law, scientific inquiry, language itself work because they let those pieces interact without anyone having to assemble them. The planner who tries to assemble them fails because this kind of knowledge cannot be assembled. It is not the kind of thing that can be written down and computed, because it exists only in the doing. Variety creates value through discovery.
In other words, true value is the product of people working out, in their own lives, what is worth caring about.
Coloring the Universe
Saturationism is essentially the polar opposite of this view of experience.
MacAskill imagines all the possible kinds of experience laid out as a map that can be measured, scored, and integrated. You might have joy in one region, awe in another, and the satisfaction of solving a problem in a third, and so on. The map has a fixed structure, which covers how similar two types of experience are, how much space they take up, and how much value they can contribute.
When a life exists, it lights up its corresponding region on the map. But the lighting saturates: the first lives in a region brighten it considerably, while additional copies of the same kind of life add less and less, with brightness eventually hitting a ceiling – in other words, the color hits saturation point. The best future is the one in which the lives that exist, taken together, light up the most of the map.
Coming up with a fixed map of experiences is an ambitious project. MacAskill is admirably candid when he admits that it’s not designed on the basis of any independent facts about the world. Instead, it is built and calibrated against the moral intuitions that in this case he and his collaborators (among which he lists ChatGPT, Claude and Gemini) endorse. MacAskill writes that it should be understood “just as mathematical structures that generate the axiology we want.”
The philosopher Karl Popper spent much of his career attacking this style of thinking, which he branded historicism. The historicist, in Popper’s account, believes that some sufficiently powerful intellectual operation can deliver answers about the trajectory of human affairs that would otherwise require the actual people involved to work out.
In his day, historicism was embodied in Marxist dialectics, but this critique could equally apply to long-termists wielding integral calculus. Popper warned that “if there is such a thing as growing human knowledge, then we cannot anticipate today what we shall know only tomorrow.” In other words, any theory that ranks possible futures has to make big assumptions about what those futures will contain – what people will discover, what they will come to value, or what forms of life will turn out to be worth living.
What gets bypassed is not just the lives in question but the process through which values get worked out at all. In real life, the question of what is worth wanting is settled by deliberation. People make their case, others object and propose alternatives. Positions are tested by argument, refined, and sometimes abandoned. The answer that emerges has standing because it survived this process. The formal apparatus replaces all of this with one philosopher and their collaborators’ settled intuitions, and presents the result as the answer rather than as one position in the argument.
The theorist is in no position to make those assumptions, because the people whose knowledge would validate them have not yet lived. With the right map, the right metric, the right function, the question of which futures are best is supposed to come out as a calculation. But when the map is drawn by a theorist and the metrics calibrated to their intuitions, the function will deliver the results they want.
In Search of Theory X
Saturationism is a product of the basic assumptions of population ethics. Popularized by Derek Parfit in the 1980s, population ethics studies how our actions affect who and how many people are born in the future. This essay belongs to the specific subdiscipline of population axiology, which compares different future population configurations. This, in essence, presupposes that populations are the kind of thing that can be ranked and that someone can do the ranking.
Population ethics is haunted by a hardcore variant of consequentialism, known as totalism. The totalists seek to maximize the total sum of net wellbeing across a population, regardless of how it is distributed. This disregards how ethics has been studied by the mainstream for centuries; even the early utilitarians would ask about the consequences of particular acts for actual people. In population ethics, people are swapped out for “value-bearers,” lives are “added” to populations, and “compared” across possible worlds.
MacAskill believes that the arguments for totalism are “fairly strong,” but that it leads to four conclusions that he cannot accept.
First, that a vast population of awful lives can outweigh a smaller population of blissful ones. Second, that any guaranteed good outcome can be beaten by a highly improbable gamble if the payoff is sufficiently astronomical. Third, that it breaks down in worlds with infinite populations, losing the ability to tell obviously different futures apart. Finally, it recommends filling the universe with as many identical copies of whichever life scores highest as possible.
The first three are old problems in the field, which Parfit himself acknowledged needed solving by a “Theory X”. The fourth is MacAskill’s own contribution. Ultimately, he sees the other three problems as the product of totalism letting one kind of value pile up without limit, so by solving the problem of variety, you can crack the other three.
On one reading, this is a heroic attempt to preserve as much of the consequentialist framework as possible while patching up its worst outputs. But as a rescue job, it doesn’t work. Totalism’s unsavoury outputs come from treating the value of a future as a quantity a theorist can compute from outside. If you keep the broad framework, but change the formula, you open as many new problems as you create.
You can see this in MacAskill’s own treatment of the negative side. Saturation caps how good a future can get by piling up identical lives, but it also has to handle how bad a future can get. If you cap suffering in the same way, a billion tortured lives could count for the same as a trillion. But if you leave it uncapped, then tiny probabilities of immense suffering swamp every calculation and reintroduce the wild gambles MacAskill was trying to avoid. There is no third option and, to his credit, he acknowledges this.
The Pretence of Knowledge
The problem with Saturationism is not that MacAskill made the wrong choices about parameters. Saturationism’s verdicts are an improvement on standard totalism, but top-down population theorizing will always lead to problems, because it has to compute answers to questions whose answers cannot be computed. Sometimes the baby does need to be thrown out with the bathwater.
It’s almost a truism to say that when a measure becomes a target, it ceases to be a good measure. There’s nothing wrong with comparing humanity’s different futures per se. Some are (much) better than others. But the variety that matters is the byproduct of free people developing in their own directions, under conditions that support this. Variety is not an end-in-itself.
Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.
So...
Measurement → distortion (“when a measure becomes a target…”)
That’s not always true.
Markets measure. Science measures. Games measure.
And those systems generate discovery, not kill it.
While I agree that: No one can compute the best future.
Shouldn't we compute the process by which people discover it?