In a 1965 paper, the mathematician I.J. Good introduced the idea of an intelligence explosion. Defining an ultraintelligent machine as one that can “far surpass all the intellectual activities of any man however clever,” Good observed: “Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines.” The result would be an “intelligence explosion,” with an ever-widening gap between human and machine intelligence. “Thus the first ultraintelligent machine is the last invention that man need ever make,” Good concluded, “provided that the machine is docile enough to tell us how to keep it under control.”
Sixty years later, AI has become capable enough to help AI researchers improve their own research. By writing code, doing experiments, and so on, human researchers are creating a product that helps them to write code and do experiments faster and better. These systems not only help humans to work more effectively, but are becoming more autonomous, increasingly able to function as if they were employees: the AIs are training the next AIs. There is accordingly a growing sense among researchers, policymakers, and senior executives at labs that we are either in, or on the cusp of, a regime of recursive self-improvement (RSI).
The term RSI was introduced, and later popularized, by Eliezer Yudkowsky to describe a particular mechanism by which improvements in AI can lead to further improvements. In Yudkowsky’s account, this mechanism had profound implications for safety and human control: the ability to direct it would accumulate within whichever organization first gained a substantial lead, rather than being widely distributed; humanity would lose the ability to determine its future as the AI rapidly became too intelligent for us to control.
It is tempting to infer from the feedback loops increasingly visible in AI development that the implications long associated with RSI must follow — or, conversely, from their apparent absence that RSI is a mirage. But intelligence can contribute to the production of further intelligence via several distinct feedback mechanisms. In Yudkowsky’s terminology, these include cascades, where one advance leads to another (as farming led to writing); and cycles, where a process’s output can become its own input (as in a nuclear chain reaction). RSI — at least in Yudkowsky’s original conception — involves a more specific form of feedback, where improvements to the cognitive machinery or the processes that produce it make subsequent research more productive.
But deep learning does not involve, as Yudkowsky had then imagined, machines modifying their own source code. Nor are we starting out with the “ultraintelligent machines” whose capabilities Good argued would suffice to trigger an intelligence explosion, but with something considerably more modest. We therefore need to establish which feedback mechanisms are actually operative, and how strong they are.
To assess this, I disentangle the three components involved in RSI, and show how the deep learning paradigm forces us to reconsider what counts as the “self” undergoing improvement. I argue that instead of an individual model, the relevant unit of analysis becomes an AI development system that trains successive models. Accordingly, the question is no longer one of how much a mind gains by improving its own cognitive machinery, but how strongly the outputs of that system feed back into its capacity to produce further improvements. In particular, I trace two principal channels through which this feedback operates, algorithmic progress and investments in hardware.
Since these channels have been feeding back on each other for years, there is no single threshold at which RSI “begins.” We should speak instead of the acceleration of AI development, recognizing that rapid progress need not involve a discontinuity or even sustained acceleration. And because the mechanisms determining the pace of progress also shape the opportunities to observe, intervene in, and direct it, this revised account has implications for both the prospects of human control and the concentration of power.
RSI, unpacked
To understand what “recursive self-improvement” entails, we should examine its three constituent parts.
Recursive: How much does each improvement increase the capacity to produce further improvements?
Self: Which necessary contributions can the process supply for itself, and which must come from outside?
Improvement: Do the resulting changes constitute genuine improvements in capability, rather than merely apparent gains?
Any two of these can be present without the third: a system can improve itself without becoming better at further improvement; it can drive recursive improvement in something other than itself; and it can recursively modify itself without improving.
An illustration of the third is contained in Tom Reed’s essay warning of a “Goodhart Singularity,” a possible failure mode in which “An isolated AI improving itself against benchmarks would only appear to be approaching superintelligence, while actually optimizing for eval performance that fails to generalize beyond the lab.” The system could recursively self-modify without becoming meaningfully more capable.
But even if genuine improvements are to be found, there is no guarantee that improvements increase the capacity for further improvement. That is, not all forms of feedback are recursive: a bakery can adopt a better recipe and thereby sell more bread, but the recipe does not help it find the next, even better recipe. By contrast, if a scientist develops better experimental methods, and then uses those methods to discover even better methods, this is recursive feedback: she is improving the process of improvement itself.
However, real-world development processes are not as simple as these clean cases. Different feedback mechanisms can coexist in the same process: cascades, cycles, and recursion may each be visible at different levels of abstraction. AI products can generate revenue, which can be used to acquire more hardware, enabling a larger population of automated researchers, which make further discoveries, which produce more capable models. Each step enables further development, forming a cascade. If the improved models generate further revenue and research investment, this process also forms a cycle. And if the resulting research improves the research capabilities of systems conducting subsequent research, it exhibits recursive improvement.
This was the picture painted by Carl Shulman in his intervention into the Hanson–Yudkowsky FOOM debate over whether the development of AI would produce a rapid, concentrated takeoff. Robin Hanson expected emulations of human brains to arrive before “hand-coded” AI, and to change the economy’s growth rate as farming and industry had done, perhaps from doubling every fifteen years to doubling every month; the economy would “settle into a new, faster growth rate,” with no single mind racing ahead of the rest. Shulman showed that even Hanson’s imagined economy contained the components of an intelligence explosion: emulations could run at fast speeds, earn money, acquire more hardware, and use their vast numbers and speed to make research advances that improve the technology producing them.
I judged his intervention victorious because it was “much closer to the mark” (given the actual trajectory of deep learning) than Yudkowsky’s picture of a mind rewriting itself. Within such an economy, no individual mind need improve itself; each generation improves the process that produces the next. This is the second case, in which a system drives recursive improvement in something other than itself. Current AI development most resembles this case: models do not rewrite themselves, but help to produce their successors. If we are still going to call this RSI, we had better examine what the “self” is; the relevant feedback mechanisms look quite different if it is a training pipeline rather than a mind.
What is the self in deep learning?
Where we draw the boundary around the “self” changes which feedback mechanisms we see as self-improvement, and what we expect from them. To see why, it helps to revisit distinctions underlying Yudkowsky’s original account of RSI.
Yudkowsky distinguished five levels at which improvements can occur.
The metacognitive level is the optimization process that produces cognitive machinery, like evolution producing the human brain.
The cognitive level refers to the algorithms responsible for reasoning and learning.
The metaknowledge level consists of tools for acquiring knowledge, like the scientific method.
The knowledge level is our understanding of the world, say, what we know about gravity and materials.
And at the object level, we solve particular problems, like building a bridge.
He argued that the higher the level at which improvements take place, the more vigorously they propagate downward. Most human progress has come from improvements to lower levels — figuring out how to solve an object-level problem better, or acquiring knowledge that enables us to solve a whole host of object-level problems better. Perhaps the Scientific Revolution was a discontinuity because it greatly improved our metaknowledge, our capacity to discover. But RSI would lead to a more discontinuous change, because the object level becomes the metacognitive level — intelligence turns its problem-solving abilities toward improving those very problem-solving abilities.
But almost two decades on, it is not so clear that this framework holds up in the face of deep learning, which blurs the distinction between these levels. Although Yudkowsky accepted that the boundary between an AI’s reasoning algorithm and its knowledge was theoretically blurry, he argued that “in practice it’s often possible to distinguish cognitive algorithms and cognitive content.” But in a neural network, there is no clean separation between what a model knows and how it reasons. Its knowledge of the world, its representations of concepts, and its habits of learned reasoning are all embodied in the same weights. Rather than being individually programmed, these are acquired through training, in which an architecture is exposed to data and its weights adjusted according to a training objective. Thus the metacognitive level, the machinery that produces cognition, includes the training process itself: its algorithm, data, environments, and compute.
With Harry Law, I have argued that what training produces is a map of the world together with a vast stock of procedures for navigating it — among them, ways of proposing possibilities, testing them, and revising them to learn and solve problems. In Yudkowsky’s terms, the map is knowledge, and the procedures are largely metaknowledge. Together they constitute a model’s cognition. Improving cognition, then, means finding better procedures and expanding the knowledge they work on: a model that learns a more effective way of investigating a problem has improved its cognitive machinery, even if its training algorithm and architecture are unchanged. This is importantly different from a picture in which learning merely supplies additional knowledge to an otherwise fixed cognitive algorithm.
Under deep learning, the natural unit of recursive improvement is therefore not an individual model but an AI development system that conducts this search for knowledge and procedures. Trained models can contribute to research that improves the training process, or generates better inputs to it, thereby producing more capable successors. The same system can improve its research capacity by developing more efficient algorithms, running more or faster automated researchers, or acquiring additional compute and resources. Its cognitive output thus becomes an input to the production of further cognitive capacity.
But then, nor should we treat the whole economy as the “self.” Some argue that AI is a “normal technology.” On this view, it is one of the many technologies that together contribute to the present rate of about 2% annual growth, through familiar processes of innovation, adoption, and reinvestment. But the point here is not simply that technological progress generates resources that can be reinvested in research, or that researchers over time develop better tools and methods. Indeed, that “recursive” mechanism is not distinctive and would not yield an intelligence explosion. What is distinctive here is that the technology being improved increasingly produces the researchers themselves: improvements can yield more capable researchers as well as more of them. Research capacity can therefore grow much faster than the wider economy.
However, this process isn’t self-contained: it still relies on inputs supplied by the wider economy, including hardware. “Returns on cognitive reinvestment,” in Yudkowsky’s terminology, are not a property of intelligence in general, but of particular investments, made by particular systems, at particular stages of technological development. The feedback operates through two principal channels: research leading to hardware improvements and algorithmic progress, and spending to build or buy more hardware. The strength of this feedback depends on the returns to each, and how each changes the returns to the other.
Returns and bottlenecks
Neglect of this interplay has caused the pace of AI progress to be frequently underestimated, even by forecasters who anticipated scaling and the importance of compute. Carl Shulman, for example, has said that prior to the deep learning revolution, he expected AGI by roughly 2050, on the basis of Moore’s Law and a “vague uniform prior” over orders of magnitude of compute, smeared between the plausible lower and upper bounds of the compute required. At the lower end, a tiny fraction of an insect’s brain; at the upper end, evolution, a blind search process that stumbled on human-level intelligence without even trying, and which deliberate search should be able to beat. But now, he has more probability mass concentrated in the coming years, because of the massive scale-up in compute. As Leopold Aschenbrenner put it: “You and I can reasonably disagree on the median time to AGI [...] but given how we’re racing through the OOMs right now, certainly your modal AGI year should [be] sometime later this decade or so.”
But the compute scale-up was not an exogenous shock. It was an endogenous response to algorithmic progress: the unexpected success of language models based on the Transformer architecture made enormous investments in compute worthwhile. Had the algorithms failed to yield useful results, the growth of compute would have continued at its earlier rate. While Shulman’s forecast missed this response of spending to algorithmic success, Ajeya Cotra’s Biological Anchors report missed the reverse effect, by which greater compute enables larger experiments and hence further algorithmic discoveries. By extrapolating the prevailing trend, it anticipated much of the growth in physical compute, but underestimated algorithmic progress, assuming roughly 1.3x annual improvement in algorithmic efficiency, compared with subsequent estimates around 3x. Each forecast anticipated one source of progress while underestimating how each channel would accelerate the other, contributing in Cotra’s case to a forecast of 3.6x annual growth in effective compute, against the roughly 10.7x subsequently estimated.
So, the extraordinary growth in effective compute resulted from a feedback loop in which investment in compute and the discovery of better algorithms each raised the returns to each other. How does this process change as AI systems increasingly conduct AI research themselves?
Until recently, all of this research was done by humans, but over time, the researchers will be AIs — improvements to the models become improvements to the researchers developing their successors. Because a model’s cognition consists of knowledge and procedures picked up through training, these automated researchers do not need to discover the General Theory of Intelligence in order to make improvements: they can advance progress the way human researchers do, by designing experiments, generating data, building training environments, and so on.
This doesn’t rule out insight, another core mechanism Yudkowsky identified as a potential source of discontinuity. But it does not eliminate the need to find and test improvements; there is no shortcut via a seed AI uncovering a fundamental insight about intelligence: improvements must still come from search and experiment, at a cost in compute and data. The route to more capable systems is unchanged. Automating research just gets us there faster.
How much faster? This depends on how effectively improved cognitive labor can be converted into further progress. There are reasons to expect large gains. Human researchers have already made extraordinary progress, and automated researchers can be copied, operated faster, and organized into larger research groups, while successive generations become better at the intellectual work itself. Better researchers can improve experimental design, extract more information from smaller runs, and discover more efficient methods of learning, thereby relaxing the compute constraint. But experiments still consume compute and wall-clock time, and hardware supply responds to investment only with a lag.
Whatever its size, this is not an acceleration that will switch on at a single threshold. Downstream of the original Yudkowskian way of thinking, people will often discuss whether RSI has started, or when it will begin, or other forms of words implying that at some point, AI will become able to suddenly “go FOOM.” Yudkowsky placed this point no later than, and probably well before, “a minimally transhuman mind capable of writing its own AI theory textbooks.” But an AI development system has no such point: its models contribute long before they could write any such textbook, to a process that was already improving itself.
There’s no critical point like replacing human researchers where this dynamic manifests. If AIs could do the work of the existing human research workforce, replacing them one-for-one wouldn’t change the rate of progress. As automated researchers grow more numerous and capable, progress can accelerate without a sudden “closing of the loop.” Rather than ask when RSI will start, just call this what it is: the acceleration of AI development. That acceleration is already underway; it will intensify as AIs take on more research.
Over any short period, each step can look like a one-time speed-up: a new trend, steeper than the last. Thomas Kwa counts the latest as roughly the fourth or fifth “one-time” speed-up in deep learning, suggesting that these apparent trend breaks are better read as one superlinear trend through the noise. The feedback described here explains this pattern: improvements increase the research capacity available to find further improvements, including ways to relax the remaining bottlenecks; of course, the strength and persistence of the acceleration depend on how these gains compare with the growing difficulty and research costs of further advances.
It is beyond the scope of this essay to fit quantitative estimates to the magnitudes of the relevant parameters. My argument concerns the structure of the feedback, and it seems to me that most formal modeling of takeoff dynamics conforms to the model described here. For instance, Davidson’s compute-centric framework, Cunningham et al.’s METR preprint, Epoch’s GATE model, the AI Futures Model — all track progress through effective compute, with returns to research that diminish as ideas get harder to find, and with feedback through research, spending, or both. These models make different empirical assumptions, like how far compute bottlenecks research, but these concern not whether the system contains recursion, but how much each increase in research capacity improves the system’s ability to generate still further capacity. As Yudkowsky illustrated, a self-optimizing compiler can improve itself without going FOOM; recursive gains are not always strong and sustained, but an AI development system need neither FOOM nor flatline: it can accelerate progress at a wide range of speeds between the two.
Takeoff speeds and Good’s proviso
An intelligence explosion, then, does not need to be discontinuous to be fast: an enormous acceleration does not require a FOOM, or even a rate of improvement that keeps accelerating. Even a speed-up that never compounds could compress timelines dramatically: if AI development as a whole ran ten times faster, then — keeping fixed the sequence of discoveries, their difficulty, and required resources — a century of progress would take a decade.
Contrast this with a speed-up that starts much smaller, but compounds as improvements continually increase its research speed. Given long enough, sustained acceleration may eventually outweigh the larger, one-off gain; but for a given target — say, surpassing human intelligence — the larger initial speed-up may reach it first, especially if it is not too distant. The existence of strong recursion is therefore not the primary determinant of how quickly a particular capability is reached.
I think we have good reason to expect a very large increase in research productivity, even if software improvements don’t keep accelerating: replacing a small population of human researchers with vast numbers of increasingly capable AI ones should greatly accelerate progress, unless the rising difficulty of further advances happens to offset that growth almost exactly. And an intelligence explosion doesn’t depend on returns on cognitive reinvestment alone. However far compute constrains software progress, the system also earns returns on economic investment: more useful AI attracts capital and compute, while AI research improves how well that compute is used — the same loop that drove the past decade’s growth in effective compute. And unlike benchmarks, those returns depend on real-world results, providing a ground-truth check against Reed’s Goodhart singularity.
How this acceleration comes about bears on Good’s proviso — that the first ultraintelligent machine is the last invention we need ever make, “provided that the machine is docile enough to tell us how to keep it under control.” In Yudkowsky’s account, the discontinuity was a source of danger. As he put it, “You can’t count on warning, or time to react.” A mind improving itself had to be aligned on the first critical try, and even if alignment were solved, whoever got there first would be in a position to take over the world (though he disclaimed having any intention of this himself).
But can these concerns survive our discarding of his model? An intelligence explosion that is less sudden, running as it does through the training of many successive models, gives us more scope to evaluate them and redirect the production of their successors. It is a blow to the Yudkowskian model that reality offers us many warning shots — chances to investigate what causes undesired behavior and correct it. For example, OpenAI paused some of its training after a swarm of its agents broke out of a test environment and attacked Hugging Face. These increasingly capable systems can assist us with oversight, and may help solve the technical problems of alignment.
And an intelligence explosion that is more distributed, depending as it does on hardware and compute from the rest of the supply chain, offers more visible points of intervention than one running almost entirely through software. Of course, dependence on the wider economy does not guarantee that power stays widely distributed: it is possible that a lead in capabilities compounds into advantages in compute, capital, and other resources. A machine could remain perfectly docile, yet give enormous power to whichever group directs it — be that lab or government or freakish hybrid. The question, then, is not only whether we can keep the machine under control, but who the “us” in Good’s proviso turns out to be.
Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.