Pacing and the Peril of Neutrality

Be careful what you wish for

Juan Antonio Ribera, Cincinnatus Abandons the Plough to Dictate Laws to Rome (1806)

In July, more than 1,300 employees of the world’s leading AI companies signed an open letter called Pacing the Frontier. The letter argued that frontier labs may be approaching the point at which AI research could be automated, potentially allowing capabilities to accelerate at a rate “beyond our ability to understand or control the resulting systems.”

Arguing that labs are incentivized to race ahead recklessly, the signatories asked the US government to support an international effort to develop the technical and governance tools that might be needed to deliberately slow the “pace” of frontier AI development.

As well as its size, the pacing coalition is notable for its breadth. Many of its signatories have diverging intuitions about openness, the role of state power, and the nature and extent of risk from AI.

One signatory, Dario Amodei, has since published an essay on pacing the frontier, arguing for embedded external evaluators at frontier labs, coordination between democracies, and international agreements. This immediately attracted the endorsement of Elon Musk and Sam Altman.

Other Pacing the Frontier signatories include Dawn Song, a long-standing advocate of decentralization. There is also John Schulman, who talks about the “possible need” for such a mechanism, and Ilya Sutskever, who argues that future AI will be so powerful that we will need “unprecedented measures.” Others cite issues such as social adaptation, institutional preparedness, the “liberties of the free world,” or just the value of buying time.

These concerns are not mutually exclusive, but they imply very different ideas about what a pacing mechanism should look like in practice.

A signatory who is primarily worried about existential risk from runaway recursive self-improvement is likely to tolerate significantly more intrusive controls than one who is motivated by a desire to preserve optionality or mitigate labor market disruption. Similarly, someone focused on American national security may be skeptical of international cooperation that risks eroding America’s strategic edge.

All of these positions can coexist under the injunction to “build the capacity to pace.”

But this stores up the problem. While the coalition can agree on the need to create a capacity, they don’t seem to have a shared view on who should ultimately exercise it or to what end. But any attempt to separate these two issues runs into uncomfortable questions about power.

Retreating to safer ground

In 1929, the German philosopher Carl Schmitt wrote an essay called “The Age of Neutralizations and Depoliticizations.” Schmitt argued that throughout European intellectual history, when conflict became too intense in one domain, people looked for another domain where agreement might be easier.

For example, people incapable of reaching agreement about religion might be able to find common premises if the question was translated from theology into the language of nature, reason, or law.

This, Schmitt argued, is why the intellectual center of European life shifted from theology to metaphysics, then to humanitarian morality, economics, and eventually technology.

This isn’t necessarily bad from a liberal point of view. A viable plural society can’t require agreement on ends before people are able to cooperate. Liberalism’s genius lies in its ability to find procedures and institutions that allow people with incompatible convictions to live alongside each other.

Schmitt, however, argued that neutrality degrades as “the newly won neutral domain … become[s] immediately another arena of struggle, once again necessitating the search for a new neutral domain.” In Schmitt’s view conflicts persist because human groups form around incompatible ways of life, with politics reappearing when those distinctions become existentially important. Neutralization can delay it, but the same tensions always resurface. For example, wars of religion gave way to larger wars of nation-states and economic competition.

Technology may seem to offer the ultimate neutral ground. A machine or a scientific technique can be used by people with completely different moral or political objectives. Meanwhile, technological questions appear factual and soluble in a way moral and theological ones never did.

For Schmitt, however, technology could never provide the neutral settlement people wanted from it. Technology is a supplier of means, not ends. Thus, “every strong politics” will attempt to master it.

What’s in a name?

Many of the most important questions about AI governance are contested. When is the state justified in restricting technological development? How much risk is required to justify such an intervention and how do we measure it? How should governments weigh domestic safety against strategic competition? How much weight should policymakers place on the most extreme scenarios of recursive self-improvement?

Some of these questions could be resolved with more evidence. But better evidence itself does not dictate what form intervention should take. This isn’t an argument about how “AGI-pilled” you are: people with the same beliefs about model capabilities (including those with nontrivial fears of catastrophic outcomes) could reach radically different answers.

The language of “pacing” suggests that we can build the capacity now and settle the politics of how to use it later. But a capacity to compel restraint must have an object, a trigger, and means of enforcement. Giving them institutional forms forces us to make judgments about the nature of the risk and what constitutes legitimate coercion.

First, someone has to decide what is being paced.

The progress of frontier capabilities is shaped by a combination of compute, algorithms, training data, post-training techniques, institution design, and so on. Whichever combination a mechanism attempts to limit would entail a judgment about the origins of any risk.

Consider the signatories’ concern about “loss of control.” Losing control is an outcome, not a threat model. It could describe a model deceiving the humans overseeing it, agents finding unintended ways of coordinating, or systems exploiting unknown access to external infrastructure. Under some definitions, it could even mean an institution delegating decision-making to the point where it can no longer supervise it effectively. Saying that you are trying to avoid “loss of control” doesn’t actually provide a guide to action.

Second, someone would need to determine when intervention is justified. What evidence means that the government should stop an otherwise lawful experiment? What kind of action would be taken, and how should we judge its proportionality? When should we be biased in favor of experimentation and when should we embrace a version of the precautionary principle?

Finally, any pacing regime would need a model of enforcement. If compliance requires access to internal experiments or model weights, choices must be made about surveillance and disclosure. For example, if there are exemptions for work focused on national security, that would implicitly privilege strategic advantage over safety.

If we look at these three questions, Amodei’s pacing proposals are incredibly uneven. He offers concrete ideas around monitoring and verification, with significant word count expended on the access and independence of the external evaluators. But he never specifies what is actually being paced, the conditions that would trigger restraint, and the consequences for crossing such a threshold. For example, one of his proposals is simply: “Some kind of ‘speed limit’ on the rate of recursive self-improvement.”

A verification regime can tell us whether or not someone has complied with the rules; it does not tell us what the rules should be. Amodei has continued to postpone the harder choices the pro-pacing coalition would rather leave murky.

Of course, that isn’t to say that these questions are unanswerable. It is possible that fresh evidence, research, and robust debate could lead the frontier labs and others in the field to a broad consensus. This consensus could then be translated into a set of concrete proposals. But we are not even close to approaching this point.

The state of exception

It would be easy to dismiss this as semantics. After all, many successful political coalitions have been built around constructive ambiguity. But vagueness is dangerous when the issue people are dancing around is state capacity. If a broad coalition agrees that the government should build the capacity to “pace” AI, without agreeing how or under what conditions, the unresolved questions are deferred to whoever ends up exercising that power.

If you believe that AI can become dangerous with little warning and that labs lack transparency then you will likely believe that anything other than tough, intrusive measures will be ignored or circumvented thanks to the power of incentives. This means that the government would need to take a hands-on approach to both regulating and monitoring.

These measures could include visibility into training runs and internal evaluations, control over access to compute and advanced chips, powers to compel disclosure, and authority to enforce a freeze. These powers would need to be reinforced by mechanisms to detect and halt jurisdictional arbitrage, and we would need a policy for what happens when a strategic rival refuses to play the pacing game. In essence, we are preserving the ability to act when ordinary arrangements no longer suffice.

Schmitt famously proposed that “sovereign is he who decides on the exception.” That is, rules presuppose normal conditions. So in exceptional cases, their application becomes uncertain, and someone has to assume extraordinary authority.

When a regime is justified by the need to act rapidly against poorly specified extreme risks, it will by definition involve setting aside normal conditions. This will mean ever greater executive discretion.

A pacing regime would have to act on incomplete evidence before a decisive risk of catastrophe had been identified (otherwise it would have been caught by the labs’ existing procedures). For example, officials might have to determine whether anomalous evaluation results represent alarming capabilities, or whether evidence about one model warrants restricting another.

There is also a risk that this regime would become ever more secretive. Labs would want to guard their commercial edge, and there would be concerns that even descriptions of cyber or biological vulnerabilities could be dangerous if made public. Moreover, evidence about foreign actors may come from intelligence sources whose provenance cannot be made public. Whoever is deciding that situations are “exceptional” may also control access to the very evidence that others would need to challenge their decision-making.

There is a related problem of duration. Schmitt distinguished between a short-term “commissarial” dictatorship, whose primary function was to restore the previous order, and a sovereign dictatorship that paved the way for a new one.

Pacing advocates don’t all have the same regime in mind. Some hope for a temporary setup to buy time. But based on their public statements, there are many who are (not particularly secretly) hoping for something more sweeping.

For example, restrictions may be introduced on account of a specific threat that emerged in large training runs or evaluations. If it subsequently became possible to produce the same threat through post-training, inference-time compute, or agent scaffolding, the authority would need to expand its mandate. It’s easy to see how a series of individually reasonable extensions could lead to the emergence of a frontier AI governance system by the backdoor, operated through executive fiat.

Of course, a chunk of the pacing coalition would be delighted at this outcome. But it’s easy to see how the powers could evolve in directions even they didn’t anticipate.

The agencies capable of monitoring compute, compelling disclosure, or controlling access to advanced technology already have important responsibilities, such as preserving military advantage, export control and industrial policy. These agencies may decide that they care about AI safety more than they do about strategic competition and act accordingly, but it’s a big gamble.

Amodei’s proposals note these tensions, but make no effort to resolve them. He conditions pacing agreements in democratic countries on maintaining a lead over China and even proposes using technology restrictions to widen this lead. But if China managed to erode this lead before labs had completed whatever safety work was necessary, the government would have to choose either attempting to slow China further or accepting greater risk. This simply introduces yet another unresolved political question.

Red lights on the dashboard

Once the government can alter the pace at which strategic technologies are developed and deployed, especially through the use of an ambiguous and potentially secretive process, it has acquired the power to decide who is permitted to move with the frontier.

Advocates for pacing may therefore convince governments of the exceptional character of frontier AI risk, only to lose all control over what happens next.

We’ve already seen an early glimpse of what this can look like. When Anthropic refused unrestricted military use of Claude, the Pentagon sought a federal ban, prohibited defense contractors from conducting unrelated commercial business with Anthropic, and designated the company a supply chain risk.

It is true that a pacing regime could create a series of transparent conditions and trigger points that make it harder for Leviathan to lash out arbitrarily in a fit of panic. But there are asymmetric incentives at work here.

Given that a pacing mechanism would be addressing potentially unknown danger, officialdom’s natural inclination is to set the threshold for intervention low. After all, you are much more likely to face consequences for allowing something bad to happen or for failing to respond to public panic than you are for slowing hypothetical economic or scientific progress. The history of America’s national security state shows that codifying powers does not necessarily constrain them. Anthropic managed to overturn its designation as a supply chain risk in the courts, but a future decision may be harder to contest.

In his writing on neutralization, Schmitt noted that the people who establish a new domain often do not become its political masters. Nearly a century ago, he could see that it was unlikely that a “politically dominant elite would develop out of the community of technical inventors.” The ability to forge a new domain or category does not imply the ability to shape its politics.

The need for politics

Carl Schmitt may seem an unlikely ally in the fight against unconstrained state power. After all, in Schmitt’s eyes, liberalism sought to replace politics with procedure, whereas he yearned for decisive political authority. This would sit comfortably with his subsequent unrepentant embrace of fascism.

But we can learn from his analysis while rejecting his political conclusions. Before handing governments potentially coercive powers, advocates should explain which powers they support, what would justify their use, what limits would be placed upon them, and how civil society could challenge them. They need to reject the comfort of proceduralism and rediscover politics.

They would almost certainly discover that they support an array of incompatible answers to these questions. But it is far better to put forward competing proposals and argue about their merits than to preserve apparent agreement until the government resolves them for you, in a way nobody intended.

This argument would establish what this coalition is actually capable of endorsing. If the apparent only extends to researching potential mechanisms, then it is important that the members of the coalition acknowledge this boundary. That means the supporters of more sweeping measures would have to defend these powers without being able to rely on the endorsement of people whose agreement is conditional on limits that haven’t yet been negotiated.

Advocates might argue that it’s unreasonable to demand a full institutional blueprint at this stage. But whether or not you would support a pacing regime is presumably dependent on questions like its scope or duration. Retroactively withdrawing endorsement is much harder than providing it.

Stepping up

It would be misleading to imply that no one in the pro-pacing camp is taking these questions seriously.

Several groups have started to engage with these questions. AI 2040 was an early attempt, with its idealistic vision of “total research transparency,” while Jasmine Li and Jason Hausenloy have published proposals on how compute verification could enable a coordinated global slowdown.

Yesterday, a group led by Raymond Douglas at ACS Research published their own pacing research agenda. The strength of this agenda, alongside its detailed list of potential intervention targets, is its admission that there isn’t a single “pacing program.” The measures you would support will diverge significantly depending on your underlying intuitions about AI. This is a step toward the political contestation that we badly need.

But these are small third-party efforts, while the institutions with the most political power and the greatest ability to implement these ideas at the frontier continue to duck these questions, in favor of vague calls for pacing.

With great responsibility comes great power. There may well be situations that present such grave risks as to justify extraordinary top-down intervention. But if the advocates for pacing with the most leverage want to empower the government, they need to describe the regime they actually have in mind to avoid indeterminate mission creep. These questions are too important to bury beneath the agreeable language of scientific cooperation or international coordination. When you turn to the federal government for salvation, be careful what you wish for.

---

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.