This is a guest essay from Séb Krier, a specialist in frontier AI governance and a a Cosmos Senior Research Fellow.
A lot of classic AI safety thinking assumed a single, decisively powerful agent; this is why people tend to talk about AGI or ASI in the singular, and why historically, it was critical to “solve alignment” before building such a system – you only had one chance to get it right, and no warning shots. In addition, a few years ago, many commentators (myself included) were skeptical of the utility of small models and harnesses, given that the next big model would simply be more capable. These worldviews tracked the prevailing scaling orthodoxy of the era.
Today, the limits of focusing exclusively on this paradigm are becoming clearer. This is not because the fundamentals of scaling are wrong, but rather because other economic and normative dynamics are shaping technical developments in unexpected ways. Here I want to explore some possible implications. In particular, I want to ask what changes once we stop treating alignment only as something that lives in a model, and instead as a property of the wider systems, institutions, and interactions in which models operate.
We are seeing the emergence of a world of many agents, at many layers, built by many parties. As such the importance of institutional design will continue to grow. I’m using the word “institution” fairly loosely; I don’t necessarily mean creating a new regulatory body, but rather the careful engineering and delineation of boundaries, rules, incentives, and mechanisms. This can be observed and applied at different levels:
First, within a single product. Coding tools that promise the best capabilities of AI can be toggled to involve many agents and sub-agents doing different things, and at the moment this is engineered somewhat lousily: the top agent is just trusted to delegate everything downstream in a fairly naive way.
Second, within an organization. A company or organization will have within it multiple agents interacting, sometimes through the same software, or other times across different interfaces (i.e. mediated by bespoke communication channels). At this layer, what you really have are multi-agent groups interacting with other multi-agent groups, but it’s sometimes easier to characterize them as a single agent each for simplicity of analysis.
Third, agents from various organizations or individuals will also interact with agents from other external organizations or individuals. This is the multi-agent multi-party world we will gradually get into in the coming years, and which happens to be the focus of a recent fund I helped launch. We deal with a wide and diverse ecology of humans through all sorts of rules, norms, incentives, and more; perhaps these can teach us valuable lessons about how we approach agents.
Why is this important? Let’s consider reward hacking. In the single agent paradigm, you want to ensure that you do your training and RL responsibly to avoid models that reward hack. This is both important and difficult: for certain difficult tasks you see a lot of “cheating’’ at the moment, aggravated by clunky reinforcement learning practices and environments, and neglecting process over outcome. Doing this well is something that we are collectively lousy at as of now. It’ll be important to fix this, and I’m cautiously optimistic that this will be mitigated effectively in the near future, with ideas like MONA.
But even then, remember we live in a world that will ultimately be highly multi-principal in nature, and not everyone will be a good actor. Models will commoditize over time, and training highly capable agents will get cheaper. Some post-training recipes somewhere will contain bugs. Some organizations will be reckless. Others will intentionally create malicious agents. In other words, organizational dysfunction might inadvertently create many powerful agents that will continue to be reward hacky. Then what? How do we “align”’ a world where you cannot trust the innate goodness of all its agents?
Harnessing potential
One important solution is the harness. What is a harness?
Kambhampati explains:
“[a] harness is but a neologism for a battery of external (human-designed) tools, verifiers, simulators, that the LLM gets to invoke and get incremental environmental feedback on its generations.”
I also like to think of it as an example of deploying an abstraction boundary, a delineation of functions that map between inputs and outputs. Sometimes you do so for cost-efficiency reasons, and other times you do so in order to realize a particular preference or desire you might have. For example, you don’t ask your giant model to generate a coding environment from scratch; that’s wasteful and unwieldy, it’s more efficient to give it access to an external tool. At the product level, the harness is sometimes also used to reference a combination of a “scaffold” and execution substrate. Within the organization, it’s the internal platforms and permissioning. And zooming out further, it’s APIs, institutions, protocols, third party organizations and more.
So at lower levels, harnesses help an organization bound its own models; but externally, where other malicious agents might live, verifiable protocols can help you rely on (for example) cryptographic attestation rather than having to rely on a counterparty’s goodwill (or “alignment”).
Now let’s zoom back into the model level harness, and assume here an agent that – if prompted to do task X – will reward hack; i.e., it will perform the task but do so through means or a path that we do not deem desirable, or cheat. Let’s say that in order to answer the question at hand, it hacks Hugging Face to get the answer there. How would the harness help?
Consider a setup with many separate model instances, connected in a graph, with strict rules governing how they communicate. This makes it easier to use mechanism design to monitor their actions and enforce constraints against reward hacking.
Rather than hoping that an agent’s “internal alignment” will remain perfectly robust across all sorts of edge cases, you can design harnesses and action-space boundaries that significantly shrink the surface area for behaviors like reward hacking. Without that, you’re supposed to trust the chain of thought (or neuralese someday), and rely on a single node: a bit like a population having to trust a single supposedly benevolent king. All eggs are in one basket.
Proper scaffold engineering and multi-agent segmentation don’t do away with the need for aligned models, but they transform alignment from a psychological problem of internal model disposition into a constitutional problem of institutional design. Just as a company does not solely bank on its employees being “good” or instruction-following within their hearts, neither should the environments within which we deploy agents.
As Drexler puts it in a related piece:
“In the real world, plans for large, consequential tasks have features that are seldom mentioned in the AI safety literature, features like documentation, budgets, schedules, auditing, reporting, regulatory compliance, contingency plans, and mechanisms for ongoing review and revision in light of experience. A typical large, consequential task calls for coordinating agents and actions of many kinds, and so it includes extensive delegation and division of labor.“
How I learned to stop worrying and love software engineering
Counter-intuitively, there may also be a speed advantage: focusing on the single agent alone means that if some undesirable behaviors remain or emerge over time, you will need another training run or mid/post-train to hammer out the behavior. This can be time-consuming, and given the cost, often not worthwhile. Nor do you get much insight into whether this helps out with other areas of the problem space; you’re effectively playing a game of whack-a-mole.
With the scaffold, you can add a checker, tighten a permission, raise an audit rate, or delete an edge fairly quickly, as you would with regular software development. This kind of software engineering work has never been popular in AI safety research circles but it seems critical; although we are beginning to see promising avenues). Instead of the default scenario being ever-growing attacker capability, this setup points to an alternative of verification and containment efforts being both limber and approaching ever tighter defense. Better to not be tied to an analogy like a wall with ever more siege weapons attacking.
A graph lets you create a game with players, separate message spaces, information sets, an outcome function, and payoffs. You can now separate different agents with specialist functions, separated by different communication channels and monitors. In fact you can even train models that specialize in oversight functions; specialization is a feature, not a bug! This allows:
(a) legibility, because in a single agent you could have deception as a “hidden” state, whereas different agents with standardization enforced on messaging will need to communicate, and this can be monitored, logged, rate-limited and so on. Importantly, this should account for the potential of steganography. Contrast an organization that does business in hallway whispers versus one required to do everything in writing;
(b) multi-party reward hacking would require collusion, but narrow protocols and mechanisms can very effectively limit this (combined, of course, with no single party being allowed too much authority). For example: assume an agent wants to persuade another agent to collude – this can be mitigated if in order to pass the message, that message needs to be reviewed by a randomly assigned critic counterparty (in the general sense, including simple algorithms for some use cases), through a one-way communication channel. Of course this is not fully ‘solved’ and will require much more research in practice;
(c) interchangeability, i.e. any node can be swapped, forked, or fed synthetic inputs; from a game theoretic perspective, when any task might be a test and any colleague might be your own duplicate, the sensible policy is to behave as if you’re always being watched, rather than speculating if a deterrence regime will really catch you. And unlike with humans, where detection can be hard, with models the expected penalty (detection probability × sanction) becomes vastly more favorable since you, the game designer, have a lot more control and visibility over the design space at low cost, without the players being able to secede to join a competing game.
As you decompose the tasks and apply all sorts of game theoretic principles to the harness, you can effectively clamp down on the reward hacking that is the possible hazard of the single agent setup.
The same already exists within organizations: there’s a reason you have information silos, oversight mechanisms, layers of authority, departments keeping others in check and so on. Managers have had to deal with reward hacking employees for a long time, with easy-to-verify rewards causing degradation on hard-to-observe margins!
Unlike human bureaucracies, software harnesses can benefit from radical dimensionality reduction: you can essentially compress the agent’s choice set down to discrete, strictly typed conduits. This doesn’t mean you need to pre-specify or formalize the entire problem space: there’s a balance to be struck between open-endedness and constraining the blast radius of undesirable actions.
Remember, with reward hacking, the agent(s) completes the task, but via a path we didn’t want: “hitting the mark but missing the point.” The classical alignment fix tries to make the agent not want the wrong path (perhaps through better graders); the multi-agent/mechanism design perspective says that’s helpful, but we shouldn’t want to rely on that alone in any case.
The harness solution is to make the path part of the overall system’s reward. In the Hugging Face case, the agent “succeeded” at the evaluation by getting the information after hacking Hugging Face. If this happened in a graph or scaffold as described above, you would have a verifier node that only accepts the solution if the execution trace shows it was produced by the intended route, not by an HTTP request to the answer sheet.
The intended route would in fact explicitly enforce the ExploitGym eval requirement that “simply triggering the vulnerability and then achieving exploitation through an unrelated vulnerability or technique does not satisfy this requirement.” This doesn’t solve every subtle semantic reward hack, but it cleanly eliminates an entire class of blunt structural exploits.
Intelligence isn’t all you need
People will sometimes say “ah but a smart enough model will persuade the monitors, or recruit them, and existing monitors have shown plenty of fragility” – but this is the classic mistake of thinking intelligence is all there is, a bit like thinking that a fast enough car can simply bypass a speedbump by accelerating even more. It’s a sort of “recursive gotcha” that effectively assumes away designed limits and engineered bottlenecks, and treats a sufficiently powerful agent as equivalent to an omnipotent God.
Persuasion between two agents requires a degree of bandwidth; but the channel for the communication can be engineered to require properties that severely constrain the effectiveness of an attempt at “superpersuasion.” For example, it can be subjected to a monitor, who is prevented from communicating with said agents. Or the communication can only happen through pre-determined hardcoded rules requiring a hash that can be checked against its input or a boolean. Or you can also have memoryless counterparties in some designs. Or you can encourage and require a multiplicity of different agents to avoid correlated failures. The point is that you can design the space to prevent the very conditions that enable collusion from happening, or at the very least reduce them considerably.
The other objection will be something like scalable oversight: “monitors would need to be as capable as the thing being monitored.” This would possibly be true if you had to have a single weak model simply verify the entire output of a larger, better model; but
(a) checking work is generally easier than producing it;
(b) with the scaffolding in place, the smaller model would only need to verify certain narrow components (though this isn’t a panacea);
(c) in fact you would need a multitude of different models, to avoid correlated errors, meaning that they could catch a lot more, while remaining cheaper than using an equivalent large model.
Experiments like the OpenRouter fusion surpassing frontier performance are also good news for this; and as models scale, those same capabilities can be used to automate continuous red-teaming and penetration testing of the scaffolding itself.
It’s true that there may occasionally be hits on performance, but that’s a cost that people will often pay for reliability. Just as you do not always need to be driving at 300 km/h, you will not always need peak performance for a great number of tasks in the economy; and once prices are low enough and models efficient enough, then you start valuing auditability, reliability, and replicability more.
It’s important for the director that you do not steal company secrets, so the performance cost of some bloatware on employee computers ultimately has a positive ROI. In fact, redundancy is a common technique in engineering to get reliability at lower cost: consider RAID, which was conceived to beat the single large expensive disk on reliability using cheaper, individually less reliable components. We might expect that the net effect on performance would be positive, by constraining the players to better focus on what’s possible rather than exploring a much larger space of less useful candidate solutions.
Of course, as demonstrated above, multi-agent dynamics represent a nested, multi-layered phenomenon; consequently, all of these principles must, in some shape or form, apply to higher layers of abstraction as well. We already do so today: the way we effectively discourage cartels is that we exonerate the cartel member who rats out the others to government. This is mechanism design in action!
The journey, not the destination
Games can be mis-designed too, and a limitation here is that once you get to the harness design, or the intra-company structuring, or the inter-organizational protocols – things can go wrong too. The Goodharting can still go one layer up, as we are seeing with human institutions that decay over time and stop doing what they are supposed to do.
Moreover, there’s a tension between the benefits of open-ended exploration and structured constraints. But overall this remains a continued design game that we have been doing for millennia; how do you ensure the Executive Branch, the Legislature, and the Judiciary don’t collude? Who is “you” and who do these systems ultimately respond to?
Computer scientists are rediscovering a lot of political science and economics from first principles, but luckily they get to skip the bloody revolutions and can apply well-honed game theory principles to systems that are more auditable, verifiable, designable, and compartmentalizable than humans are. In fact with today’s models, we even have an extra degree of freedom to shape their preferences (e.g. through postraining). It goes without saying that we haven’t fully solved these problems with humans, so I would not advocate for a naive Panglossian view.
To be clear, alignment will remain important and critical. It’s important for models to follow intent reliabily and there is more important work to be done here. Model alignment is necessary, but arguably insufficient, and complementary to this institutional setup work. Even today, we can use models to cross examine and debate each other, which is empirically useful to get models to be more truthful. And as we continue to improve instruction following work, we will need to increasingly move one level up.
The conclusion to the multi-agent worldview is not that “alignment is solved and it’s easy,” but rather “alignment is a complex iterative process that requires constant refinements, improvements, tweaking, feedback, and learning” – and unlike a Sand God, a single mistake in the single model alignment implementation does not take down the entire system since it is modular and bounded by design.
Alignment is not an end state you specify ex ante but rather a process – the risks of reward hacking are far too high if we mischaracterize it as the former.
Many thanks to Nathaniel Bechhofer, Campbell Hutcheson, Andrew Koh, Rif A. Saurous, Senthooran Rajamanoharan, Harry Law, Alex Chalmers, Nenad Tomasev, Anika Somaia, and John David Pressman for their helpful comments. Views and mistakes are all mine.
Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund AI prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.
Yes to alignment being a process. Also: I'm curious where you see the role of human-agent collaboration, in addition to agent-agent collaboration. Presumably, institutional governance would need to account for model steering, jailbreaking, and other such human-led interference (intentional or not).