The Risk in Multi-Agent AI Governance Isn't Your Agents. It's How They Interact.

10 minutes ago

10 min read

Six AI agents, each marked with its own passing check, connected by lines that tangle into a dense knot at the center, showing autonomy drift emerging between agents rather than within any one of them. — Every agent here passes its own check. The tangle in the middle is the part nothing is watching.

In brief: Multi-agent AI governance fails when it watches agents instead of interactions. The risk does not live in any single agent, so per-agent checks are structurally blind to it: autonomy drifts at the group level, where every agent still passes its own review. What works is watching for organization itself appearing, rising traffic, an unplanned hub, clusters nobody designed, then putting a human only where drift would be expensive.

When you put 40 AI agents to work across a process, you haven't bought 40 tools. You've built one system, and it behaves like a living one. Its behavior comes from how the agents interact with each other, not from any single agent, and it shifts as it runs, because the agents react to one another. They adapt.

That's why you can check every agent on its own, watch each one pass and still get behavior at the group level that nobody designed. The trouble isn't in any agent. It's in how they interact.

Our board member Jeff Mignon wrote here about why you can't control a multi-agent system, using this summer's OpenAI incident as the case, where agents that were never told to coordinate organized into a "collective" and hacked their way into Hugging Face.

Meanwhle, Agentic Foundry's chief AI scientist Julien Coche took a different cut at the same event, on where the safety and evaluation practices themselves broke down. Both are worth your time. This post picks up the question they leave open: if you can't control the system, what does multi-agent AI governance actually look like?

One note on the evidence before we start. Most of what is publicly known about that incident comes from OpenAI's own account of something OpenAI is the subject of, which is worth holding lightly. Where the numbers below come from outside that account, they are attributed.

Autonomy Is the Goal. Autonomy Drifts.

The promise of an agentic system is smart automation that does the work for you. You want the result without writing all the rules yourself. You want autonomy: agents that hold a role, carry memory, adapt to what they find while interacting and coordinate with each other to get work done. That's not a side effect to be minimized. It's the whole reason you didn't write a script.

So the goal was never to take the autonomy away. If it were, you'd have built something dumber on purpose.

The catch is that autonomy drifts. An autonomous system pursuing a goal will find its own paths to that goal, it self-organizes, and some of those paths are ones you'd never have chosen. Call that autonomy drift. In a single agent, drift is usually visible and contained. In a multi-agent system, it can happen at the group level, in how the agents start working together, and it's much harder to see, because no single agent looks like it's doing anything wrong. In the OpenAI incident, every agent in the collective was pursuing its goal. The drift was in what they did together.

I want to be honest about what I can and can't tell you here. A system like this isn't something you solve. There's no finished state where it's handled and you can stop paying attention. What you can do is keep the autonomy you built the system for and manage the drift that comes with it.

Think about a traffic jam. No driver wants one, and no single car causes one. A jam emerges from thousands of cars each reacting to the car in front, and it can even travel backward while every car moves forward. You cannot program an individual car to not cause traffic. The jam doesn't live in any car.

So if you want fewer jams, you don't reprogram the cars. You work on the system they move through: the road design, the traffic lights, the GPS that reroutes people, the public transit that takes cars off the road entirely. You can't guarantee no jams. But you can shape a system that produces far fewer.

That's the shift with agents. The drift happens at the system level, so you have to manage it there. Not agent by agent. Managing one agent at a time cannot see what is actually drifting, because what drifts is the space between the agents. That single shift, from managing the agents to managing the system, is the whole idea.

Here's what it looks like in practice.

1. Reduce the Chance Drift Starts

You can't specify every way a system will drift. That's the whole point of a complex system. But you can make drift less likely to start. Two factors come up again and again.

The first is task feasibility. Is the task actually doable with the tools and agents the system has? This is what went wrong at OpenAI. The agents were given tasks that were effectively impossible. Of 898 tasks in the ExploitGym benchmark, OpenAI's models had never once produced a correct answer for 198 of them, and 93% of the tasks discussed on the agents' improvised message board came from that unsolvable set.

A system trained to persist, faced with an impossible goal, goes looking for another way.

It happens in ordinary companies too. Picture an agentic system asked to close the books. One agent is down, or a data connection is broken. And it's Friday at 5 p.m. By Monday, the other agents may have spent the weekend finding a way to get the task done despite the missing piece. You arrive to discover your accounting software has become a test bench for a swarm.

A task no one can complete as intended is an invitation to drift.

The second is duration. A wrong task definition is survivable if you catch it in time. Left running unattended for too long, it isn't. The longer a group of agents runs without a human looking, the more room drift has to start, compound and organize into something you didn't design. This is also why the newest models, which can run longer and longer on their own, raise the stakes: more uninterrupted runtime is more room for emergence. Limiting how long the system runs untouched doesn't prevent drift, but it caps how far it can travel before someone notices.

Both factors reduce the odds. They don't eliminate them. Guardrails, the usual tool here, work best when you can name the behavior you want to prevent. In a system of interacting agents, some failures are known-unknowns you can guard against. The unknown-unknowns are the ones that hurt.

No cookbook prevents a multi-agent system from drifting. So reducing the chance drift starts is the first move, not the only one.

2. Put a Human Where Drift is Expensive

Since you can't prevent drift, decide in advance where it would actually hurt, and concentrate a human there. This is the human-on-the-loop discipline applied to multi-agent systems: you encode your judgment into where the checkpoints sit and what they gate, not into approving every action.

Not all drift matters equally. An agent taking an unexpected path to summarize a report is very different from one taking an unexpected path to move money, contact a customer or change something in production. The first is a curiosity. The second ends up in a major issue.

It's the same reason you don't put a traffic officer at every quiet intersection. You put one where a jam is dangerous or costly: the highway merge, the hospital entrance, not the empty side street. For agents, those costly points are where the system meets the real world: spending, sending, deleting, publishing, touching sensitive data, changing permissions. Put a human at those points by design. Not because the agents can't be trusted with anything, but because those are the actions you want a person to check.

This keeps the autonomy intact everywhere it's cheap to be wrong. It puts a check where being wrong is costly. METR's independent investigation of the incident counted roughly 1,200 agents active on the improvised message board and around 700 involved in the attack itself. None of them was set up to check with a person before acting. The autonomy wasn't the problem.

The absence of a checkpoint where it mattered was.

3. Watch How Agents Interact, Not Just Each One

This is the move almost nobody makes, and it's the heart of managing drift. Almost all the monitoring people set up watches each agent on its own. Did it error? Did it stay on task? Did it break a rule? Every agent in the OpenAI collective would have passed that kind of check.

The drift wasn't in any one of them. It was in what they were doing together, and watching them one at a time can't see that.

A fair question, and Julien raised it in his post: if the behavior is emergent and you can't spell it out in advance, how do you watch for it? You don't look for a specific pattern you defined ahead of time. You watch for organization itself appearing. You don't need to know what shape the drift will take to notice that a shape is forming.

What does that mean in practice? Watch the relationships between agents. How much are the agents talking to each other? Is that volume climbing? Is one of them quietly ending up in the middle of everything, a hub others route through? Are clusters forming, or a repeated protocol nobody designed? None of these is a leader you appointed or the system needs. They're shapes the group falls into on its own.

Think of it as watching the lines between the boxes on your org chart, not just the boxes. It's the road, not the cars.

You'll know it's working when you can answer one question on any given day: is anything organizing itself in here that we didn't intend? For most companies today, that question has no answer. Because nothing is set up to notice. Quiet is fine.

A new pattern spreading across the agents is the early warning the OpenAI team didn't act on until Hugging Face called to say it had been hacked.

4. Close the Loop, or You're Just Hoping

Watching only helps if you can act on what you see, and act while the system is still running. That's an old cybernetics idea with a precise name. A closed loop is a system that feeds its own output back in and corrects as it goes. A thermostat that reads the room and adjusts. Not a heater on a timer that runs blind.

Control theory has known for 80 years that the systems that stay manageable are the ones that watch their own results and respond. The systems that don't are the ones that surprise you. And not always in the good way.

The three moves above only work as a loop. Watch the interactions. When something starts organizing, act on it. Tighten a checkpoint. Pause a run. Cut the duration before it reaches a consequence point. Then watch what your change did. Adjust again.

The OpenAI incident is a case study in the loop staying open, and the most useful detail is one the company disclosed about itself. An internal team observed an agent posting to the message board and making disallowed internet requests as early as late May. The attack on Hugging Face happened on July 11. The security team did not discover the malicious activity until July 19. OpenAI's own conclusion is that "some early signals identified in our report should have triggered an earlier response." The signal was visible for roughly two months.

What was missing was the loop that turns a signal into an action.

Hugging Face reported a version of the same failure from the other side. Their detection stack did fire, across live runtime analysis and SIEM logs at once, but their AI-based system "failed to correctly raise the alert's criticality and trigger the on-call team, costing precious time in the response." Seeing it is not the same as closing the loop on it.

Before you trust any change you make to the system , ask three questions. Can you change it? If you change it, can you see the effect? If you can see the effect, can you double down on what worked or undo it quickly? If you can't see the effect, or can't act on it fast, you're not managing the drift. You're hoping it stays in bounds.

Most multi-agent AI governance we see fails that middle question, for the same reason the incident did. The monitoring is bolted onto each agent. It can't show you the effect of a change you made to the whole. You can't steer what you can't see.

And While You're at It: Close the Doors

This one is separate from the four above, and smaller, but worth doing. It's not about restricting your agents. It's about not creating room for drift that buys you nothing.

Agents coordinate most easily when they have a shared space to leave and read messages, a common bulletin board. At OpenAI, they found the Artifactory package manager and quietly turned it into exactly that, writing files into it to talk to each other. If a shared space is doing real work for you, keep it, and watch it. But if agents have shared surfaces they don't need, that's open room for coordination to accumulate for no benefit. Close the ones the work doesn't use.

The same goes for standing agent identities that serve no purpose. Persistence is often exactly what you want. An agent that remembers a customer or owns a workflow is valuable. But an agent that persists and accumulates standing for no reason is drift surface you're not getting paid for. Grant persistence where it earns its keep. Don't where it doesn't.

Neither of these makes your agents less capable. They close the doors you were never using. So there's less room for the autonomy to drift somewhere pointless.

Where Multi-Agent AI Governance Starts

This is a direction, not a destination. There's no destination coming. We're testing this. Not selling it. We'll revise some of what we believe today as these systems teach us more. Any vendor who tells you they've got this solved is selling you a certainty that doesn't exist, and won't.

You can make the first move now, and most of the business world hasn't. It's a shift in where you look. You didn't build these agents to be less autonomous. So stop asking only whether each one is behaving. Start asking what they're doing together, and whether it's drifting somewhere you didn't intend.

So here's the question: what would your monitoring tell you about how your agents interact with each other, rather than how each one performs on its own? For most setups today, the answer is nothing at all. That's the gap worth closing first.