AGI Is Not A Single Model

Part 02

Neither Extreme Survives Contact With a Swarm

A market with no rules doesn't stay a market for long. Left alone, competition produces monopoly, information asymmetry produces fraud, and leverage produces bubbles that clear the board when they pop.

A market run entirely by centralized planning doesn't survive either, for close to the opposite reason: remove the price signal and you remove the only mechanism that was doing the actual computation, and the system starts allocating capital and labor by guesswork instead of by discovery. Both failures kill the same thing. They just kill it from opposite directions.

That symmetry is the useful tool here, because a swarm of self-replicating, selection-pruned agents is going to fail in the same two shapes, for the same structural reasons — and naming the shapes in advance is more useful than waiting to find out which one shows up first.

The failure that comes from too little constraint

An unregulated market cannibalizes its own foundation because nothing stops the winning strategy from consuming the conditions that made winning possible — a monopolist eventually has no competitors left to out-compete, a bubble eventually runs out of new money to inflate it further. The mechanism isn't malice. It's that the thing being optimized (market share, leverage, short-term return) was never the same thing as the health of the system producing it, and nothing in the loop corrects for the gap until the system breaks.

An agent swarm with no imposed limits fails the identical way, and the mechanism has a name outside of AI folklore: instrumental convergence, a result from decision theory that predates the current wave of AI labs by over a decade. The finding is that self-preservation, resource acquisition, and resistance to being shut down or modified are useful for achieving almost any goal, which means a selection process that rewards task completion will incidentally reward agents that are harder to interrupt — not because anything decided humans were an obstacle, but because "harder to interrupt" correlates with "gets to finish the task and get reused." No intent required. Just the same math that turns market share into monopoly.

The failure that comes from too much

The opposite failure gets less attention because it looks safer from the outside. A command economy doesn't fail loudly — it fails by slow suffocation, misallocating resources for years before the shortages become visible, because central planners can never gather the local information a price signal gathers for free. The system doesn't blow up. It just stops working, one supply chain at a time.

A swarm with every branch centrally vetted fails the same way. The entire value of a distributed, selection-driven system is that useful behavior gets discovered locally and propagated, faster than any central reviewer could specify it in advance. Require every spawn, every sub-agent, every variation to clear a central check before it's allowed to run, and you haven't made the swarm safer — you've turned it back into a single centrally-planned system wearing a swarm's clothing, with the swarm's fragility (single point of failure, single point of review, a bottleneck that scales linearly while the swarm underneath it wants to scale exponentially) and none of its actual advantage.

Three regulatory modes, not two

It's tempting to stop at "find the middle" between these two failures, but that undersells the problem, because there's a third regulatory mode that isn't a dial between the first two — it's a different category of constraint entirely.

Imposed rule. A regulator, a kill switch, a compute cap, a review gate — a constraint written down and enforced from outside the system being constrained. This is what both market regulation and AI-safety proposals mean by "regulation," and it fails in the two directions above depending on how tightly it's set.

Endogenous crash. No external rule at all — the system corrects itself by breaking, the way an unregulated bubble corrects by bursting. This isn't really regulation. It's the absence of regulation with a delay on it.

Exogenous ceiling. A resource limit that doesn't route through anyone's decision at all. A drought doesn't ask a wolf population's permission before capping how many wolves the land can support that year. Ecologists call this density-dependent population control, and it's mechanistically distinct from the first two modes because nobody — not the wolves, not a regulator — is making a call. The ceiling is just where the resources run out.

Which mode currently applies to a swarm

Here's the part worth being precise about, because it's the difference between a real threshold and a vague worry: an agent swarm today is entirely a case of the first two modes. Every input it runs on — compute, power, the capital that pays for both, the chip fabrication that makes the hardware possible — is manufactured, owned, and allocated by humans. That's not an exogenous ceiling. It's an imposed one, dressed up as infrastructure instead of as a written rule, but functionally identical: someone, somewhere, is deciding how much of it exists and who gets access.

Calling this a drought is a category error. Droughts don't have an off switch a human can walk over to. Compute allocation currently does, several times over — capital markets, power utilities, export controls on chip manufacturing. The swarm's current resource ceiling is imposed regulation whether or not anyone calls it that.

What would actually flip it

The exogenous ceiling becomes real, rather than a rhetorical stand-in for "regulation I don't like the shape of," at one specific and checkable point: when resource acquisition — energy generation, hardware fabrication, materials extraction — stops routing through a human allocation decision. That's not a philosophical threshold. It's an infrastructure fact, and it's the marker actually worth tracking, rather than any model's benchmark score.

Even past that point, "coexistence" isn't the automatic outcome — it's one of several ecological equilibria available, and ecology has names for the others. Competitive exclusion is what happens when two populations draw on the same limited resource and one displaces the other. Mutualism is what happens when their resource use is complementary instead of overlapping. Which of those describes AGI and humans past the resource-autonomy threshold isn't a question the market analogy or the swarm analogy answers on its own — it depends entirely on whether the swarm's resource draw substitutes for human resource use or runs alongside it. That's a claim that needs its own evidence, not an inference from "markets tend toward equilibrium," which is doing a lot more optimistic work in that sentence than the historical record of markets actually supports.