AI Fleet Architecture: Orchestrating 100+ Agents Without the Stampede

Every platform team running coding agents at fleet scale hits the same wall, and it is not the model. Anthropic — operating Managed Agents, a hosted service for long-horizon agent work — and Linear, which published its agent-era CI rework today, both describe the identical failure shape: unbounded parallel agents stampede the resources behind them. Anthropic’s early agents spawned 50 subagents for simple queries and “scoured the web endlessly for nonexistent sources.” Linear’s test suite almost quadrupled since the start of 2026 under agent-written code, adding roughly 2,000 tests a week, until CI became the constraint on the whole engineering org.

This guide is the architecture that survives the stampede: an admission-controlled agent fleet on Kubernetes where compute, sessions, and spend are three separate, bounded planes. It is built from patterns Anthropic actually runs in production and Kueue primitives you can apply on Monday, not reference-architecture marketing.

TL;DR

• The stampede is multi-resource. A fleet of 100+ agents fails on four planes at once: sandbox compute, API/token spend, downstream shared systems (CI, registries, source control), and context. Fixing one plane just moves the explosion.
• Decouple brain, hands, and session — Anthropic’s core pattern. The harness (brain) is a stateless process outside the sandbox (hands); the session is an append-only log outside both. Result in their production fleet: p50 time-to-first-token dropped roughly 60%, p95 dropped over 90%.
• Admission control before autoscaling. Kueue gates how many agent workloads start at all, per team, with borrowing between team quotas and preemption policies — the same discipline batch HPC has used for years.
• Budgets are kill switches, not dashboards. Anthropic’s Managed Agents session budgets hard-stop a session’s spend at public list rates. Multi-agent systems burn about 15× the tokens of a chat interaction — a 100-agent fleet multiplies that per agent.
• When not to use this: if you run fewer than ~10 concurrent agents, or your agents are chat-copilots rather than batch workloads, this machinery is overkill. A queue-per-team and a spend alert will do.

What the Stampede Actually Kills First

Anthropic’s post-mortem style posts are unusually specific about failure sequences, and they map cleanly onto what platform engineers see on their own clusters:

• Head-of-line blocking on shared state. Anthropic’s first Managed Agents design put the harness, the session, and the sandbox in one container. “If a container failed, the session was lost. If a container was unresponsive, we had to nurse it back to health.” Debugging an unresponsive container required a shell — but the container held user data, so “that approach essentially meant we lacked the ability to debug.” A pet, in the pets-vs-cattle sense, at fleet scale.
• Amplified, opaque failures. In the coupled design, a harness bug, a dropped WebSocket event, and a dead container all presented the same symptom. That is the operational definition of a stampede: you do not know which of 100+ agents is sick, only that throughput collapsed.
• Token burn. Anthropic measured it: agents use about 4× more tokens than chat interactions; multi-agent systems about 15×. Their own analysis of BrowseComp found token usage alone explains 80% of performance variance — multi-agent systems win by spending more, which is exactly why they need spending controls.
• Downstream saturation. Linear’s numbers show the blast radius: with agents authoring most code, every PR still passes through CI, so CI runner time and queue wait became the org-wide bottleneck. Their fix — batching seven checks into two jobs — saved 87,000 runner-minutes per month, 11.8% of total CI usage, on its own.
• Resource floors that are also ceilings. Anthropic’s eval-infra team found that treating per-task resource specs as both guarantee and hard limit OOM-killed containers on transient spikes — a 6-percentage-point swing on Terminal-Bench 2.0 (p<0.01), with up to 6% of tasks failing on pod errors unrelated to model capability. Agent workloads spike; zero-headroom quotas kill them.

The Reference Topology

The pattern that survived production at Anthropic has three planes, and each is independently scalable and disposable. Mapped onto Kubernetes:

flowchart TB
    subgraph INGEST["Work Ingest (API / queue / cron)"]
        REQ["Agent task requests"]
    end
    subgraph ADMIT["Admission Control — Kueue"]
        CQ1["ClusterQueue: team-platform
nominalQuota: 40 pods"]
        CQ2["ClusterQueue: team-data
nominalQuota: 20 pods"]
        COHORT["cohortName: agent-fleet
(borrow unused quota)"]
        WPC["WorkloadPriorityClass
(prod > batch > background)"]
    end
    subgraph BRAINS["Brain plane — stateless harnesses"]
        H1["harness pod"]
        H2["harness pod"]
        H3["harness pod N"]
    end
    subgraph SESSION["Session plane — append-only log"]
        SS["session store
(emitEvent / getEvents / wake)"]
    end
    subgraph HANDS["Hand plane — sandboxes as cattle"]
        SB1["sandbox pod (repo cloned, token wired into git remote)"]
        SB2["sandbox pod"]
        SB3["sandbox pod N (provisioned on demand)"]
    end
    subgraph SPEND["Spend plane"]
        BUDGET["Session budgets
(hard stop at list cost)"]
        ROUTER["LLM router / gateway"]
    end

    REQ --> WPC --> CQ1 & CQ2
    CQ1 & CQ2 --- COHORT
    CQ1 & CQ2 --> H1 & H2 & H3
    H1 & H2 & H3 <-->|"execute(name, input) -> string"| SB1 & SB2 & SB3
    H1 & H2 & H3 <--> SS
    H1 & H2 & H3 --> BUDGET --> ROUTER
    SB1 & SB2 & SB3 -->|"MCP via vault proxy
(harness never holds credentials)"| ROUTER

Three things in that diagram are load-bearing, and all three come from Anthropic’s production writeups rather than whiteboard theory:

1. The brain is cattle. The harness holds no state that must survive: “nothing in the harness needs to survive a crash. When one fails, a new one can be rebooted with wake(sessionId).” In Kubernetes terms, harness Deployments with aggressive maxSurge, no local state, no PVCs.
2. The hands are a tool, not a home. The sandbox is called the way any tool is called — execute(name, input) → string — and provisioned on demand via provision({resources}). If it dies, the failure surfaces as a tool-call error the model itself can retry. Sandboxes are Kueue-managed pods or Jobs, never StatefulSets.
3. The session is outside the context window. Long-horizon work exceeds any context window, so the session log is the durable source of truth and the brain interrogates it positionally (getEvents(), rewind, re-read). Context engineering — compaction, trimming, cache-hit optimization — happens in the harness on top of the log, so it can change without touching storage.

The security consequence of the split is the part most homegrown fleets get wrong. Anthropic’s structural fix for prompt-injection-led credential theft: “make sure the tokens are never reachable from the sandbox where Claude’s generated code runs.” Git credentials are used once, at sandbox initialization, to clone and wire the remote — the agent never handles the token. MCP tool credentials live in a vault; a proxy fetches them per call. The harness is never made aware of any credentials. We covered the same egress-control discipline for model-serving infrastructure in our exfiltration-controls analysis, and the fleet version is stricter, because now the attacker has a loop that runs code.

Layer 1 — Admission Control: Kueue in Front of the Fleet

The cheapest way to survive 100+ agents is to never have 100+ agents running when the cluster can only handle 40. Kueue (current stable v0.19.5, released September 17, 2026, with v0.18.9 maintained in parallel) is a job-level manager that decides when workloads are allowed to start — exactly the admission discipline an agent fleet needs, and the same one our agent-on-Kubernetes isolation guide uses for sandbox placement.

The unit of admission is a Workload. For an agent fleet, one Workload = one agent task (or one team’s batch of agent tasks). Teams get ClusterQueues; ClusterQueues share a cohort so idle quota is borrowed, not wasted:

apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: team-platform-agents
spec:
  namespaceSelector: {}          # match agent namespaces
  cohortName: agent-fleet        # borrow/lend within the fleet
  queueingStrategy: BestEffortFIFO
  resourceGroups:
  - coveredResources: ["cpu", "memory", "pods"]
    flavors:
    - name: default-flavor
      resources:
      - name: cpu
        nominalQuota: 96
        borrowingLimit: 48       # cap the stampede: never borrow more than 2x nominal
      - name: memory
        nominalQuota: 384Gi
        borrowingLimit: 192Gi
      - name: pods               # "pods" is reserved — bounds concurrent agents
        nominalQuota: 40
        borrowingLimit: 20
  preemption:
    reclaimWithinCohort: Any     # reclaim nominal quota from borrowers, any priority
    withinClusterQueue: LowerPriority
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  name: agent-tasks
  namespace: agents-platform
spec:
  clusterQueue: team-platform-agents

Field notes, verified against the current Kueue concepts documentation and the v1beta2 API types:

• queueingStrategy: BestEffortFIFO is the fleet default (and Kueue’s default too): a big old task that cannot be admitted does not block newer small tasks that fit. StrictFIFO is the stampede-friendly option — avoid it for agent fleets where task sizes vary wildly.
• The reserved pods resource is the concurrency cap. You cannot request pods in a Pod spec, but ClusterQueue quotas can count pods, and Kueue computes the count per Workload. This is your “max concurrent agents per team” knob.
• borrowingLimit is the anti-stampede valve. Within a cohort, a ClusterQueue can borrow unused nominal quota from teammates — but if the field is empty it can borrow up to the sum of everyone’s nominal quota. During a stampede, everyone goes maximal at once. Cap borrowing explicitly.
• Preemption makes priorities real. reclaimWithinCohort: Any lets a team take back its own nominal quota from borrowers; withinClusterQueue: LowerPriority lets high-priority work evict background agents. Candidate selection prefers borrowing queues, then lowest priority, then most-recently-admitted — i.e., the youngest, least valuable agent dies first, which is what you want.

Priority is a label away. Define a WorkloadPriorityClass and stamp agent tasks with it — this drives both queue ordering and preemption:

apiVersion: kueue.x-k8s.io/v1beta2
kind: WorkloadPriorityClass
metadata:
  name: agent-prod
value: 10000
description: "Interactive, user-waiting agent tasks"
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: WorkloadPriorityClass
metadata:
  name: agent-background
value: 1000
description: "Overnight batch agents — preemptable by prod"
---
apiVersion: batch/v1
kind: Job
metadata:
  name: refactor-agent-4821
  labels:
    kueue.x-k8s.io/queue-name: agent-tasks
    kueue.x-k8s.io/priority-class: agent-prod
spec:
  # ... the agent harness job ...

Anthropic’s own scaling rules fit this taxonomy cleanly: their research system’s early failure — spawning 50 subagents for a trivial query — is a priority-class inversion where a task that deserved agent-background with one worker got agent-prod with fifty. Their fix was prompt-level (“simple fact-finding requires 1 agent with 3–10 tool calls; complex research might use more than 10 subagents”); yours should be quota-level as well, because prompts are advisory and quotas are enforced.

Layer 2 — The Brain/Hands Split in Kubernetes Terms

Anthropic’s decoupling was not motivated by elegance. Two production forces drove it:

Customer VPCs. When the harness lived inside the container, connecting an agent to customer infrastructure required network peering — a per-customer, security-review-heavy path. Once the harness became just another client of execute(name, input) → string, the “hands” could live anywhere, including the customer’s own environment, and the brain stays central. For your fleet, this is the same reason your orchestrator should call sandboxes through a tool interface rather than baking kubectl exec into the harness: the execution plane becomes swappable per tenant, per compliance boundary, per cloud.

Time-to-first-token. In the coupled design, “many brains required as many containers,” and every session paid full container setup before the first inference call. After the split, inference starts as soon as the orchestration layer pulls pending events; containers are provisioned by tool call only if needed. Anthropic’s measured result: p50 TTFT dropped roughly 60%, p95 dropped over 90%. At fleet scale, that is the difference between a pool of harnesses that is mostly doing model I/O and a pool mostly booting containers. Linear saw the same physics on the CI side: their API test shards spent 110–140 seconds per shard on setup before any test ran; capping fetch depth took the slowest gate from 94s to 20s, and the median change-detection job from 26s to 8s.

The session plane is what makes brains disposable. Anthropic’s interface set is deliberately tiny — emitEvent(id, event) to record, getSession(id)/getEvents() to recover, wake(sessionId) to resume — and it predates any specific harness implementation the way read() predates the SSD. In Kubernetes, the session plane is an append-only store (Postgres, ClickHouse, or a managed stream) that harness pods mount nothing from; recovery is a pod restart plus a log replay. Their long-running-harness work shows why the log must beat the context window: agents working across many context windows need “structured artifacts to hand off context between sessions” — a feature list, a progress file, git history — and irreversible in-window compaction is what loses state when a crash lands mid-summary. Durable log first, compaction second, never the reverse.

Layer 3 — Spend Control: Budgets as Circuit Breakers

Admission control bounds compute. It does not bound API spend — an admitted agent can burn tokens indefinitely. This is where the numbers get unforgiving: Anthropic measured agents at ~4× chat token usage and multi-agent systems at ~15×, and found that 90.2% eval improvement of their multi-agent research system over a single Opus 4 agent is substantially a token-spend effect (token usage alone explained 80% of BrowseComp variance). Multi-agent fleets are a token-buying strategy. Treat spend like capacity.

Anthropic’s production answer is instructive because it is a platform primitive, not a prompt: Managed Agents session budgets (Beta) price everything a session consumes at public list rates and stop issuing new model requests once the session’s list cost hits the cap — the in-flight request finishes, the session pauses idle rather than terminating, and raising the budget resumes it:

{
  "agent": "agt_refactor_runner",
  "environment_id": "env_sandbox_vpc",
  "budget": {
    "type": "limit",
    "max_list_cost": {
      "amount": "125",
      "currency": "USD"
    }
  }
}

Three design decisions in that API are worth copying into any self-built fleet control plane:

• The budget is evaluated by the platform, not the agent. The harness cannot negotiate, plead, or round past it. Your equivalent is the LLM router/gateway refusing to forward requests for a session whose meter tripped — the enforcement point we demonstrated hands-on in our LLM gateway tests, where router-level fallback behavior determined whether fleets leaked errors or survived a provider outage.
• Pausing beats killing. A budget-tripped session goes idle and resumes when the budget moves. For long-horizon agent work, killing means replaying expensive session history; pausing costs only the wait. Your queue system should treat spend-tripped workloads the same way Kueue treats inadmissible workloads: parked, observable, resumable.
• Budgets attach per session, deployments apply them per spawned session. Fleet-level spend caps compose from per-unit caps. Per-agent caps are what stop the 2 a.m. infinite-retry loop that one confused agent can produce.

The Downstream Plane: CI Is Part of Your Agent Fleet

The most under-modeled part of fleet architecture is what the fleet does to shared downstream systems. Linear’s post today is the best public dataset on this — and it doubles as a capacity-planning warning for anyone whose agents write code:

The architectural lesson is not “optimize CI” — it is that agent fleet capacity planning must include the downstream systems the fleet writes to. Linear’s shard math makes the coupling explicit: “further sharding only pays off when the fixed cost per shard is low, since doubling the shard count also doubles the workflow time spent on setup.” At their old 110–140s setup, eight shards would have burned 15–19 minutes of runner time on setup alone — more than the tests. Your agent fleet’s sandbox-provisioning cost obeys the same law, which is precisely why Anthropic’s on-demand provision({resources}) and session-decoupled TTFT mattered so much.

If your agents open PRs, your fleet SLA is your CI queue depth. If your agents call internal APIs, your fleet SLA is those services’ rate limits. Instrument both, or the stampede just changes address.

Cost Math (Date-Stamped: September 21, 2026)

Two arithmetic models, one published number and one derived (labeled as such):

Published: Linear’s check-batching change saved 87,000 runner-minutes/month, which they state equals 11.8% of their total CI usage. That is 1,450 runner-hours per month from consolidating seven jobs into two.

Derived: if 87,000 minutes is 11.8% of total CI, Linear’s implied CI footprint is roughly 737,000 runner-minutes/month. At GitHub’s published baseline-Linux list rate of $0.006/minute (Windows baseline $0.010/minute, verified today), that is on the order of $4,400/month of runner compute at list rates before any runner upgrades — which is why Linear moved to third-party runners and why your agent-fleet CI bill deserves a line item of its own. These are order-of-magnitude figures from list prices; your contract rates will differ.

Spend multiplier: using Anthropic’s published ratios, a single-agent session burns ~4× a chat session’s tokens and a multi-agent session ~15×. A 100-agent fleet running multi-agent patterns is therefore on the order of 1,500× a single chat session’s token consumption per task-wave. That multiplier is the reason Layer 3 is not optional: the difference between “agents are expensive” and “agents are a line item you forecast” is per-session budget enforcement.

Real Deployments Cited (All Named, All Public)

Observability: Watch the Queue, Not Just the Agents

Kueue ships Prometheus metrics that map directly to stampede symptoms. The ones that earn their dashboard tiles:

# Concurrency pressure: pending vs admitted per team queue
kueue_pending_workloads{cluster_queue="team-platform-agents",status="active"}

# Admission latency — the fleet's "queue depth" SLA
histogram_quantile(0.95,
  sum by (le, cluster_queue) (
    rate(kueue_admission_wait_time_seconds_bucket[15m])
  ))

# Preemption storms: rising evictions = priorities fighting, not coordinating
sum by (cluster_queue) (
  increase(kueue_preempted_workloads_total[1h])
)

# Spend canary (router-side): sessions near budget cap per team
# forward your gateway's per-session token meter here; alert on
# approaching max_list_cost, not only on trips

kueue_pending_workloads splits into active (in the admission queue) and inadmissible (failed an admission attempt and parked until conditions change) — the second is your signal that quotas, not demand, are the constraint. Anthropic’s observability lesson from the research system generalizes: they added full production tracing not to watch conversations but to watch decision patterns and interaction structures — how many subagents, how many tool calls, which tools — because the failure modes (50 subagents on a trivial query) are visible in shape, not content.

When NOT to Use This Architecture

• Fewer than ~10 concurrent agents. Kueue, cohorts, and a session store are real operational surface. A single namespace with kubectl apply and a router-side spend cap does the same job with less to own.
• Agents that are copilots, not batch workloads. If a human is in the loop and concurrency is bounded by seat count, admission control solves a problem you do not have. Anthropic’s own guidance is that most coding tasks “involve fewer truly parallelizable tasks than research.”
• Tight-deadline interactive work. Queueing adds latency by design. If your agents must start within seconds and demand is spiky, overprovision and cap spend instead of admitting — preemption of an interactive agent is a user-visible failure.
• Multi-agent for its own sake. Anthropic is blunt: domains where “all agents to share the same context” or with “many dependencies between agents” are a bad fit today. If your task is a pipeline, run a workflow — orchestration frameworks and event-driven patterns (SQS/PubSub/EventGrid) are the lighter tool.
• You cannot yet measure per-session token spend. Everything in Layer 3 assumes a metering point. Without one, budgets are theater; build the meter first (our Kubernetes cost-visibility guide covers the cluster side).

The Pattern, Compressed

Admission control bounds how many agents run. Brain/hands/session decoupling makes each agent cheap to lose and quick to start. Budget enforcement bounds what each agent can spend. Everything else — the framework, the orchestrator, the mesh — is implementation detail that those three constraints let you swap without fleet-wide incidents. Anthropic built the interfaces first precisely because “harnesses encode assumptions that go stale as models improve”; your fleet will outlive at least two model generations, so the durable asset is the queue, the log, and the meter — not the agent.