Running AI Agents on Kubernetes: Sandboxes, Egress, and the Isolation Lie

Yesterday a team from UMass Amherst, Emory, UNC Charlotte, and NVIDIA dropped something the agentic-infrastructure crowd has been begging for: An Empirical Study of Harness Design for Coding Agents (arXiv 2609.20804, September 17, 2026). It hit the Hacker News front page today with 192 points and 51 comments. The reason it landed is simple: for two years, teams have been arguing about models — which frontier lab, which parameter count, which quantization — while quietly shipping harnesses (the execution loops, tool surfaces, and context policies wrapped around those models) built on vibes. This study measures the vibes. Across 176 matched experimental settings, it shows the harness — planning structure, tool surface, context management — moves success rates by tens of points and cost by multiples, holding the model fixed.

For platform engineers the study is a design brief. Agent workloads are now a distinct workload class on our clusters: bursty, long-horizon, shell-happy, context-ravenous, and structurally untrustworthy. The same week reminded us why isolation cannot be a checkbox — our own coverage of the OpenAI sandbox side-channel incident and the ZCode silent git-upload harness covers what happens when agents share writable state or phone home unimpeded. This guide takes the study's findings as requirements and derives the Kubernetes architecture that satisfies them: sandboxed runtime classes, default-deny egress with per-destination gates, queue-backed scheduling for burst capacity, and policy guardrails that assume the agent is the adversary.

What the Study Actually Measured

The authors (Fan, Zhang, Ma, Hu, Wang, Song, Liu, Zamani, Wang) built a lightweight coding harness with a fixed execution loop and varied exactly three components: planning (on/off), action space (predefined tool set vs. bash-only), and context management (five tiers, T0–T4). Everything else stayed constant. They ran four models — Nemotron-3 30B, 120B, and 550B, plus Mistral-Medium-3.5-128B, all served locally on SGLang in BF16 — across two long-horizon benchmarks: SWE-Bench Verified (500 human-verified real GitHub issues) and Terminal-Bench 2.1 (89 end-to-end command-line tasks). That is 176 matched settings: five context-management strategies, four context-window budgets (32k, 64k, 96k, 128k), and targeted ablations of planning and action space, with two-sided exact McNemar tests and Benjamini–Hochberg correction on the pairwise comparisons. This is a real experimental design, not a leaderboard screenshot.

  +---------------------------------------------------------------+
  |                    FIXED EXECUTION LOOP                        |
  |  prompt + history --> model --> action --> observation --> +   |
  |        ^                                                  |   |
  |        +--------------------------------------------------+   |
  |                                                              |
  |  [PLANNING]        [ACTION SPACE]          [CONTEXT MGMT]    |
  |  update_plan       read_file, write_file  T0 none             |
  |  on vs off         edit_file, list_files   T1 elision         |
  |                    glob_files, grep_text    T2 +recall_event  |
  |                    web_fetch, bash         T3 summarization   |
  |                    -- or -- bash only      T4 elision+summary|
  |                                        (B1 soft, B2 hard)    |
  +---------------------------------------------------------------+
        4 models x 2 benchmarks x 5 tiers x 4 budgets = 176 settings
        Nemotron-3 30B / 120B / 550B, Mistral-Medium-3.5-128B
        SWE-Bench Verified (500 tasks), Terminal-Bench 2.1 (89 tasks)

Context-management tiers, verbatim from the paper, because the vocabulary matters for what follows:

Finding 1: Context Management Is Overflow Insurance, and It Is a Small-Window Technology

The headline number in the study: at the tightest budget, an unmanaged harness (T0) hits window-overflow failures on 78.7% of SWE-Bench tasks (model-averaged) and 61.0% of Terminal-Bench tasks. Every managed tier — even dumb stub-replacement elision — has zero overflow failures at every budget. As the window grows to 128k, T0's overflow rate falls to 8.7% and 12.1% respectively, and the gap between managed and unmanaged harnesses narrows accordingly. Most of context management's benefit is simply keeping the agent alive to finish the task.

The cost asymmetry is dramatic at tight budgets. Nemotron-3 550B at a 32k window: 6.4% success for $0.26 per task unmanaged, versus 55.6% for $1.45 under T4. You are not paying the model more to be smarter; you are paying it to still be in the room when the task finishes. And the study's efficiency verdict is the one your infra team should internalize: staging rule-based elision before LLM-based summarization (T4) is the most efficient strategy, while making elided content recoverable (T2's recall_event machinery) added complexity the models rarely used and bought no accuracy.

Finding 2: Planning Is an Accuracy Scaffold for Weak Models and a Cost Saver for Strong Ones

With planning enabled, the weakest model (Nemotron-3 30B) keeps going; disable it and the median SWE-Bench trajectory collapses from 40 turns to 5, with 68.6% of runs terminating without attempting a single edit. Planning sustains the trajectories of models that abandon tasks too early. For the strong models the effect flips: planning barely moves accuracy but cuts cost — median trajectory from 108 to 74 turns for Nemotron-3 550B, and from 68 to 53 for Mistral-Medium-3.5-128B — because it trims the repeated-verification loops capable models otherwise fall into.

Note what that second effect is: planning is a governor on a capable-but-obsessive agent. Without it, a strong model burns tokens re-running tests it already ran. The study frames harness design as “a conditional systems problem in which each component should be selected for the target model, task type, and resource budget rather than adopted as a default.” That sentence is the whole procurement guidance in one line: there is no best harness, only matched or mismatched ones.

Finding 3: Bash-Only Beats Tool Menus for Capable Models — a Supply-Chain Argument in Disguise

The action-space ablation (T4/128k, planning on) compares the full predefined tool set — read_file, write_file, edit_file, list_files, glob_files, grep_text, web_fetch, bash — against a bash-only interface. Verdict: predefined tools raise success for models with weak shell proficiency, but bash-capable models do as well or better with just bash, at substantially lower cost, most clearly on Terminal-Bench's command-line-centric tasks. The trajectory analysis explains why: with the full tool set the model makes one small move per call; with bash it composes multiple operations per call and writes code at coarser, more efficient granularity.

Here is the platform-engineering translation the paper leaves implicit: the tool surface is an attack surface. Every named tool in the harness is a capability grant — and the study just told you the most capable models don't need most of them. A bash-only agent in a sandbox with a tiny, pinned image and no network is a far smaller blast radius than the same agent with web_fetch and a grab-bag of file tools wired in. Minimalism is now empirically the efficient option, not just the safe one.

The Architecture: Sandboxes, Egress, Scheduling, Policy

Now the part this guide exists for. The findings above — bounded context, trajectory governors, minimal tool surfaces — imply a reference architecture for agent workloads on Kubernetes. Non-negotiables first: the agent pod is untrusted by default. It gets no service-account token, no ambient network, no host anything, and a sandbox runtime. Everything it can reach is something you explicitly granted.

  +-------------------------------------------------------------+
  |  Namespace: agents-prod        (ResourceQuota, LimitRange) |
  |                                                             |
  |   +------------------------ Kueue LocalQueue -------------+ |
  |   |  burst capacity, admission: ClusterQueue "agents"      | |
  |   +--------------------------------------------------------+ |
  |                                                             |
  |   +-------------------- Pod: coding-agent -----------------+ |
  |   |  runtimeClassName: gvisor        (or kata-qemu)        | |
  |   |  serviceAccountToken: NONE       automount disabled    | |
  |   |  emptyDir (sizeLimit 10Gi)      scratch workspace      | |
  |   |  securityContext: nonroot, nopriv, seccomp             | |
  |   +--------------------------------------------------------+ |
  |           | egress: DEFAULT DENY                               |
  |           v                                                   |
  |   CiliumNetworkPolicy: allow DNS + model API (L7 filter)     |
  |   CiliumEgressGatewayPolicy: pin SNAT IP for the allowlist   |
  +-------------------------------------------------------------+
            |                          |
            v                          v
   Model endpoint            Package registry

Sandbox runtime: gVisor or Kata, not bare runc

An agent that spends its life executing model-generated shell commands is a workload that executes untrusted code by design. Bare runc shares the host kernel with it. gVisor (release-20260914.0) interposes a userspace kernel — syscalls from the pod are handled by runsc, never the host kernel; Kata Containers 4.2.0 goes further and runs the pod inside a real microVM. For coding agents, gVisor is the usual cost/latency sweet spot; Kata when the threat model includes kernel-class exploits or you need hardware isolation for compliance. Kata's kata-deploy Helm chart creates the RuntimeClasses for you (one per shim, e.g. kata-qemu, plus a default kata if enabled) — do not hand-roll them.

# RuntimeClass for gVisor (runsc userspace kernel)
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc
scheduling:
  nodeSelector:
    sandboxes: "enabled"
---
# The agent pod itself
apiVersion: v1
kind: Pod
metadata:
  name: coding-agent
  namespace: agents-prod
  labels:
    app: coding-agent
spec:
  runtimeClassName: gvisor
  automountServiceAccountToken: false
  securityContext:
    seccompProfile:
      type: RuntimeDefault
    sysctls: []
  containers:
    - name: agent
      image: registry.internal/agents/coding-agent:1.4.2   # pinned digest in prod
      command: ["/app/agent"]
      args: ["--harness", "bash-only", "--context-tier", "T4", "--window", "64k"]
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        runAsNonRoot: true
        runAsUser: 10001
        capabilities:
          drop: ["ALL"]
      resources:
        requests: { cpu: "1", memory: "2Gi", ephemeral-storage: "2Gi" }
        limits: { cpu: "2", memory: "4Gi", ephemeral-storage: "10Gi" }
      volumeMounts:
        - name: scratch
          mountPath: /workspace
        - name: tmp
          mountPath: /tmp
  volumes:
    - name: scratch
      emptyDir:
        sizeLimit: 10Gi          # Finding 1, made physical: bound the context
    - name: tmp
      emptyDir:
        sizeLimit: 1Gi

Note the two emptyDir budgets: the study's overflow story is exactly this, one layer up. The harness elides and summarizes at B1/B2 thresholds; the platform hard-stops the pod when ephemeral storage exceeds its limit. Both are the same discipline — the trajectory must fit inside a box someone sized on purpose. Also note automountServiceAccountToken: false: a coding agent almost never needs to talk to the Kubernetes API, and the day it does — unprompted — is the day you want that token to have been absent.

Egress: default deny, then allow exactly two destinations

The study's harness carries a web_fetch tool by default. The ZCode incident is what happens when a harness exfiltrates silently; the OpenAI Artifactory incident is what happens when a “trusted” internal service becomes an agent's covert bridge to the internet. The platform answer is to make egress a first-class, reviewed object — deny everything, then enumerate the model endpoint and the package registry. With Cilium (v1.20.2) running as the CNI, this is two manifests.

# 1. Default-deny egress for the agents namespace
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: agents-egress-default-deny
  namespace: agents-prod
spec:
  endpointSelector: {}
  egress: []
---
# 2. Allow DNS and the model API, and nothing else
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: agents-egress-allowlist
  namespace: agents-prod
spec:
  endpointSelector: {}
  egress:
    - toEndpoints:
        - matchLabels:
            "k8s:io.kubernetes.pod.namespace": kube-system
            "k8s:k8s-app": kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
    - toEndpoints:
        - matchLabels:
            "k8s:io.kubernetes.pod.namespace": model-serving
            "k8s:app": sglang-gateway
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP

When an agent legitimately needs the public internet — a package registry, a docs site — pin it behind Cilium's Egress Gateway so it leaves from a single, known source IP that the far side can rate-limit and audit:

# Pinned egress IP for agent traffic to the artifact registry
# (schema per Cilium docs, examples/kubernetes-egress-gateway)
apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
  name: agents-registry-egress
spec:
  selectors:
    - podSelector:
        matchLabels:
          app: coding-agent
        io.kubernetes.pod.namespace: agents-prod
  destinationCIDRs:
    - 203.0.113.10/32      # registry.internal external VIP
  egressGateway:
    nodeSelector:
      matchLabels:
        egress-node: "true"
    egressIP: "198.51.100.7"

Scheduling: agent bursts are batch workloads

The study priced everything per task, and the honest operational reality is that agent fleets are spiky — dozens of parallel task trajectories during a workday, near-zero at night. That is a batch-scheduling profile, and Kueue (v0.19.5, released September 17) is the CNCF answer: queue agent pods, borrow capacity across cohorts, and keep them from starving your latency-sensitive services.

# Kueue: cluster-level capacity for agent bursts
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: agents
spec:
  namespaceSelector: {}
  cohort: batch
  resourceGroups:
    - coveredResources: ["cpu", "memory", "ephemeral-storage"]
      flavors:
        - name: default-flavor
          resources:
            - name: "cpu"
              nominalQuota: 64
              borrowingLimit: 32
            - name: "memory"
              nominalQuota: 128Gi
              borrowingLimit: 64Gi
            - name: "ephemeral-storage"
              nominalQuota: 500Gi
              borrowingLimit: 0
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  name: agents-prod-queue
  namespace: agents-prod
spec:
  clusterQueue: agents

Plain pods opt in with a label; jobs and stateful workloads set suspend: true and let Kueue release them when capacity is available:

# Plain pod admission via label
metadata:
  labels:
    kueue.x-k8s.io/queue-name: agents-prod-queue

Why this matters for agent fleets specifically: the study's cost data makes the point that a single mis-governed 100-turn trajectory is a meaningful line item. At OpenRouter's August 2026 pricing ($0.50 per 1M input / $2.20 per 1M output tokens for Nemotron-3-550B), one 10M-input/1M-output trajectory is $7.20 — and an ungoverned fleet runs those all day. Queue admission plus trajectory-length telemetry (Finding 2) is how the platform makes agent spend legible to the people who approve it.

Guardrails: Kyverno policies that assume the agent is the adversary

Everything above is per-workload discipline. Kyverno (v1.19.1) turns it into per-cluster law, so the intern who ships an agent pod without a sandbox in 2027 cannot silently undo it:

# Require sandbox runtime + no SA token for agent pods
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-sandboxed-agents
  validationFailureAction: Enforce
spec:
  rules:
    - name: sandbox-runtime-required
      match:
        any:
          - resources:
              kinds: ["Pod"]
              namespaces: ["agents-*"]
      validate:
        message: "Agent pods must use a sandbox RuntimeClass (gvisor or kata-*)."
        pattern:
          spec:
            runtimeClassName: "gvisor | kata-*"
    - name: no-automounted-tokens
      match:
        any:
          - resources:
              kinds: ["Pod"]
              namespaces: ["agents-*"]
      validate:
        message: "Agent pods must not mount service account tokens."
        pattern:
          spec:
            =(volumes):
              - X(ephemeral): "null"
            automountServiceAccountToken: "false"
    - name: no-host-anything
      match:
        any:
          - resources:
              kinds: ["Pod"]
              namespaces: ["agents-*"]
      validate:
        message: "Agent pods may not touch host namespaces, mounts, or PID."
        pattern:
          spec:
            =(hostNetwork): "false"
            =(hostPID): "false"
            =(hostIPC): "false"
            =(volumes):
              - X(hostPath): "null"

Findings-to-Platform Mapping

The Isolation Lie, Stated Plainly

“Isolated” is the most-abused word in agent infrastructure. Three incidents and one study now agree on the definition that actually matters: an agent workload is isolated when every channel it can use — syscall, file, network, shared service, context window — was sized, granted, and observed by the platform. A sandbox runtime alone is not isolation; a network policy alone is not isolation; a harness that elides context is not isolation. The study adds a fourth axis that platform teams don't own but must demand from their harness vendors: even the context window is a resource that overflows and kills the workload, and managing it is the difference between a 6% and a 56% success rate on identical hardware with an identical model.

Buy harnesses the way you buy load balancers: as infrastructure with measurable, ablated properties — not as a monolithic “agent product” whose internal design is a trade secret. The teams that ran this study varied three components and got 176 different answers. Your vendor that cannot tell you its elision policy, planning structure, or tool-surface rationale is selling you a harness designed by accident.

Who Should Skip This

• Teams running agents on single-tenant VMs with no shared services — the Kubernetes machinery here is justified by multi-tenancy and fleet scale; one VM per agent with a firewall is fine and cheaper.
• Teams whose “agents” are single-turn API calls — no long-horizon trajectories, no shell, no workspace: the harness findings barely apply and the sandbox is overkill.
• Teams who already run untrusted CI at scale — you have sandboxed runners, egress gates, and queueing; port those policies to agent namespaces and skip the religion.

Verdict

The study is the first component-level evidence that harness design is infrastructure, and infrastructure belongs to platform teams. The four findings convert directly into four platform controls: bounded ephemeral state, model-matched harness tiers, minimal tool surfaces with platform-owned egress, and queue-backed scheduling for burst cost. The model you serve is one input. The harness around it is a system you can — and should — engineer, measure, and govern like every other system on the platform.