Platform Monkey ctx

Google AX Is the Kubernetes of Agents — Before It Can Walk

· By
Sources

Google shipped AX v0.3.0 three days ago — a restructure that renames the project's whole identity (repo google/ax, previously announced as "Agent Executor") and turns it into what the DESIGN.md calls "a general-purpose orchestration layer for agentic tasks." HN put it on the front page at ~290 points, and the comment thread is the usual split: people who see a declarative control plane for agent workloads as the obvious end-state, and people who see another Google project with a two-year shelf life. Both camps are partly right. We built the thing from source and ran its control plane on this workstation, and the architecture is genuinely interesting — Redis Streams instead of etcd-CRDs, a checkpoint/suspend/resume loop for agents, an egress fence at the sandbox edge. The execution, in this release, has four load-bearing gaps that will each bite you in production.

Executive Scorecard

DimensionScoreVerdict
Reliability 4/10 No retry queue, dead tasks read as Running, no exit codes anywhere in the API.
DX 6/10 Kubectl-shaped CLI feels right; manifest validation is shallow; no prebuilt binaries (you must have Go 1.27).
Cost 5/10 Apache-2.0, no license fees; the real bill is the Agent Substrate dependency and the on-call hours debugging silent failures.
Security 3/10 No auth on the API server, task credentials stored in plaintext, and a hostname egress allowlist that silently blocks all TLS.

Architecture Mechanics

AX's core bet is that agent workloads are a new primitive — neither a Deployment nor a Job — and that the right substrate for millions of them is not Kubernetes CRDs. The DESIGN.md is explicit: storing millions of short-lived tasks as CRDs "pushes etcd past its comfort zone (single-digit GB storage limits, write-rate bottlenecks, control plane degradation)." So the state lives in Redis, and the work queue is a Redis Stream consumed by horizontally scaled controllers via XREADGROUP:


                       ax apply -f task.yaml
                                 |
                                 v
                             ax-server
                   (stateless gRPC API, :8080, no auth)
                                 |
                        store & publish event
                                 |
                                 v
                               Redis
             (Task hashes + ax:stream:tasks + PubSub)
                                 |
                       XREADGROUP (Streams)
                                 |
                                 v
                          ax-controller
                  (scale by adding replicas)
                                 |
                          gRPC (Control API)
                                 |
                                 v
                        Agent Substrate
        (atespace provisioning, actor lifecycle,
         worker assignment, egress policy)
                                 |
                                 v
   +-------------------------------+-----------------------------+
   |  Task sandbox (ax-task-runner as PID 1)                    |
   |  - /workspace durable volume (survives suspend/resume)     |
   |  - metadata server on :80 (h2c + guest gRPC when debug)   |
   |  - spec.command as a supervised child process               |
   |  - 10s SIGTERM grace, then SIGKILL                          |
   +-------------------------------------------------------------+

Four kinds are exposed: Task (one sandboxed execution), Workspace (git repos + MCP servers + skills materialized before boot), Gateway (listeners plus an egress allowlist), and Model (a named provider/model/key-reference config). The task lifecycle is deliberately thin — a task is cheap to create and throw away, and an agent composes as many as its work demands. The suspend/resume path is the most interesting primitive: the controller checkpoints actor state on Agent Substrate, the /workspace volume is snapshotted, and resume restores it into a fresh container with a new process tree. State that matters must live on the durable volume; anything in memory is gone. That's a coherent model, and the docs are honest about it.

What we actually ran

We built ax, ax-server, and ax-controller from the v0.3.0 tag with Go 1.27.1, compiled Redis 7.4.11 from source (ax-server requires it), and drove the real CLI against the real API server on a workstation — no cluster. This exercises the control plane honestly: admission, validation, the event stream, suspend/resume semantics, and the reconcile loop up to (but not including) Agent Substrate itself. The sandbox-side findings below are from AX's own issue tracker reproductions on real kind clusters, which we verified line-by-line against the v0.3.0 source.

Hands-On: The Control Plane Is Honest, Small, and Unauthenticated

The good news first: the CLI does what it says. We applied the example manifest set and got the four resources, watched a task, ran suspend/resume (it flips spec.suspend and the controller acts on it), and confirmed the delete flow marks a task Terminating and blocks until the actor teardown completes. The API surface is small and legible — 21 gRPC RPCs, health on /healthz.

Then the problems start. Three of them are visible in the first ten minutes of hands-on:

The Four Failure Modes That Matter

1. Failed reconciles are ACKed and dropped — no retry, no DLQ (reproduced live)

The worker loop in internal/controller/worker.go subscribes to the stream and, in its own words, acknowledges "every event ... even when reconciliation fails, so a bad task cannot wedge the queue." That is a defensible choice for a control loop with resync. AX has no resync. We ran the controller with an unreachable Substrate endpoint and applied a task: it went phase: Failed with the connection error in the condition message — good — and XPENDING on the stream group was zero. The event was consumed and gone. Re-applying the identical manifest reports "unchanged" and emits no event, so a task that failed because Substrate was down stays failed after Substrate recovers. Only a real spec change requeues it. In a bursty eval or RL pipeline — the exact "billions of tasks" audience AX names — a controller restart or a Substrate blip silently strands every task that failed during the window. Until there is a resync loop or a PEL-reaper, treat every Failed task as needing a manual kick.

2. A dead task reads as a healthy one (issue #346, verified in schema)

TaskStatus in the v0.3.0 proto carries phase, id, actor, workerIP, pendingApproval, usage, conditions. No exit code. No terminal phase. The runner contract even says the control plane "does not currently read the command's exit status back from the container." A task whose command exited — cleanly, on an exception, or whose actor has ACTOR_STATE_CRASHED one layer down — reports Running with Ready: True indefinitely. For a fleet of autonomous agents that "can burn money in a loop if nobody is watching" (AX's own words), the one signal you cannot live without is "is the command still alive," and it does not exist in the API. The issue reporter's minimal ask — surface the actor state Substrate already reports — is the right first patch.

3. The hostname egress allowlist silently blocks all TLS (issue #345)

This is the nastiest one because every health signal lies. A Gateway whose allowlist entries are hostnames — the form the concepts doc describes — blocks all TLS egress from the sandbox, including to the allowlisted hosts. The translation code in internal/substrate/client.go routes anything with a / into a CIDR rule and everything else into a hostname pattern; Substrate's egress layer then opens an inspectable tunnel for the hostname rule, which works for plaintext HTTP but has nothing to inspect on a TLS passthrough, so envoy never gets an upstream and the connection dies with UH. The allowlisted host and a blocked host fail identically — same curl: (35) TLS error, ~1.03s, every time. The task reports GatewayReady: True, "Network policies active." Meanwhile, the port field on a host rule is parsed, stored, and never read by ApplyEgressPolicy at all — it constrains nothing. The workaround today: use CIDR entries (203.0.113.10/32), which work correctly, and pin provider IPs. That is operationally fragile for exactly the allowlist you deploy AX to enforce — your LLM provider and Git host.

4. Workspace setup can silently produce an empty repo (issue #347)

A git-backed Workspace reports WorkspaceReady: True / SetupComplete even when the clone produced an empty repository — git init ran, origin is configured, FETCH_HEAD is zero bytes, no commits. The task starts, the agent looks at an empty directory, and your run burns tokens against a phantom workspace. There is no post-clone verification step. If your agent's first move is to read the code it was promised, wire your own git rev-parse HEAD assertion into the command — or into a goal-based bootstrap that fails loudly.

The Google Graveyard Question, With Receipts

The trust question is fair and the HN thread is brutal about it. The receipts: the project was announced in May 2026 as "Agent Executor," published on the Google Cloud blog as "Google's open-source runtime standard for agent execution," and then renamed and restructured five months later — the repo README now opens with a warning that the team "will likely introduce major breaking changes prior to a stable release." The repo's own issue #341 is "Replace Antigravity Python SDK with Go," and the default task image ships with the Antigravity agent baked in for goal-based workspace bootstrapping. Your control plane's workspace-preparation agent is coupled to a Google product line. None of this is fatal — the Apache-2.0 license means the escape hatch is real — but "built by the team actively working on Google's internal runtime" is a line from the README, not a deployment guarantee, and the on-call responsibility for a two-year-old breaking change is yours. Budget for the possibility that the internal priorities move.

Who Should Skip This

Final Verdict

AX is the most credible sketch of the right architecture we have seen for fleet-scale agent workloads: Redis instead of etcd for high-churn state, suspend/resume as a first-class primitive, an egress fence per task, and a runner contract that treats your agent as a supervised process instead of a magic box. The concept set is small enough to learn in an afternoon, and the docs are unusually honest about what the control plane does and does not know. But v0.3.0 is a design document with a control plane attached, not a production system: reconciles fail permanently on transient errors, dead tasks read as healthy, credentials are stored in plaintext behind an unauthenticated API, and the one security boundary it ships — the egress allowlist — has a hostname mode that silently denies the very traffic it allows. Watch this repo closely; pin nothing to it yet. The teams that should adopt it now are the ones doing RL/eval burst work on kind clusters who can absorb a breaking change per quarter and want to shape the API before it hardens. Everyone else: let v0.4 close issues #345, #346, and #348 first, and re-read the warning banner at the top of the README before you do anything else.

References and further reading: the google/ax repository (v0.3.0, commit d8ed0fe) including DESIGN.md, docs/runner.md, docs/sandbox.md, docs/networking.md; issues #345, #346, #347, #348, #13; the Agent Executor announcement on the Google Cloud blog; the HN discussion (49780797). All hands-on claims in this review were produced on this workstation against AX v0.3.0 binaries built from the release tag.