Every platform team building AI-agent training infrastructure eventually hits the same wall: reinforcement learning on real infrastructure — Kubernetes, charts, admission controllers, CNIs — needs thousands of isolated, resettable, and above all identical worlds. The standard answer is a cold kind or k3s cluster per trial: about 30 seconds of boot time and a full RAM bill per copy. When your GRPO loop wants a group of 16 rollouts from the same starting state, that tax compounds into the dominant cost of the entire training run.

k7d — a Rust VMM and containerd shim from Katakate (docs at docs.katakate.org) — takes the opposite bet: boot the cluster once, then fork the live machine. The headline: a running 3-VM Kubernetes cluster forks in ~105 ms on the minimal CI guest, a Ubuntu k3s+Cilium+Tetragon cluster in ~1.1 s, and 50 three-VM cluster forks land in ~4.1 s on one 64 GB bare-metal box. Yesterday's v0.8.0 release added coordinated cluster keyframes that cut the source pause down to the residual dirty pages via live pre-copy — the most architecturally significant change since the project went public in August.

Methodology disclosure, because we owe you that: our analysis box has no /dev/kvm, so nothing in this piece is hands-on. What we did instead is audit the source of truth most VMM projects don't have: k7d wires every latency claim on its README into CI-enforced budgets (LATENCY_BUDGETS.md, constants imported from crates/k7d/src/latency.rs into crates/k7d/tests/test_latency.rs). Every number below is traced to a repo file. That is k7d's single most credible property — and, as we'll see, its own budget file is refreshingly honest about where the numbers are loose.

The structural shift: fork the universe, don't reset it

Group-relative policy optimization — GRPO, introduced in the DeepSeekMath paper — compares rewards within a group of rollouts from a shared prompt. The group-relative baseline is what makes it sample-efficient, and it is also what makes it brutally sensitive to environment state. If member A starts from a cold page cache, a different etcd revision, or a half-ready Deployment while member B starts warm, the reward gap between them is noise, not signal. The paper's authors controlled this with math on the model side; on the infrastructure side, nobody controls it — because a "reset" script that re-applies manifests never reproduces the same kernel timers, the same TLS session state, or the same in-flight reconcile loop.

k7d's answer is to make the copies byte-identical by construction: a fork is a copy-on-write clone of the live machine state — RAM, disk, processes, established TCP between member VMs, in-cluster TLS, kube-apiserver state, everything. From inside the fork, nothing moved. The README's phrasing is right: that is the difference between "we reset the env" and "we cloned the universe."

The second shift is economic. CoW forks share the parent's memory pages until someone writes: 50 cluster copies cost 50× the dirty pages, not 50× the guest RAM. That is what makes a 64 GB box a plausible RL gym. The daemon exposes the fork loop as a budgeted tree — you protect winners, prune losers, and let the daemon auto-evict under RAM and disk caps while the loop runs.

The third shift is the one nobody else does: k7d owns the guest clock. Because the VMM controls kvmclock, per-vCPU TSC offsets, and timer re-arms, the same pause that makes a fork correct can also move every member's clock by the same delta. An 8.7-minute guest-time episode that is mostly kubelet waiting on probes compresses 1.83× with lockstep clock jumps at zero fidelity cost — the parts of an RL episode that are pure waiting stop costing wall-clock time.

Architectural blueprint: what actually happens in 105 ms

The design is one daemon (k7d, one process, one address space, many VMs) owning a KVM VMM, a snapshot-tree manager, and the guest clock. Your trainer talks JSON-lines over a Unix socket at /run/k7d/k7d.sock. A fork, end to end, is a short critical section on the source:

sequenceDiagram
    participant T as Trainer (JSON-lines client)
    participant D as k7d daemon
    participant S as Source VMs (3-node cluster)
    participant C as Child VMs (forks)
    T->>D: tree_fork_batch(n)
    D->>S: pause every vCPU, drain in-flight virtio I/O
    D->>D: dirty bitmap = KVM dirty log ∪ device-written pages
    D->>C: map parent RAM copy-on-write, copy only dirty pages
    D->>C: new bridge per child, replay guest IPs / MACs
    D->>C: restore vCPUs, re-arm timers, KVM_SET_CLOCK (never backwards)
    D->>S: resume
    D-->>T: n children + latency_ns