1. TL;DR — The Verified Numbers

Container checkpoint/restore via CRIU is the only mechanism Kubernetes has for moving a running process tree — heap, open file descriptors, process hierarchy, the works — from one node to another without the application persisting anything. A fresh empirical study (Khasmammadov, v1.3.0, October 2026) ran the full pipeline on a three-node AWS EC2 cluster in eu-central-1 — CRI-O 1.31, CRIU v4.3, Buildah 1.37, Cilium 1.16 eBPF host routing — with 36 measured CRIU migrations against 20 native cold reschedules and a 50 Hz external prober. The published raw dataset backs every number below. Treat the results as one carefully-instrumented testbed, not a universal benchmark; the mechanics are the transferable part.

Metric (study, N=36 CRIU vs N=20 cold)CRIU decoupled migrationNative cold rescheduleDelta
Median client-visible blackout (operator)1064.5 ms1019.0 ms+45.5 ms (+4.5%)
Median blackout (50 Hz external prober)1080.0 ms1040.0 ms+40.0 ms (+3.8%)
Blackout std. deviation241.1 ms231.2 msPLEG relist noise floor
Statistical significanceMann-Whitney U=252.5, p=0.066Not significant at alpha=0.05
In-memory state (RPO)669 ms bounded drift, RAM 100% intactTotal loss (cold start)The entire point

The headline finding is brutal for the migration hype cycle: at small heap sizes, a CRIU migration has statistically indistinguishable client-facing downtime from a plain pod deletion and reschedule. The p-value (0.066) does not reject equal distributions — partly because the 1-second PLEG relist interval injects roughly uniform 0–1000 ms detection noise (theoretical sigma of a uniform [0,1 s) variable is 1000/sqrt(12) = 288.7 ms; measured sigma was ~241 ms on both sides). What CRIU buys is not faster cutover. It buys bit-exact RAM state — verified SHA-256-identical across nodes at every heap size from 26 MB to 1 GB — at a cutover cost within noise of a cold restart. That is a trade most stateful workloads want and almost none of the marketing admits.

The second finding is the one that decides whether you can run this on Spot: the 120-second AWS Spot interruption window is the real SLA, and the pipeline fits inside it with room to spare at 1 GB. Worst-case evacuation measured on the study testbed — 5.0 s interruption detection plus a 7.464 s CRIU dump plus 13.743 s of uncompressed Buildah packaging plus the registry push — totaled 36.78 s, only 30.6% of the 120 s budget. But the margin collapses non-linearly with heap size, and the wall is not CRIU: it is the Linux dirty-page writeback threshold, then the single-threaded gzip layer that Buildah runs by default.

Allocated heap (study, worst-case Spot path)26 MB64 MB128 MB256 MB512 MB1024 MB
CRIU freeze (T_dump)0.144 s0.216 s0.258 s0.375 s0.760 s7.464 s
Buildah packaging (gzip)0.371 s0.761 s1.082 s5.796 s16.120 s32.060 s
Buildah packaging (-D, no compression)0.261 s0.361 s0.528 s1.473 s4.210 s13.743 s
Worst-case evacuation (of 120 s Spot budget)5.70 s (4.8%)6.19 s (5.2%)6.71 s (5.6%)8.53 s (7.1%)13.24 s (11.0%)36.78 s (30.6%)
Restore on target node1.066 s0.951 s1.695 s2.108 s7.807 s17.328 s
SHA-256 heap verificationMatch100%100%100%100%100%

Why migrate at all if cutover is a wash? Because rebuilding the same state from external storage is not free either: the study estimates reconstructing a 1 GB in-memory snapshot from S3 at 11.2–29.8 s on assumed 300–800 Mbps cloud bandwidth, and puts the checkpoint/restore break-even against cold-rebuild-from-object-storage at roughly 256 MB of live heap (2.10 s restore vs 3.5–7.5 s rebuild). Below that, just restart the process. Above it, checkpoint/restore starts winning — and wins more the bigger and slower-to-warm your heap is.

The rest of this guide covers the mechanics you actually operate: what CRIU does to a process, what the kubelet checkpoint API does and does not promise, how the checkpoint becomes an OCI image and why that format is CRI-O-only pre-alpha, why PLEG — not CRIU — dominates your cutover variance, what eBPF does and does not fix, a production build on AWS with every config file, and where the whole approach breaks.

2. Why This Is Worth Your Time Right Now

Checkpoint/restore in Kubernetes is usually filed under "2022 blog post, forensic analysis, harmless." Three things changed. First, the upstream is finally building the pod-level API this feature always needed: KEP-5823 (Pod-level Checkpoint/Restore, opened January 2026) defines a first-class PodCheckpoint API object, a kubelet watch-driven execution model with no privileged HTTP endpoint, and a declarative spec.restoreFrom restore path — alpha-targeted at the v1.38 cycle, with the CheckpointPod/RestorePod CRI RPCs already merged into the v1.37 CRI API proto. Second, the PLEG problem that dominates cutover variance got a credible fix path: Evented PLEG was redesigned in v1.36–v1.37 into a termination-hint fast path and re-targets Beta in v1.38. Third, someone finally published a properly-instrumented empirical study of the whole pipeline on AWS with a raw dataset — which is what turned this from API trivia into an operations guide.

Upstream eventVersion / dateWhat it means operationally
KEP-2008 container checkpoint API, alphav1.25 (Aug 2022)POST /checkpoint/{ns}/{pod}/{container} on kubelet :10250
ContainerCheckpoint gate to Beta, default onv1.30 (Apr 2024)Endpoint exists on default kubelet configs; still Beta at v1.37.2 (current stable)
containerd wires the CRI checkpoint RPC (PR #6965)Merged Mar 2024, ships in containerd 2.xcontainerd 1.7 does not implement CheckpointContainer — kubelet returns 500 on 1.x AMIs
CheckpointPod/RestorePod RPCs in CRI APIv1.37 proto (Oct 2026)Pod-wide consistent cut is specified at the runtime interface
KEP-5823 opened (PodCheckpoint, restoreFrom)Jan 2026; alpha target v1.38Gate PodLevelCheckpointRestore not yet in the tree at v1.38.0-beta.0 — plan on 1.38 final or later
Evented PLEG redesigned to termination-hint pathv1.36/v1.37 (PRs #137362, #139262); Beta re-target v1.38Cut the detection noise that dominates migration blackouts
Empirical AWS study v1.3.0 (56-run dataset, Zenodo DOI)Oct 2026Real numbers for cutover, RPO, heap scaling, Spot budget

Honesty note on the lead: the Hacker News thread that surfaced the study (item 50280010, October 28, 2026) has one point. The value here is the data and the verified mechanics, not community buzz — I re-verified every upstream claim in this guide against the KEPs, the v1.37.2 and v1.38.0-beta.0 source trees, the CRI-O and containerd sources, and the study's own published artifacts.

3. The Problem: Stateful Pods on Ephemeral Nodes

AWS gives you a two-minute execution notice on Spot: when EC2 decides to reclaim the instance, the interruption is visible in IMDS at /latest/meta-data/spot/instance-action (values hibernate, stop, terminate) and via EventBridge, and the instance goes away roughly 120 seconds later. The one exception that catches people: hibernation-behavior interruptions do not give you the two-minute warning — the hibernation begins immediately. Everything below assumes terminate/stop behavior.

Two minutes sounds generous until you decompose what a stateful pod must do inside it. The default Kubernetes answer is "delete the pod, let the scheduler recreate it, and the application rebuilds its state" — which is exactly what the study measured as the cold-reschedule baseline: ~1 second of client blackout at small heaps, and an RPO of everything in RAM that was not persisted. For a game-server lobby, an in-memory feature store, a stream processor with warm buffers, or a JVM that spent eight minutes JIT-ing and loading classes, "cold rebuild" is not an RPO, it is an outage. The standard escapes each move the state somewhere else: externalize to a database (latency, rewrite), persist to EBS via StatefulSet (still loses RAM), or run a VM and live-migrate it (KubeVirt — real live migration, real VM overhead). Checkpoint/restore is the fourth door: freeze the process tree, capture RAM as data, and resume it elsewhere bit-identical.

t=0s      IMDS spot/instance-action appears (2-min notice starts)
          |
          v
[detect]  IMDS poller notices instance-action          <= 5.0 s (worst case)
[cordon]  kubectl cordon + migration trigger              ~1 s
[dump]    CRIU checkpoint: freeze + write memory images    7.464 s   } dirty-page
[build]   buildah -D: tar into OCI layer (uncompressed)  13.743 s   } writeback wall
[push]    registry push over VPC                          ~10 s     } at 1 GB
          |
          |  TOTAL EVACUATION (source side): 36.78 s = 30.6% of the 120 s budget
          v
[kill]    delete source pod (gracePeriodSeconds=0)  <---- client blackout starts
[restore] pull OCI layer + CRI-O unpack + CRIU restore  17.328 s   (~8.5 s pull
          first HTTP 200 from restored process          + 4.8 s unpack
                                                      + 3.2 s CRIU restore
                                                      + 0.8 s kubelet sync)
          |
          v
t=~54 s   service serving again, heap identical (SHA-256 verified)
          remaining Spot budget: ~66 s of head room