Stateful Pod Migration on Kubernetes: CRIU Checkpoint/Restore Under Production Load

1. TL;DR — The Verified Numbers

Container checkpoint/restore via CRIU is the only mechanism Kubernetes has for moving a running process tree — heap, open file descriptors, process hierarchy, the works — from one node to another without the application persisting anything. A fresh empirical study (Khasmammadov, v1.3.0, October 2026) ran the full pipeline on a three-node AWS EC2 cluster in eu-central-1 — CRI-O 1.31, CRIU v4.3, Buildah 1.37, Cilium 1.16 eBPF host routing — with 36 measured CRIU migrations against 20 native cold reschedules and a 50 Hz external prober. The published raw dataset backs every number below. Treat the results as one carefully-instrumented testbed, not a universal benchmark; the mechanics are the transferable part.

The headline finding is brutal for the migration hype cycle: at small heap sizes, a CRIU migration has statistically indistinguishable client-facing downtime from a plain pod deletion and reschedule. The p-value (0.066) does not reject equal distributions — partly because the 1-second PLEG relist interval injects roughly uniform 0–1000 ms detection noise (theoretical sigma of a uniform [0,1 s) variable is 1000/sqrt(12) = 288.7 ms; measured sigma was ~241 ms on both sides). What CRIU buys is not faster cutover. It buys bit-exact RAM state — verified SHA-256-identical across nodes at every heap size from 26 MB to 1 GB — at a cutover cost within noise of a cold restart. That is a trade most stateful workloads want and almost none of the marketing admits.

The second finding is the one that decides whether you can run this on Spot: the 120-second AWS Spot interruption window is the real SLA, and the pipeline fits inside it with room to spare at 1 GB. Worst-case evacuation measured on the study testbed — 5.0 s interruption detection plus a 7.464 s CRIU dump plus 13.743 s of uncompressed Buildah packaging plus the registry push — totaled 36.78 s, only 30.6% of the 120 s budget. But the margin collapses non-linearly with heap size, and the wall is not CRIU: it is the Linux dirty-page writeback threshold, then the single-threaded gzip layer that Buildah runs by default.

Why migrate at all if cutover is a wash? Because rebuilding the same state from external storage is not free either: the study estimates reconstructing a 1 GB in-memory snapshot from S3 at 11.2–29.8 s on assumed 300–800 Mbps cloud bandwidth, and puts the checkpoint/restore break-even against cold-rebuild-from-object-storage at roughly 256 MB of live heap (2.10 s restore vs 3.5–7.5 s rebuild). Below that, just restart the process. Above it, checkpoint/restore starts winning — and wins more the bigger and slower-to-warm your heap is.

The rest of this guide covers the mechanics you actually operate: what CRIU does to a process, what the kubelet checkpoint API does and does not promise, how the checkpoint becomes an OCI image and why that format is CRI-O-only pre-alpha, why PLEG — not CRIU — dominates your cutover variance, what eBPF does and does not fix, a production build on AWS with every config file, and where the whole approach breaks.

2. Why This Is Worth Your Time Right Now

Checkpoint/restore in Kubernetes is usually filed under "2022 blog post, forensic analysis, harmless." Three things changed. First, the upstream is finally building the pod-level API this feature always needed: KEP-5823 (Pod-level Checkpoint/Restore, opened January 2026) defines a first-class PodCheckpoint API object, a kubelet watch-driven execution model with no privileged HTTP endpoint, and a declarative spec.restoreFrom restore path — alpha-targeted at the v1.38 cycle, with the CheckpointPod/RestorePod CRI RPCs already merged into the v1.37 CRI API proto. Second, the PLEG problem that dominates cutover variance got a credible fix path: Evented PLEG was redesigned in v1.36–v1.37 into a termination-hint fast path and re-targets Beta in v1.38. Third, someone finally published a properly-instrumented empirical study of the whole pipeline on AWS with a raw dataset — which is what turned this from API trivia into an operations guide.

Honesty note on the lead: the Hacker News thread that surfaced the study (item 50280010, October 28, 2026) has one point. The value here is the data and the verified mechanics, not community buzz — I re-verified every upstream claim in this guide against the KEPs, the v1.37.2 and v1.38.0-beta.0 source trees, the CRI-O and containerd sources, and the study's own published artifacts.

3. The Problem: Stateful Pods on Ephemeral Nodes

AWS gives you a two-minute execution notice on Spot: when EC2 decides to reclaim the instance, the interruption is visible in IMDS at /latest/meta-data/spot/instance-action (values hibernate, stop, terminate) and via EventBridge, and the instance goes away roughly 120 seconds later. The one exception that catches people: hibernation-behavior interruptions do not give you the two-minute warning — the hibernation begins immediately. Everything below assumes terminate/stop behavior.

Two minutes sounds generous until you decompose what a stateful pod must do inside it. The default Kubernetes answer is "delete the pod, let the scheduler recreate it, and the application rebuilds its state" — which is exactly what the study measured as the cold-reschedule baseline: ~1 second of client blackout at small heaps, and an RPO of everything in RAM that was not persisted. For a game-server lobby, an in-memory feature store, a stream processor with warm buffers, or a JVM that spent eight minutes JIT-ing and loading classes, "cold rebuild" is not an RPO, it is an outage. The standard escapes each move the state somewhere else: externalize to a database (latency, rewrite), persist to EBS via StatefulSet (still loses RAM), or run a VM and live-migrate it (KubeVirt — real live migration, real VM overhead). Checkpoint/restore is the fourth door: freeze the process tree, capture RAM as data, and resume it elsewhere bit-identical.

t=0s      IMDS spot/instance-action appears (2-min notice starts)
          |
          v
[detect]  IMDS poller notices instance-action          <= 5.0 s (worst case)
[cordon]  kubectl cordon + migration trigger              ~1 s
[dump]    CRIU checkpoint: freeze + write memory images    7.464 s   } dirty-page
[build]   buildah -D: tar into OCI layer (uncompressed)  13.743 s   } writeback wall
[push]    registry push over VPC                          ~10 s     } at 1 GB
          |
          |  TOTAL EVACUATION (source side): 36.78 s = 30.6% of the 120 s budget
          v
[kill]    delete source pod (gracePeriodSeconds=0)  <---- client blackout starts
[restore] pull OCI layer + CRI-O unpack + CRIU restore  17.328 s   (~8.5 s pull
          first HTTP 200 from restored process          + 4.8 s unpack
                                                      + 3.2 s CRIU restore
                                                      + 0.8 s kubelet sync)
          |
          v
t=~54 s   service serving again, heap identical (SHA-256 verified)
          remaining Spot budget: ~66 s of head room

Read that timeline carefully, because it contains the two design decisions that make or break a Spot-oriented migration system. First, the source container is never stopped for the user during dump, build, and push: CRI-O checkpoints with KeepRunning: true (the forensic use case the API was built for), so the pod keeps serving until the moment you delete it. The client blackout is therefore only the delete-plus-restore tail, not the whole pipeline. Second, the dump is a snapshot: writes the source makes after the dump are not in the checkpoint. The study measured that drift — RPO — at 669 ms median (7.5 plus or minus 0.6 state updates at the workload's 10 Hz write rate) for the decoupled mode. If you need zero drift you must freeze-kill the source first ("stop-and-copy"), which the study models at 1878 ms analytical blackout. Pick your poison per workload; there is no mode that gives you both zero RPO and short blackout.

4. CRIU Under the Hood: What Actually Happens to Your Process

CRIU (Checkpoint/Restore In Userspace, current release v4.3) dumps a process tree from outside the kernel. A dump, mechanically: CRIU ptrace-seizes every process in the container's PID namespace, injects a small piece of parasite code into each process (this is how it gets a reliable vantage point — and why KEP-2008's security analysis flags it), freezes the cgroup, then walks every task and serializes kernel-visible state into image files: the process tree, memory maps, and every memory page read out of the process address space, file descriptors, sockets, signal dispositions. The output is a directory of image files (pages, pstree, mm, fds, sockets, and friends) plus the logs and stats. A restore is the inverse: CRIU re-creates the process tree with clone(), maps the memory images back into place at their original addresses, re-opens and re-points the file descriptors, and resumes execution exactly where each process was seized. The application's own awareness of any of this is a correctness question — time jumps forward, sockets may be dead, peer connections are gone — which is why "checkpoint-aware application" is a real requirement, not a nice-to-have.

The freeze is where your latency budget goes. CRIU must stop the world to get a consistent snapshot, and the wall it hits is not CPU — it is memory bandwidth and, at large heaps, the kernel's dirty-page writeback. The study's scaling data shows it plainly: from 26 MB to 512 MB of dirty heap, the freeze stays sub-second (144 ms to 760 ms) because the pages sit in page cache and CRIU reads them at memory speed. At 1 GB, with the default vm.dirty_background_ratio = 10, the kernel flusher had 783.6 MB of uncommitted dirty pages queued for EBS gp3, and the freeze jumped to 7.464 s. That is a kernel storage-configuration problem wearing a CRIU costume — tune dirty thresholds, provision the gp3 volume's throughput, or keep heaps under the writeback cliff.

kubelet :10250   POST /checkpoint/<ns>/<pod>/<container>  (auth: cluster-admin
   |              or nodes/proxy via apiserver; authz webhook on-node)
   v
CRI gRPC         CheckpointContainerRequest{ container_id, location, timeout }
   |             location = /var/lib/kubelet/checkpoints/
   |             archive name = checkpoint-<pod>_<ns>-<container>-<timestamp>.tar
   |             timeout = 0 -> kubelet CRI default (2 minutes)
   v
CRI-O            server/container_checkpoint.go
   |             lib.ContainerCheckpointOptions{ TargetFile, KeepRunning: true }
   v
OCI runtime      crun/runc checkpoint -> exec's CRIU (v4.3) against the cgroup
   |
   v
CRIU             ptrace-seize all PIDs -> inject parasite -> freeze cgroup
   |             -> read /proc/<pid>/mem + descriptors -> write image files
   |             -> tar: config.dump, spec.dump, dump.log, stats-dump,
   |             rootfs-diff.tar (writable layer delta), bind-mount info
   v
/var/lib/kubelet/checkpoints/checkpoint-<pod>_<ns>-<container>-<ts>.tar
   (mode 0600, root-owned: contains ALL process memory - treat as Secrets)

Two CRIU behaviors decide whether a restored process can talk to the network at all, and they are the root cause of the study's most instructive architectural failure. By default CRIU refuses to dump containers holding established TCP connections (the safe, forensic-grade behavior: the socket cannot survive a cross-machine move, so dumping it would produce a restore that lies). The --tcp-established option lets CRIU capture and re-create established connections — but connection repair only works when the restore lands on the same host and IP, which node-to-node pod migration never is. The --tcp-close option is the honest one for migration: every TCP socket in any state other than CLOSED and LISTEN is restored in the CLOSED state, and the application's next syscall on that socket simply returns an error, which well-written network code interprets as "peer went away, reconnect." Per the CRIU documentation, this is also what lets you restore a server that listens on 0.0.0.0 under a different Pod IP. The study ships this globally on its nodes via /etc/criu/default.conf — a config you will meet again in the production build below.

For completeness: CRIU also has pre-dump (iterative dumping using the kernel's soft-dirty bit so only changed pages transfer on the final pass), a page server (stream pages directly to a remote restore host), and lazy-pages (restore starts before all pages arrive, fetched on demand). These are the primitives real live migration is built from — and none of them are reachable through the Kubernetes checkpoint API today. The kubelet path is a stop-the-world dump to a tar file, full stop. The KEP-5823 non-goals list streaming checkpoint data between nodes as explicitly out of scope for alpha, with the criu-image-streamer project referenced as future plumbing. When a vendor tells you their Kubernetes "live migration" product streams memory between nodes today, ask which kubelet API they are calling — the answer is "none."

5. The Kubelet Checkpoint API: Exactly What You Get

KEP-2008 is titled Forensic Container Checkpointing for a reason, and the scope discipline matters when you build on it: the KEP's own non-goals section says that although checkpoint and restore can be used to implement container migration, the KEP is only about the forensic use case — checkpoint a container out of a running pod without the container noticing, then analyze the copy elsewhere. Everything in this guide is that forensic API being productively abused for migration. The upstream API surface is one HTTP endpoint and one CRI RPC:

• Endpoint: POST /checkpoint/{namespace}/{pod}/{container} on kubelet :10250, with an optional timeout query parameter in seconds (zero/absent means the CRI default — 2 minutes). Checkpoint duration scales with the container's resident memory.
• Responses: 200 archive created; 401 unauthorized; 404 when the ContainerCheckpoint feature gate is off or the ns/pod/container cannot be found; 500 when the CRI implementation hit an error or does not implement the RPC at all (the exact error you get from containerd 1.7 AMIs, which never wired the RPC).
• Gate: ContainerCheckpoint — alpha v1.25, Beta and default-on since v1.30, still Beta (not GA) at v1.37.2. Default-on means recent kubelets expose the endpoint without any flag work; the runtime under them is what actually gates success.
• Output: a tar archive at /var/lib/kubelet/checkpoints/checkpoint-<pod>_<ns>-<container>-<timestamp>.tar (path follows the kubelet --root-dir).
• Source container stays running — CRI-O passes KeepRunning: true to the checkpoint library. The KEP's design goal was that the checkpointed container never knows it was copied. This is what makes the "keep serving while we package" migration mode possible at all.

# Path 1 - through the API server proxy (requires nodes/proxy permission,
# i.e. cluster-admin or a tightly-scoped custom ClusterRole):
kubectl get --raw \
  "/api/v1/nodes/ip-10-10-1-21.eu-central-1.compute.internal/proxy/checkpoint/production/cache-warmer/cache-server?timeout=120"

# Path 2 - on the node (KEP-2008's documented example form), with a
# kubelet client certificate. The kubelet API is normally cluster-admin-only:
curl -sk -X POST \
  --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt \
  --key  /etc/kubernetes/pki/apiserver-kubelet-client.key \
  "https://localhost:10250/checkpoint/production/cache-warmer/cache-server?timeout=120"

# Both paths produce, on the node:
ls -l /var/lib/kubelet/checkpoints/
# -rw------- 1 root root 542M ... checkpoint-cache-warmer_production-cache-server-2026-11-10T08:14:03Z.tar

Three operational truths about this endpoint that the docs bury. First, nothing garbage-collects the archives: a kubelet-side GC was proposed (PR #115888) and deliberately closed — cleanup is an operator problem, and the reference answer is the checkpoint-restore operator with a retention policy. Automate checkpoint triggers without automating cleanup and you will fill the root volume and mark the node NotReady. Second, archive size equals resident memory plus the writable-layer diff — a 1 GB heap produces a ~1 GB tar (the study measured 1029.21 MiB for 1024 MB allocated), so disk headroom and transfer time must be budgeted per checkpoint, not per pod. Third, the endpoint is metrics-covered: kubelet records checkpoint operations in its metrics under the checkpoint bucket, and CRI-level failures surface through kubelet_runtime_operations_errors_total — your Prometheus alerts write themselves (see the production build).

6. Inside the Archive — And Why It Is a Secret

The tar is not an opaque blob; it is a small directory tree, and the checkpointctl tool reads it without a cluster:

$ checkpointctl show /var/lib/kubelet/checkpoints/checkpoint-cache-warmer_production-cache-server-2026-11-10T08:14:03Z.tar

CONTAINER      IMAGE                    ID             RUNTIME  CREATED                  ENGINE  CHKPT SIZE   ROOT FS DIFF SIZE
------------   -----                    --             -------  -------                  ------  ----------   -----------------
cache-server   docker.io/library/python:3.12-slim   68bb2fa6a176   crun    2026-11-10T08:13:58Z   CRI-O   1024.6 MiB   3.5 KiB

$ checkpointctl inspect checkpoint-*.tar
cache-warmer
|- Image: docker.io/library/python:3.12-slim
|- ID: 68bb2fa6a176...
|- Runtime: crun
|- CRIU compression: Disabled
|- Checkpoint data size: 1024.6 MiB
|   '- Memory pages size: 1022.1 MiB
|- Root FS diff size: 3.5 KiB
'- Process tree
    '- [1] python3
        '- [7] gunicorn-worker
            '- [8] gunicorn-worker

Contents, per the CRI-O implementation: config.dump and spec.dump (the OCI runtime metadata CRI-O needs to reconstruct the container), the CRIU image files (memory pages dominate — the 1022 MiB above), dump.log and stats-dump (CRIU's own diagnostics), rootfs-diff.tar (the writable-layer delta against the original image), and the bind-mount inventory. That bind-mount inventory has teeth on restore: CRI-O validates that the restored pod's bind-mount set matches the source, which is why the canonical restore manifest below disables the service-account token mount — the source pod was checked after kubelet injected it, the checkpoint recorded it, and a restore pod with a fresh automounted token will not match.

Now the security paragraph you cannot skip. The archive is a full memory dump of a running process, owned by root with mode 0600, and the KEP-2008 security section says outright that a checkpoint archive can contain secrets which were expected to live in memory only. Session tokens, TLS private keys, decrypted database credentials, your JWT signing key — all of it, in cleartext, inside a file you are about to buildah add into an OCI image and push to a registry. KEP-5823 elevates this to a design rule: checkpoint artifacts must be handled with the same care as Secrets. Concretely, for a production migration pipeline that means: a dedicated registry (or registry namespace) with auth scoped to the migration service, encryption at rest you actually verify, an image retention window measured in minutes, and network policies that keep the checkpoint images unpullable by anything except the migration path. A checkpoint registry that any pod can pull from is a privilege escalation primitive — anyone who can pull the image can read the heap of the process it came from, including its service-account token.

Two more KEP-2008 security facts worth knowing before an audit asks. CRIU's dump works by injecting parasite code into the checkpointed processes — on SELinux systems access to that parasite is limited to the container's own label; elsewhere it is limited to root. And the restored container inherits its SELinux labels and seccomp profile from the checkpoint, so it comes back with the same security posture it left with. The known checkpoint failures are devices: containers with external hardware (GPUs, InfiniBand) cannot be checkpointed without a device-specific CRIU plugin — only AMD GPUs have one today — and the kubelet endpoint returns 500 for them. If your stateful workload is GPU-bound, this entire guide is not for you; skip to the alternatives.

7. The Restore Path: Tar to OCI Image to Running Pod

Here is the part of the pipeline that the Kubernetes API does not help you with at all. Restore is, per KEP-2008, "a container engine only operation in this first step" — the kubelet has no restore endpoint. The working pattern (documented in the kubernetes.io restore walkthrough, used verbatim by the study) converts the checkpoint tar into an OCI image, pushes it to a registry, and creates a pod that references that image. The container runtime — CRI-O only, today — recognizes the image as checkpoint data and restores instead of starting:

#!/usr/bin/env bash
# Package the newest checkpoint archive for pod/container into an OCI image.
# --disable-compression is the single biggest lever: gzip in Buildah is
# single-threaded and serialized the 1 GB checkpoint for 32.06 s in the study;
# uncompressed packaging took 13.74 s (2.3x faster) at the cost of a
# registry-sized tar (1029 MiB for a 1 GB heap).
set -euo pipefail

NS="boutique"
POD="live-stream-pod"
CONTAINER="stream-server"
REGISTRY="10.10.1.10:5000"          # internal registry, in-cluster, over the VPC
TAG="${REGISTRY}/stream-server-checkpoint:$(date -u +%Y%m%dT%H%M%S)"

TAR=$(ls -t /var/lib/kubelet/checkpoints/checkpoint-${POD}_${NS}-${CONTAINER}-*.tar | head -n1)

CID=$(buildah from scratch)
buildah add "${CID}" "${TAR}" /
buildah config \
  --annotation=io.kubernetes.cri-o.annotations.checkpoint.name=${CONTAINER} \
  "${CID}"
buildah commit --disable-compression "${CID}" "${TAG}"
buildah rm "${CID}"
rm -f "${TAR}"

# Push over the VPC. --tls-verify=false only for an internal HTTP registry;
# use a real CA for anything that leaves the VPC.
buildah push --tls-verify=false "${TAG}"
buildah rmi "${TAG}"   # free the source node's disk immediately

The annotation is not decoration — it is the entire detection mechanism. CRI-O's restore path (server/container_restore.go) resolves the image, reads its annotations, and only treats it as checkpoint data when the checkpoint annotation is present. The six annotations CRI-O writes and reads, verbatim from its source:

Two honest warnings from the kubernetes.io blog itself: this image format is pre-alpha and CRI-O-specific ("The resulting image is not standardized and only works in combination with CRI-O"), and restore requires the target node to be able to pull the original base image too — the checkpoint layer only carries the diff, so the study's Ansible pre-pulls python:3.12-slim on every worker. The study also found CRI-O refuses to restore containers under a signature policy and disables signature validation for restore nodes — audit that decision, do not inherit it blindly.

The restored pod manifest is deliberately minimal, and every field in it is load-bearing:

apiVersion: v1
kind: Pod
metadata:
  name: live-stream-pod          # same name: the cutover must be a rename-in-place
  namespace: boutique
  labels:
    app: live-stream             # the Service's selector keeps pointing here
spec:
  automountServiceAccountToken: false   # bind-mount set must match the source pod
  nodeName: k8s-worker-2               # pin the restore node; skip the scheduler race
  tolerations:
    - operator: Exists
  containers:
    - name: stream-server
      image: 10.10.1.10:5000/stream-server-checkpoint:20261110T081403Z
      imagePullPolicy: Always           # never reuse a stale local checkpoint layer
      ports:
        - containerPort: 8090

Deleting the source pod and applying this manifest is the cutover. The blackout clock — measured by the study's 50 Hz prober against the ClusterIP Service — runs from the delete to the first HTTP 200 from the restored process, and it decomposes into four phases with weights worth memorizing (1 GB case): registry pull over the VPC ~8.5 s (instance-bandwidth-bound), CRI-O layer extraction and unpack ~4.8 s, CRIU restore and unfreeze ~3.2 s, kubelet status reporting ~0.8 s. In other words, at 1 GB, three quarters of the blackout is image logistics, and only 18% is CRIU. Optimizing CRIU flags on a slow migration is tuning the wrong third of the pipeline.

The study's most instructive failure is architectural, and it is a direct consequence of the API's container-level granularity. They deployed a realistic two-container pod: stream-server listening on 127.0.0.1:8090 plus an Envoy sidecar on 0.0.0.0:15006 forwarding over loopback — the textbook service-mesh shape. Checkpointing only stream-server and restoring it on another node severed the loopback peering: CRIU's tcp-close restored the loopback connections CLOSED, the sidecar stayed on the source node, and the restored pod never answered — measured cutover was infinite, not slow. KEP-2008 checkpoints one container at a time; there is no pod-wide consistent cut in the HTTP API today. This is precisely the gap KEP-5823 exists to close (pod-wide freeze via the pod cgroup freezer, all containers in one checkpoint, sidecars restored running), and until it ships, the study's recommendation stands: sidecar-heavy pods must be redesigned so L4/L7 routing happens at the node level (Cilium eBPF host routing, kube-proxy ClusterIP) rather than over pod-internal loopback, or they simply cannot be migrated by this mechanism.

8. The Empirical Evidence, Read Critically

What makes this study worth citing instead of yet another vendor demo is the methodology: dual independent clocks (the operator's own status reporting and an external 50 Hz HTTP prober sampling every 20 ms through the ClusterIP Service), 56 paired runs persisted as a raw CSV (the first row is real: freeze 144 ms, build 371 ms, push 299 ms, restore 838 ms, operator blackout 838 ms, external 859 ms, paired difference +21 ms), a SHA-256 content digest of the restored heap compared across nodes, and statistics reproducible from the CSV with a standalone script. The testbed is modest — three m7i-flex.large/c7i-flex.large instances (2 vCPU, 8/4 GiB), Ubuntu 24.04, kernel 6.8 — which is the point: you do not need exotic hardware to reproduce it, and you should before trusting any number for your workload.

The three-mode comparison is the decision table for real deployments. The study measured Mode A (decoupled: source keeps serving through dump/build/push, cutover at the end) and native cold reschedule, and analytically projects Mode B (stop-and-copy: freeze-kill the source before packaging) from the measured component latencies:

The statistical honesty in the paper is itself a finding: the CRIU-vs-cold difference (+45.5 ms median, p=0.066, Welch's t=1.91) is underpowered against the PLEG-induced noise, and the authors say so rather than claiming a win. Two practical readings. First, if you were hoping checkpoint/restore would beat a cold restart on raw cutover time at small heaps: it does not, and the data says stop hoping. Second, the noise floor itself is actionable — the variance is dominated by the kubelet's 1-second PLEG relist cycle, which means your measured blackouts will be dominated by it too, and the fix is a kubelet-side feature (Evented PLEG), not anything in the migration path. That is the next section.

One number the study measured that marketing decks will never quote: the paired operator-vs-external lag. The external prober consistently saw recovery ~15.8 ms (CRIU) to ~18.5 ms (cold) after the operator declared success, ranging -12 to +48 ms across runs — the residual is Service routing propagation on Cilium eBPF host routing plus the sampling convolution of the 50 ms poll and 20 ms probe (theoretical ~15.5 ms, matching). When you build migration tooling, measure blackout at the Service endpoint, not at your operator's status field, or you will systematically under-report by double-digit milliseconds.

9. PLEG: The One-Second Floor Under Every Cutover

The Pod Lifecycle Event Generator is the kubelet's runtime-state poller, and its mechanics put a hard floor under how fast any pod transition — migration included — becomes visible to the control plane and to your tooling. The constants, verified in the v1.37.2 source (pkg/kubelet/kubelet.go): the generic PLEG relists every 1 second (genericPlegRelistPeriod = time.Second * 1), reports unhealthy if a relist is more than 3 minutes stale (genericPlegRelistThreshold = time.Minute * 3 — the famous "PLEG is not healthy" node condition), and a single relist can itself exceed 1 s when the runtime is slow or many containers change state at once. PLEG polls ListContainers across every pod on the node, diffs against its cache, and emits PodLifecycleEvents; kubelet workers then reconcile. The consequence for migration: after your source pod dies and your restore pod starts, kubelet's view of the world advances on 1-second ticks. Cutover detection is therefore uniform on [0, 1 s), and the study's measured sigma (~241 ms across 36 runs) sits right under the theoretical uniform-distribution sigma of 288.7 ms. Your migration operator can be perfect and your blackouts will still jitter by a second.

GENERIC PLEG (v1.37.2 default, always on)
 t=0.0s   t=1.0s   t=2.0s   t=3.0s   t=4.0s   t=5.0s
  |---relisterelisterelisterelisterelisterelisterelisterelis--> 1 s period
                    ^                 ^
              container dies somewhere in [2.0, 3.0)
              next relist sees it: detection latency = uniform [0, 1 s)
              sigma = 1000/sqrt(12) ~ 288.7 ms  (measured: ~241 ms)

EVENTED PLEG (gate EventedPLEG; alpha at v1.37.2, beta target v1.38)
 CRI GetContainerEvents stream
   |
   |- CONTAINER_STOPPED_EVENT(pod UID)   <-- from the runtime, best-effort
        |
        v
   EventedPLEG watcher -> GenericPLEG.RequestRelist(podUID) -> on-demand
   relist of that pod only (bounded queue, coalesced while SyncPod runs,
   global relist keeps running at 1 s as the source of truth)

 Detection latency: ~event delivery (ms-scale) instead of up to 1 s.
 Correctness unchanged: events are hints; GenericPLEG still reconciles.

Evented PLEG (KEP-3386) is the fix, and its history is a production lesson in itself. It went Alpha in v1.26 (2022), Beta in v1.27, and was reverted to Alpha in v1.30 and backported after it broke static pods (kubernetes #121349, #121003) — the earlier design let the CRI event stream write pod-cache state directly, and an edge-triggered stream with no replay guarantees is a terrible source of truth. The v1.36/v1.37 redesign (PRs #137362, #139262) abandoned that: Evented PLEG now only watches the new GetContainerEvents CRI stream and, on a valid CONTAINER_STOPPED_EVENT carrying the pod UID, asks Generic PLEG to relist that one pod immediately. Generic PLEG keeps polling at 1 s and remains the only component that reads runtime state, updates the cache, and emits lifecycle events; events that are lost, duplicated, or stale merely cost you the latency optimization, never correctness. The gate is still Alpha-default-off at v1.37.2 and v1.38.0-beta.0 (checked in the feature-gate registry of both tags), with Beta re-targeted at v1.38 per the KEP — so treat "Evented PLEG fixes my migration jitter" as a v1.38+ story, one that also requires runtime support (CRI-O implements the events stream; enable it with enable_pod_events = true in CRI-O config) and kubelet restarts to take the gate.

While you wait, the practical mitigations for the 1-second floor are the ones the study used: measure cutover where it matters (external prober through the Service), size your SLOs with the ~290 ms sigma in mind, and remember the floor applies equally to cold reschedules — it is a reason to stop blaming (or crediting) the migration mechanism for second-scale jitter.

10. The eBPF Angle: Routing Resilience, Not State Migration

The "eBPF" in this story is smaller than the headline suggests, and being precise about it saves you from building the wrong thing. What the study actually used Cilium 1.16 for is host-routing: service translation and pod networking implemented as BPF programs on the host, with no iptables rules in the datapath. That choice pays off twice in a migration pipeline. First, service endpoints converge fast on cutover — the measured external-prober lag over the ClusterIP Service was tens of milliseconds. Second, and more structurally: node-level routing decouples L4/L7 traffic steering from the pod's network namespace, which is the study's recommended answer to the sidecar problem. When your pod's entire serving path is "sidecar on loopback," the pod is unmigratable at container granularity, because the checkpoint carries the loopback sockets as already-dead connections. When routing happens on the host (ClusterIP via Cilium or kube-proxy, ingress via node-level proxies), the restored pod just needs its new Pod IP registered — which kubelet and the CNI do as part of ordinary pod bring-up.

What eBPF does not do here is move connections or memory. A BPF program cannot carry a TCP socket across nodes; that requires CRIU's connection-repair machinery (same-host only today) plus CNI-level IP preservation, which KEP-5823 explicitly defers to a future live-migration KEP after SIG Network confirmed feasibility but significant work. So the honest architecture statement is: eBPF is your routing plane insurance for the post-restore world; CRIU is your state plane; nothing bridges live connections. Every client of a migrated pod will reconnect. Design your clients (and your load balancer keep-alive timeouts) accordingly, or your measured blackout will include a long tail of stale-connection retries that no migration tuning can fix.

11. The AWS Production Build

Everything below is the minimum viable migration stack on EC2-backed nodes, assembled from the study's published automation and the upstream configs, with every knob explained. Topology: one internal registry reachable over the VPC (the study runs registry:2 as a hostNetwork pod on the control-plane node — for production, use ECR with pull-through or a dedicated internal registry, and think hard about the secrets-in-images problem from section 6 first), plus a privileged migration DaemonSet on every worker.

# /etc/criu/default.conf - global CRIU options, applied to every dump.
# Reference: https://criu.org/CLI/opt/--tcp-close
#
# tcp-close      Restore every TCP socket that is not CLOSED/LISTEN as CLOSED.
#                This is what lets a server bound to 0.0.0.0 come back under a
#                new Pod IP; the app sees a network error and reconnects.
#                Do NOT use --tcp-established for cross-node restore: connection
#                repair only works on the same host and IP.
# shell-job      Restore attached shells/jobs safely.
# ext-unix-sk    Checkpoint external Unix sockets.
# skip-in-flight  Do not dump TCP connections in flight (avoids dump failures).
# skip-mnt       Paths that must not enter the checkpoint archive.
tcp-close
shell-job
ext-unix-sk
skip-in-flight
skip-mnt /run/.containerenv

# /etc/crio/crio.conf.d/99-criu.conf
#
# CRI-O v1.31 (what the study ran): the single switch is enable_criu_support,
# and it requires the criu binary in $PATH on the node.
[crio.runtime]
enable_criu_support = true

# CRI-O current (v1.37.x) replaces the flag with a table; the equivalent is:
#   [crio.checkpoint_restore]
#   container_level_enabled = "checkpoint_restore"
# "checkpoint_only" is the default (dumps work, restores are refused);
# you need "checkpoint_restore" to boot pods from checkpoint images.
#
# If you also want Evented PLEG's fast path (K8s 1.38+, CRI events):
#   enable_pod_events = true

If your nodes run containerd instead (EKS default), the requirements differ in ways that matter: containerd only wired the CRI checkpoint RPC in March 2024 (PR #6965, shipping in the 2.x line) — a 1.7-era AMI implements nothing and every checkpoint call returns 500. On containerd 2.x, CRIU support is gated by enable_criu in the CRI runtime plugin config (default-on in the stock 2.1 config, but verify yours was not flipped), and containerd runs a startup probe: it resolves the criu binary in the shim path or system PATH and fails with criu binary not found in shim path or system PATH or a version-floor error — so install CRIU v4.3 before the first checkpoint, not after the first incident. The containerd restore story is also not CRI-O's annotation-image path; treat containerd nodes as checkpoint-source-only until you have tested your specific versions end to end. The study standardized on CRI-O precisely because its checkpoint-image restore path is the one upstream documented.

#!/bin/bash
set -euo pipefail

# ContainerCheckpoint is Beta default-on since K8s v1.30: no kubelet flag
# work needed on modern EKS. The node needs: (1) the criu binary, (2) for
# containerd nodes, the CRI plugin un-gated, (3) the original base images
# pre-pulled so restores do not race the registry.

# 1. CRIU v4.3 (AL2023 ships criu; verify, and pin v4.3 if you need parity
#    between dump and restore hosts):
if ! criu --version 2>/dev/null | grep -q "4.3"; then
  dnf install -y criu || {
    dnf install -y make gcc git protobuf-c-devel protobuf-devel \
      libnl3-devel libcap-devel nftables-devel libaio-devel libbsd-devel
    git clone --branch v4.3 --depth 1 https://github.com/checkpoint-restore/criu /opt/criu
    make -C /opt/criu -j"$(nproc)" && make -C /opt/criu install
  }
fi

# 2. containerd: explicit CRIU enablement via a config import (containerd 2.x).
#    On the stock EKS AMI, /etc/containerd/config.toml has no imports line.
mkdir -p /etc/containerd/conf.d
grep -q "^imports" /etc/containerd/config.toml || \
  printf '\nimports = ["/etc/containerd/conf.d/*.toml"]\n' >> /etc/containerd/config.toml
cat > /etc/containerd/conf.d/99-criu.toml <<'EOF'
version = 2

[plugins.'io.containerd.cri.v1.runtime']
  enable_criu = true
EOF
systemctl restart containerd

# 3. Dirty-page tuning for large heaps (the 1 GB freeze wall was 7.464 s with
#    dirty_background_ratio=10 on gp3; see section 4). Size to your volumes:
sysctl -w vm.dirty_background_ratio=5
sysctl -w vm.dirty_ratio=15

# 4. Pre-pull the base images your checkpointable pods run on (restore needs
#    the original rootfs present locally to apply rootfs-diff.tar against):
crictl pull docker.io/library/python:3.12-slim

The migration job itself, in the shape the study validated — a privileged DaemonSet that talks to the local kubelet, packages with nsenter into the host's Buildah, and drives the cutover through the API. The privileges here are real (hostPID, nsenter, cluster-admin for the kubelet call); they are also the reason KEP-5823's no-privileged-endpoint design exists. Run this in a locked-down namespace and treat the DaemonSet as cluster-admin-equivalent:

#!/usr/bin/env bash
# migrate.sh - runs INSIDE the privileged migration DaemonSet pod on the
# SOURCE node (hostPID + nsenter reach the host's buildah and kubelet dir).
# Env: NS, POD, CONTAINER, TARGET_NODE, REGISTRY, MY_NODE, KUBERNETES_* (SA).
set -euo pipefail

NS="${NS:-boutique}"; POD="${POD:-live-stream-pod}"
CONTAINER="${CONTAINER:-stream-server}"
TARGET_NODE="${TARGET_NODE:?}"; REGISTRY="${REGISTRY:-10.10.1.10:5000}"
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
API="https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"
TAG="${REGISTRY}/${POD}-checkpoint:$(date -u +%Y%m%dT%H%M%S)"

host() { nsenter -t 1 -m -u -i -n bash -c "$1"; }
k8s()  { curl -sS --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
           -H "Authorization: Bearer ${TOKEN}" "$@"; }

# --- Stage 1: CRIU dump via the local kubelet (KeepRunning: source serves on)
host "rm -f /var/lib/kubelet/checkpoints/checkpoint-${POD}_${NS}-${CONTAINER}-*.tar"
host "curl -sS -f -m 120 -k -X POST \
  -H 'Authorization: Bearer ${TOKEN}' \
  https://localhost:10250/checkpoint/${NS}/${POD}/${CONTAINER}"

# --- Stage 2+3: package the tar into an OCI image, push over the VPC
host "set -e
  TAR=\$(ls -t /var/lib/kubelet/checkpoints/checkpoint-${POD}_${NS}-${CONTAINER}-*.tar | head -n1)
  CID=\$(buildah from scratch)
  buildah add \"\${CID}\" \"\${TAR}\" /
  buildah config --annotation=io.kubernetes.cri-o.annotations.checkpoint.name=${CONTAINER} \"\${CID}\"
  buildah commit --disable-compression \"\${CID}\" ${TAG}
  buildah rm \"\${CID}\"; rm -f \"\${TAR}\"
  buildah push --tls-verify=false ${TAG}
  buildah rmi ${TAG}"

# --- Stage 4: cutover - delete source pod, create the restored pod pinned
k8s -X DELETE "${API}/api/v1/namespaces/${NS}/pods/${POD}?gracePeriodSeconds=0"
cat <<EOF | k8s -X POST -H "Content-Type: application/json" \
  --data-binary @- "${API}/api/v1/namespaces/${NS}/pods" >/dev/null
{
  "apiVersion": "v1", "kind": "Pod",
  "metadata": {"name": "${POD}", "namespace": "${NS}",
               "labels": {"app": "live-stream"}},
  "spec": {
    "automountServiceAccountToken": false,
    "nodeName": "${TARGET_NODE}",
    "tolerations": [{"operator": "Exists"}],
    "containers": [{"name": "${CONTAINER}", "image": "${TAG}",
                    "imagePullPolicy": "Always",
                    "ports": [{"containerPort": 8090}]}]
  }
}
EOF

# --- Stage 5: wait for the restored process to answer on the target node
DEADLINE=$(( $(date +%s) + 120 ))
while [ "$(date +%s)" -lt "${DEADLINE}" ]; do
  IP=$(k8s "${API}/api/v1/namespaces/${NS}/pods/${POD}" \
        | sed -n 's/.*"podIP": *"\([^"]*\)".*/\1/p' | head -n1)
  if [ -n "${IP}" ] && host "curl -sf -m 1 http://${IP}:8090/live/state" ; then
    echo "migration complete: ${POD} serving from ${TARGET_NODE} (${IP})"
    exit 0
  fi
  sleep 0.2
done
echo "migration FAILED: restored pod did not answer within 120s" >&2
exit 1

The Spot trigger that feeds it is a two-second IMDS poller — the same contract the study's operator used, implemented with the exact IMDSv2 flow from the AWS docs. Deploy as a DaemonSet with a small RBAC grant (label its own node) and let your migration controller watch the spot-interruption=true label:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: spot-watcher
  namespace: migration-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: spot-watcher
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["patch", "get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: spot-watcher
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: spot-watcher}
subjects: [{kind: ServiceAccount, name: spot-watcher, namespace: migration-system}]
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: spot-watcher
  namespace: migration-system
spec:
  selector: {matchLabels: {app: spot-watcher}}
  template:
    metadata: {labels: {app: spot-watcher}}
    spec:
      serviceAccountName: spot-watcher
      hostNetwork: true
      tolerations: [{operator: Exists}]
      containers:
        - name: watcher
          image: registry.k8s.io/bitnami/kubectl:latest
          command: ["/bin/bash", "-c"]
          args:
            - |
              set -euo pipefail
              while true; do
                TOKEN=$(curl -sX PUT "http://169.254.169.254/latest/api/token" \
                        -H "X-aws-ec2-metadata-token-ttl-seconds: 30")
                ACTION=$(curl -sf -H "X-aws-ec2-metadata-token: ${TOKEN}" \
                        "http://169.254.169.254/latest/meta-data/spot/instance-action" || true)
                if [ -n "${ACTION}" ]; then
                  echo "spot interruption notice: ${ACTION} - labeling node"
                  kubectl label node "${NODE_NAME}" spot-interruption=true --overwrite
                  exit 0   # node is dying; stop polling
                fi
                sleep 2
              done
          env:
            - name: NODE_NAME
              valueFrom: {fieldRef: {fieldPath: spec.nodeName}}

Monitoring: the signals are all already exported. Kubelet's checkpoint endpoint lands in its HTTP metrics under the checkpoint bucket; CRI failures surface in runtime_operations_errors_total; PLEG health has dedicated gauges and histograms; and archive accumulation is plain filesystem math on /var/lib/kubelet. Names below are as registered in pkg/kubelet/metrics at v1.37.2 — if your kubelet version prefixes them with kubelet_, adjust:

groups:
  - name: criu-migration
    rules:
      - alert: CheckpointOperationFailing
        expr: sum by (node) (increase(runtime_operations_errors_total{operation_type=~"checkpoint.*"}[10m])) > 0
        for: 5m
        labels: {severity: warning}
        annotations:
          # use the alert instance label (node) in your own templating stack;
          # go-template braces are omitted here because the article template
          # engine consumes them
          summary: "kubelet checkpoint CRI calls failing - inspect the node label"

      - alert: CheckpointDiskFilling
        expr: |
          (node_filesystem_avail_bytes{mountpoint="/var/lib/kubelet"}
            / node_filesystem_size_bytes{mountpoint="/var/lib/kubelet"}) < 0.20
        for: 10m
        labels: {severity: critical}
        annotations:
          summary: "kubelet root below 20% - checkpoint archives are consuming it"

      - alert: PLEGRelistSlow
        expr: histogram_quantile(0.99, sum by (le) (rate(pleg_relist_duration_seconds_bucket[5m]))) > 1
        for: 10m
        labels: {severity: warning}
        annotations:
          summary: "PLEG relist p99 above 1s - lifecycle detection is degrading"

      - alert: PLEGUnhealthy
        expr: (time() - pleg_last_seen_seconds) > 180
        labels: {severity: critical}
        annotations:
          summary: "PLEG not healthy - relist stale beyond the 3-minute threshold"

12. What Lands Next: KEP-5823 Pod-Level Checkpoint/Restore

KEP-5823 is the upstream answer to nearly every structural complaint above, and reading its design is the best way to understand what the current approach is missing. The alpha scope is warm start and fault tolerance, same-node restore only — but the API shape is deliberately built to grow into migration. The design replaces the privileged HTTP endpoint with declarative objects: a first-class PodCheckpoint resource (in checkpoint.k8s.io/v1alpha1) that the kubelet watches, and a restore driven by a new spec.restoreFrom field on the Pod. Nobody — not even the admin — needs kubelet access or nodes/proxy.

CHECKPOINT (declarative, watch-driven)
  user:  kubectl apply PodCheckpoint{spec.sourcePodName, sourcePodUID?, timeoutSeconds?}
   |
   v
  kubelet on the owning node (watches PodCheckpoints, filters its own pods):
    1. per-pod in-flight guard (no double checkpoint, no overlap with restore)
    2. validate: pod bound, non-restartable init done, containers running
    3. pin instance: live Pod UID must equal spec.sourcePodUID
    4. suspend probes (frozen pods must not be killed mid-checkpoint)
    5. CRI CheckpointPod{pod_sandbox_id, path} -> freeze pod cgroup,
       capture ALL containers as one consistent cut, resume (RUNNING)
    6. archive at /var/lib/kubelet/pod-checkpoints/checkpoint-<pod>_<ns>-<ts>/
    7. write status: Ready condition, checkpointLocation{type: NodeLocal},
       checkpointedPodTemplate (sanitized), checkpointedContainers
   |
   v
  pod-snapshot-controller: lifecycle only (finalizers, GC, future transport);
  NOT on the execution path

RESTORE (normal pod admission + scheduling)
  user:  Pod{spec.restoreFrom: my-checkpoint}
   |
   v
  API server: PodRestoreAuthorization admission plugin
    - authorize "restore" verb on the PodCheckpoint (separate from create/get)
    - inject node affinity pinning to the checkpoint's node (scheduler binds it)
    - enforce pod-spec equality vs status.checkpointedPodTemplate
   |
   v
  kubelet SyncPod: sees restoreFrom -> restorePodSandbox() instead of create
    - re-check spec equality + node match + path confinement
    - CRI RestorePod{checkpoint_path, config, container_configs}
    - restored containers come up RUNNING; start steps skipped

apiVersion: checkpoint.k8s.io/v1alpha1
kind: PodCheckpoint
metadata:
  name: myapp-warm-01
  namespace: team-a
spec:
  sourcePodName: myapp               # immutable after creation
  sourcePodUID: 7b2c1e4a-0e3a-4f1b-9c2d-2a5f6e8d1234   # instance pin
  timeoutSeconds: 30
status:
  nodeName: node-1
  checkpointLocation:
    type: NodeLocal
    nodeLocal: {path: checkpoint-myapp_team-a-2026-03-10T20:38:11Z}
  completionTime: "2026-03-10T20:38:12Z"
  checkpointedPodTemplate:
    metadata: {labels: {app: my-app}}
    spec:
      containers:
        - name: main
          image: registry.example.com/my-app:v1.4.0
  conditions:
    - type: Ready
      status: "True"
      reason: CheckpointCompleted
---
apiVersion: v1
kind: Pod
metadata:
  name: myapp-restored
  namespace: team-a
spec:
  restoreFrom: myapp-warm-01   # admission injects node affinity; do not set nodeName
  containers:
    - name: main
      image: registry.example.com/my-app:v1.4.0   # must match the template exactly

Status on the ground as of v1.38.0-beta.0, verified in the tree: the CheckpointPod/RestorePod CRI RPCs are merged in the v1.37 CRI proto, but the PodLevelCheckpointRestore feature gate, the PodCheckpoint API type, and the admission plugin are not yet in the v1.38 beta tree — the KEP (opened 2026-01-29, stage alpha, latest milestone v1.38) is still landing. Plan against 1.38 final or later, and expect the alpha to be useful for warm-start (its primary driver: checkpoint an initialized LLM inference server or JVM once, boot replicas in seconds) before it is useful for migration.

The alpha's hard limits are the checklist for what your migration design still has to own: cross-node restore (and therefore migration) is out of scope — a checkpoint's data lives on the node that took it, and restore is rejected with CheckpointWrongNode elsewhere; all established TCP connections are closed on checkpoint (TCP-close, exactly the study's config); shared pod resources (shared memory, volumes) and devices (GPUs, DRA claims) are not captured; exec sessions and port-forward are not preserved; service-account tokens in the checkpoint may be expired at restore (the KEP tells checkpointable workloads to disable automount and refresh tokens explicitly); and the kubelet currently watches PodCheckpoint objects cluster-wide — flagged in the KEP itself as a scalability follow-up (node-scoped field-selector routing comes later). Two Beta blockers are worth knowing because they are the upstream admission that the disk problem is real: kubelet-owned checkpoint GC (counting checkpoint storage toward node disk pressure) and PodCheckpoint object GC/TTL in etcd.

The security model is the most interesting part of the design and directly addresses the privilege escalation the current container-level pipeline forces on you: PodCheckpoint is namespace-scoped and can only target pods in its own namespace; restore authorization is a dedicated restore verb on the named checkpoint (so you can let a team consume warm starts without letting them create checkpoints); the status template used for spec-equality is kubelet-written and user-immutable, so a user cannot forge the record a restore validates against; and the kubelet's status writes are constrained by the Node authorizer plus NodeRestriction to checkpoints whose source pod is bound to its own node. It is a genuinely better design than "give the DaemonSet cluster-admin and nsenter into PID 1" — which is what you are stuck with today.

13. Production Realities: Where This Breaks

• The registry becomes a secrets store. Every checkpoint image is a full heap dump. Treat the checkpoint registry like a Secrets backend: scoped credentials, short retention, pull restricted to the migration path, encrypted at rest. The moment a developer can docker pull yesterday's heap of the auth service, you have built an exfiltration machine.
• Costs scale with RAM, per migration. A 1 GB heap means ~1 GB written to node disk, ~1 GB pushed over the VPC, ~1 GB stored in the registry, ~1 GB pulled and unpacked on the target. Migrate a pod hourly and that is ~24 GB/day of egress and storage churn for one pod — plus EBS write amplification on both ends. The study's Spot budget math holds at 1 GB on m7i-flex.large; a 4 GB heap will not fit in 120 seconds on that hardware without pre-dump machinery the Kubernetes path does not expose.
• PLEG sets a ~1 s detection floor and ~290 ms sigma on every pod transition you instrument until Evented PLEG ships — and Evented PLEG carries its own history (the v1.30 static-pod revert). Test the gate on a canary node pool first, and remember runtime support is required (enable_pod_events on CRI-O).
• Every TCP connection dies. Clients see sockets reset and must reconnect. That is usually fine (well-behaved clients do), but load balancers with long keep-alive, gRPC long streams, and WebSocket sessions all convert your 1-second blackout into a 30-second tail of stale-connection retries if the client stack does not handle RST promptly.
• Sidecars break the whole model at container granularity. Any pod whose serving path traverses a loopback sidecar (service meshes, most injection proxies) cannot be migrated container-by-container — the study measured a literally infinite blackout for the Envoy sidecar case. Redesign to node-level routing, wait for KEP-5823's pod-wide cuts, or do not migrate those pods.
• Version skew is a restore-time bomb. CRIU restores are only guaranteed against a compatible kernel and matching CRIU version (the checkpoint image even carries the CRIU version as an annotation so you can diagnose skew). A fleet with heterogeneous kernel minors, or a node group rebuilt on a new AMI mid-incident, will produce restore failures at the worst possible moment. KEP-5823's open questions list CPU architecture, kernel version, and CRIU/gVisor version matching as unsolved scheduling inputs.
• GPUs and InfiniBand are hard-out. No CRIU device plugin, no checkpoint — 500 at the kubelet. Only AMD GPUs have a plugin. If the stateful pod is your inference server holding a loaded model on GPU memory, this is not your mechanism; the KEP-5823 family explicitly defers devices to a follow-on.
• Nothing garbage-collects today. Kubelet never cleans /var/lib/kubelet/checkpoints; the in-tree GC was proposed and closed. Automate cleanup in the same code path that triggers checkpoints (the script above deletes the tar after packaging; the checkpoint-restore operator adds retention), and alarm on the kubelet filesystem. One forgotten cleanup path is a NotReady node at 3 a.m.
• Freeze-window disruption is unsolved upstream. During the CRIU dump the pod is frozen and unreachable; the kubelet suspends probes so it does not kill it, but nothing tells your StatefulSet, PDB, or descheduler to leave it alone — KEP-5823 documents the Checkpointing=True condition as observability-only and defers the "do not disrupt / do not route" signal to a Beta blocker. Keep checkpoints short and off pods under disruption pressure.
• The study's caveats are your caveats. N=36 on one three-node testbed, one workload shape, one kernel, p=0.066 — the authors are careful not to overclaim, and you should be more careful still: reproduce on your instance types, your kernel, your heap shape, before you bet an SLO on these numbers.

14. Alternatives: What to Do Instead (or First)

The decision rule that falls out of the data: if the application can serialize its own state, let it — app-level checkpointing is more portable, more testable, and survives kernel and node churn that CRIU cannot. CRIU-based migration earns its complexity only when the state is genuinely opaque (you cannot rewrite the legacy service, you cannot pause training long enough to re-warm) and the workload is single-container or node-routed. And if what you actually need is "never lose the process at all," the answer is a warm standby or a VM, not a faster copy.

15. Who Should Skip This

• GPU-attached workloads. Checkpointing fails at the device; no amount of pipeline engineering fixes that today. Use framework-level checkpointing for training, KV-cache offload for inference.
• Pods with service-mesh sidecars on loopback. At container granularity your migration will not just be slow, it will hang forever — the restored process waits on a peer that no longer exists on that node.
• Long-lived connection-bound protocols you cannot reconnect. If your clients are embedded devices that hold one TCP session for days, connection death is a bigger outage than any cold start.
• Heterogeneous or fast-churning node fleets. Kernel/CRIU skew is a restore failure; if your node groups drift weekly, CRIU restores will fail exactly when Spot is busiest.
• Regulated environments where a heap dump in a registry is a finding. The archive contains every secret the process ever decrypted. If your compliance regime treats memory dumps as in-scope data, the registry story has to be built before the pipeline is.
• Anyone who cannot own node images. No CRIU in the AMI means the endpoint returns 500; if you are on a managed platform where you cannot install packages or touch runtime config, this feature simply is not available to you.
• Teams hoping to beat cold-restart latency. The measured data says you will not. If your only goal is faster pod moves, spend the engineering on startup optimization and PBD tuning instead.

16. References & Further Reading

• KEP-2008: Forensic Container Checkpointing — the endpoint spec, CRI RPC, security analysis (parasite code, secrets in archives), GC decision, alpha/beta timeline
• KEP-5823: Pod-Level Checkpoint/Restore — PodCheckpoint object, declarative restore, alpha non-goals (cross-node, TCP, devices), security model, Beta blockers
• KEP-3386: Kubelet Evented PLEG — GetContainerEvents design, the v1.27 Beta and v1.30 revert history, the v1.36/v1.37 termination-hint redesign
• Kubernetes kubelet Checkpoint API reference — POST /checkpoint/{namespace}/{pod}/{container}, response codes, timeout semantics
• Kubernetes blog: Forensic container checkpointing (alpha) — the canonical buildah-to-registry restore walkthrough, CRI-O-only image format warning
• Kubernetes blog: Forensic container analysis — checkpointctl-based analysis workflow
• Khasmammadov: Evaluating Checkpoint/Restore-Based Pod Migration in Kubernetes (v1.3.0) — the empirical study: 56-run dataset, statistical methodology, AWS EC2 testbed, heap scaling, Spot budget analysis
• Study dataset (Zenodo DOI 10.5281/zenodo.23741721) — benchmark_56_runs.csv and reproduction scripts
• CRIU project documentation — dump/restore mechanics, pre-dump, page server, lazy-pages
• CRIU --tcp-close reference — exact semantics: all TCP sockets except CLOSED/LISTEN restored CLOSED
• CRI-O configuration reference — enable_criu_support (deprecated), crio.checkpoint_restore.container_level_enabled, enable_pod_events
• CRI-O checkpoint annotations (source) — the six io.kubernetes.cri-o.annotations.checkpoint.* keys, verbatim
• containerd CRI configuration — enable_criu default and the CRIU presence/version check
• containerd PR #6965: Wire through CRI checkpoint RPC — merged March 2024; containerd 1.7 has no CheckpointContainer
• AWS EC2 User Guide: Spot Instance interruptions — the two-minute notice, IMDS instance-action, EventBridge events
• checkpointctl — show/inspect/diff/build for checkpoint archives
• checkpoint-restore operator — retention and garbage collection for kubelet checkpoint archives
• Kubernetes CRI API (v1.37.2) — CheckpointContainer, CheckpointPod, RestorePod RPC definitions
• Hacker News thread for the study (item 50280010) — the original lead, October 28, 2026