For years, the Kubernetes operational mantra has been "cattle, not pets." If a pod fails or a node needs patching, you kill the pod and let the scheduler land a fresh instance elsewhere. But for stateful workloads—complex AI model weights in RAM, high-performance caches, or long-running computations—a cold restart is a performance catastrophe.
The introduction of the Kubelet Checkpoint API (KEP-2008) and the integration of CRIU (Checkpoint/Restore In Userspace) changes the game. We can now capture the exact execution state of a container—memory, registers, and file descriptors—and "hibernate" it across the cluster.
The Architectural Flow
Migrating a stateful pod across AWS EC2 nodes is not a single API call; it is an orchestrated pipeline involving the Kubelet, the container runtime (containerd), and an external transport layer (S3/ECR).
CRIU does not "snapshot" the disk; it snapshots the process. When the Kubelet triggers a checkpoint, CRIU performs the following operations:
Process Freezing: It sends a signal to the process to stop execution, ensuring a consistent state.
Memory Dumping: It iterates through the process's virtual memory mappings, dumping the heap, stack, and mmap'd regions into pages-1.img.
Context Capture: CPU registers, thread state, and signal masks are saved to core-1.img.
Descriptor Mapping: Open file descriptors and network socket states are recorded in files.img.
PLEG: The 1-Second Gatekeeper
The Pod Lifecycle Event Generator (PLEG) is the Kubelet's internal loop that polls the container runtime for status changes. This is the primary source of latency during a migration cutover.
When a pod is restored, the Kubelet must recognize the new container as Running. Because PLEG typically operates on a 1-second relist cycle, the "cutover blackout"—the time between the source being killed and the target being ready—is dominated by this cycle. In production AWS environments, the median cutover is often ∼1.1 seconds, regardless of the checkpoint size.
AWS Implementation Blueprint
To enable this on Amazon EKS (AL2023), you must overcome the fact that CRIU is not pre-installed on worker nodes and the kernel restricts process attachment by default.
1. Node Preparation
You must deploy a DaemonSet to install CRIU and tune the ptrace_scope. Without kernel.yama.ptrace_scope=0, CRIU cannot attach to the target process to dump its memory.
The Kubelet exposes a node-local proxy endpoint. An orchestrator agent (running as a privileged pod or via SSH) must call this endpoint to trigger the dump.
# Trigger checkpoint for container 'main-app' in pod 'stateful-app-1'
curl -X POST "https://<node-ip>:10250/checkpoint/default/stateful-app-1/main-app" \
--header "Authorization: Bearer <token>" \
--insecure
The resulting archive is written to /var/lib/kubelet/checkpoints/. This tarball contains the full memory state of the process. In an AWS production pipeline, this artifact should be immediately streamed to an S3 bucket or an ECR repository to be accessible by the target node.
3. Solving the IP Preservation Problem
Standard Kubernetes assigns a new IP to every pod. A restored pod on a new node will have a different IP, breaking all active client connections. To achieve "Live" migration, you must decouple network identity from the pod instance.
Kube-OVN Approach: Use the ovn.kubernetes.io/ip_address annotation to force the restored pod to inherit the source pod's IP.
Cilium eBPF Approach: Use eBPF host-routing to map the service traffic to the restored pod's new IP transparently at the kernel level, avoiding a full IP migration.
Production Realities
The Sidecar Severance
A critical architectural pitfall is the Sidecar Problem. KEP-2008 checkpoints individual containers, not pods. If your pod contains a main app and an Envoy sidecar, checkpointing the app alone severs the loopback TCP connection to the sidecar. Upon restoration, the app will find its local proxy gone, resulting in immediate 503s. The only production-grade fix is to use node-level routing (eBPF) or ensure the sidecar is also checkpointed and restored in exact lock-step.
TCP Socket Death
While CRIU can save socket states, TCP connections typically do not survive cross-node migration. The remote peer will see a TCP Reset (RST) or a timeout. Your application must implement aggressive reconnection logic or use a L7 proxy (like an ALB or Istio) that can buffer requests during the 1.1s PLEG blackout.
The RAM Tax
Checkpointing a pod with 64GB of RAM creates a 64GB file. The time to write this to disk and upload it to S3 can be significant, creating a "freeze window" where the source pod is unresponsive. For huge memory footprints, consider incremental checkpointing (where only modified pages are dumped), though this is not yet natively supported by the Kubelet API.
Who Should Skip This
Stateless Apps: If your app can recover state from Redis/Postgres in under 2 seconds, standard rescheduling is simpler and more reliable.
Strict Compliance Environments: Dumping full RAM to disk can expose secrets in plaintext. If you cannot encrypt the checkpoint artifacts in transit and at rest, avoid this pattern.
High-Churn Workloads: If pods are replaced every few minutes, the operational toil of managing checkpoint artifacts exceeds the benefit of avoiding cold starts.
References & Further Reading
Official Kubernetes KEP-5823: Pod-level Checkpoint/Restore
CRIU (Checkpoint/Restore In Userspace) Technical Manual
AWS Containers Blog: Forensic Checkpointing on EKS
The PLEG (Pod Lifecycle Event Generator) Architecture in Kubelet