containerd 2.4.0: The Post-LTS Purge — Your Tracing Config Now Silently Does Nothing

containerd 2.4.0 shipped September 16, 2026 — the first minor after 2.3 LTS (April 2026), and the first release published under the project's new 4-month cadence synchronized with the Kubernetes train (April / August / December, per the RELEASES.md policy). It is a regular (non-LTS) release with an 8-month active window — EOL May 16, 2027 — versus 2.3 LTS's April 2028. The headline is a purge: everything deprecated during the LTS cycle is now removed. And the trap is subtle: the daemon does not crash on removed keys — it logs Ignoring unknown key in TOML for plugin and moves on, which means your tracing export silently stops while the node looks perfectly healthy.

What's Actually In It

658 commits, 77 contributors, shepherded by release owners Maksym Pavlenko (@mxpv) and Derek McGowan (@dmcgowan). The changes that matter to platform engineers fall into four buckets: config removals, a mount manager for image volumes, security hardening in the fetch path, and operational behavior changes.

Breaking Changes: The config.toml Purge

The removals land in PR #14166 ("Deprecations and removals for 2.4") by Samuel Karp. containerd 2.x's config loader is strict-mode TOML (DisallowUnknownFields), but when it hits unknown keys it logs a warning and ignores them — it does not exit. The practical consequence: a config with the old tracing block upgrades cleanly, the daemon starts, nodes go Ready… and no traces leave the box, because the OTLP tracing plugin now initializes exclusively from OpenTelemetry exporter environment variables. That is a silent observability regression, not an outage — the worst kind to discover during an incident.

# BEFORE (containerd ≤ 2.3) — warned, then ignored in 2.4.0:
# [plugins.'io.containerd.tracing.processor.v1'.otlp]
#   endpoint = "http://otel-collector:4318"
#   protocol = "http/protobuf"
#   insecure = true
# [plugins.'io.containerd.internal.v1'.tracing]
#   service_name = "containerd"
#   sampling_ratio = 0.1

# AFTER (containerd 2.4.0) — no tracing block in config.toml at all.
# Configure the OTLP exporter via environment variables on the unit:
#   /etc/systemd/system/containerd.service.d/otel.conf
[Service]
Environment="OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318"
Environment="OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf"
Environment="OTEL_SERVICE_NAME=containerd"
Environment="OTEL_TRACES_SAMPLER=traceidratio"
Environment="OTEL_TRACES_SAMPLER_ARG=1.0"

Two more removals from the same PR: enable_cdi is gone because CDI is now unconditional — the Container Device Interface path is the only path (relevant if you run GPU fleets that used it as an explicit opt-in); and CNI bin_dir became bin_dirs, accepting multiple lookup paths for nodes where CNI plugins are split across distro and vendor directories.

# /etc/containerd/config.toml (config version 3)
version = 3

[plugins.'io.containerd.cri.v1.runtime'.cni]
  bin_dirs = ['/opt/cni/bin', '/usr/libexec/cni']
  conf_dir = '/etc/cni/net.d'
  max_conf_num = 1

The Mount Manager: Fixing Image Volumes That Silently Mount Nothing

The architectural change of the release is PR #13542 by Hsiang-Kao Chen (@hsiangkao): CRI image mounts — the volume source that mounts another image's filesystem into a Pod without a PVC — now route through containerd's MountManager API instead of raw mount calls. This is the fix for issue #13534: on Kubernetes 1.36 with the erofs snapshotter, image-volume Pods logged ImageVolumeMountFailed ... fstype: format/mkdir/overlay ... err: no such device and then ran anyway with an empty directory where the image content should be. The reporter confirmed the fix works on an erofs node (build d62b6e798). Critically, this fix is not backported to the 2.3 LTS patches — if you run erofs nodes with image volumes, 2.4.0 is the fix train.

The mechanism: instead of passing raw mounts to the container setup path, the CRI server now calls MountManager().Activate(ctx, id, mounts, ...) with a GC back-reference label (containerd.io/gc.bref.snapshot.<snapshotter>), so the activated mount is tracked as GC-referenced state rather than anonymous kernel state. If an activation already exists — daemon restart, or a partially-failed previous attempt — the code reuses it via Info(ctx, id) instead of leaking duplicate mounts. Snapshotters that don't implement the MountManager interface fall through an IsNotImplemented path and keep legacy behavior, which is why this ships as safe-by-default rather than a hard cutover.

BEFORE (≤ 2.3, on erofs nodes)                  AFTER (2.4.0)
┌─────────────────────────────────────┐        ┌─────────────────────────────────────┐
│ kubelet: CreateContainer            │        │ kubelet: CreateContainer            │
│   │                                 │        │   │                                 │
│   ▼                                 │        │   ▼                                 │
│ CRI: mutateImageMount()             │        │ CRI: mutateImageMount()             │
│   │                                 │        │   │                                 │
│   │ raw mount(2) syscall            │        │   │ mm.Activate(id, mounts)           │
│   │  → overlay-on-erofs → ENODEV    │        │   │  → snapshotter-managed mount     │
│   │  → error swallowed into         │        │   │  → GC backref label registered    │
│   │    "empty dir" pod that runs    │        │   │  → restart-safe (Info() reuse)    │
│   ▼                                 │        │   ▼                                 │
│ ImageVolumeMountFailed event        │        │ image content visible in container  │
│ (pod still schedules and runs)      │        │ (or a real error event, not silence)│
└─────────────────────────────────────┘        └─────────────────────────────────────┘

apiVersion: v1
kind: Pod
metadata:
  name: image-volume-probe
spec:
  restartPolicy: Never
  containers:
  - name: test-container
    image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1
    command: ["/bin/sh", "-c", "ls /volume && sleep 3600"]
    volumeMounts:
    - name: volume
      mountPath: /volume
  volumes:
  - name: volume
    image:
      reference: registry.k8s.io/e2e-test-images/kitten:1.7
      pullPolicy: Always

On a 2.3.x node with the erofs snapshotter, that Pod starts, and /volume shows the busybox rootfs listing instead of the kitten image content — no files, no crash. On 2.4.0 the mount goes through the manager and either works or fails loudly. To be clear about scope: the default snapshotter remains overlayfs (configured under [plugins.'io.containerd.cri.v1.images']); erofs is an opt-in snapshotter used by some edge/embedded images, and this fix matters if you use it with image volumes.

Security: Cross-Origin Header Leaks and Thermal Masking

PR #12889 by @1seal closes a subtle credential leak in the image fetch path. OCI manifests can carry desc.urls — arbitrary URLs from which layer content may be fetched. Historically, containerd reused the resolver's headers (including registry credentials) when following those URLs. If desc.urls point at an origin other than your registry, forwarding Authorization, Proxy-Authorization, Cookie, and Cookie2 sends your registry token to whoever authored the manifest. The fix strips exactly those four headers on desc.urls-driven requests — mirroring the Go standard library's redirect behavior — while non-sensitive custom headers still flow. The PR ships with TestFetcherDescURLsDoesNotForwardResolverHeaders proving the strip.

PR #14090 by Samuel Karp masks /proc/interrupts and /sys/devices/system/cpu/cpu<x>/thermal_throttle inside Linux containers by default, matching moby, Kubernetes, CRI-O, Podman, and buildah — closing a host-fingerprinting side channel (see the related Kubernetes issue). If your monitoring agent reads interrupt counts from inside containers, expect it to go dark; collect host-level metrics with node-exporter-style hostPath mounts instead.

Performance: Gzip Layer Decompression Gets ~1.5× Faster

The sleeper hit is PR #13560 by Dr. Jan-Philip Gehrcke (@jgehrcke): containerd now uses klauspost/compress for gzip layer decompression instead of the Go standard library. The PR includes measured numbers on arm64 (NVIDIA Grace, Go 1.26.3, n=6, RSD < 1.5%): the CUDA 13.1.2 devel image's largest layer (1.8 → 3.4 GiB) drops from 13.6s to 9.1s (~1.5×), and pytorch/pytorch:latest's largest layer (3.4 → 7.0 GiB) drops from 41.7s to 28.8s (~1.4×). Allocation churn on the CUDA image collapses from 305 MiB / 1.91M mallocs to 0.18 MiB / 1,948 mallocs. For GPU fleet nodes pulling multi-GiB AI images, this is real node startup latency and memory headroom back — and it aligns containerd with CRI-O, Podman, and Skopeo, which already share klauspost's inflate core via containers/image. Notably, the Go standard library itself will adopt klauspost's gzip code for compression in Go 1.27 (golang/go#75532).

Operational Behavior Changes

• restart=always aligns with Docker semantics — containers explicitly stopped via crictl stop no longer spring back to life until the daemon restarts (PR #13993). If your runbooks rely on "stop then let it come back", rewrite them.
• OCI runtime feature introspection for non-runc runtimes (PR #13504) — the kubelet previously rejected hostUsers: false Pods with runtimeClassName: gvisor because containerd only queried the runc shim's features; now all shims are asked (containerd-shim-runsc-v1 already implemented -info).
• User namespaces + host networking unblocked — runtimeFeatures.UserNamespacesHostNetwork defaults to true (PR #13162), implementing KEP-5607 for the Kubernetes 1.37 pairing.
• NRI deprecation warnings (PR #13916 by Krisztian Litkey (@klihub)) — plugins using deprecated NRI interfaces now emit warnings; build against the v2 NRI API.
• erofs warm image cache (PR #13813 by Maksym Pavlenko) — the erofs snapshotter can consume a pre-warmed read-only shared image cache mounted from external storage, the pattern behind AWS's image prefetch guidance and GCP's prewarm docs, without re-baking node images whenever the image set changes.

Upgrading: The Order of Operations

The 4-month cadence pairs 2.4.0 with Kubernetes 1.37 (support matrix: K8s 1.37 → containerd 2.4.0+ or 2.3.x). Recommended: 2.4.0 on new/upgraded 1.37 nodes; 2.3 LTS for fleets that want two years of support. If you manage containerd at fleet scale, note the managed paths: GKE controls the runtime for you, and the extended-support branches for 1.7/2.0 exist specifically to back Google's fleet; EKS ships containerd through its AMIs — self-managed node images (Packer-built golden images, kubeadm clusters) are where this upgrade is yours to own.

# 1. Dump the running config and grep for the removed options.
containerd config dump | grep -E 'enable_cdi|bin_dir[^s]|service_name|sampling_ratio'

# 2. Check for the old tracing block in the on-disk config.
grep -n -A3 'tracing' /etc/containerd/config.toml

# 3. If found: strip the block, move settings to OTEL_* env vars on the unit,
#    then systemctl daemon-reload BEFORE restarting containerd.
#    2.4.0 logs 'Ignoring unknown key in TOML for plugin' and otherwise
#    runs fine — your traces silently stop exporting.

# 4. Verify the daemon after upgrade:
ctr version
crictl info | jq '.status.config'

Who Should Skip?

Fleets on 2.3 LTS with no image volumes, no erofs snapshotter, no third-party desc.urls pulls, and tracing already on env vars: skip — there's nothing urgent here. The must-take cases: (1) erofs nodes with image volumes (silent empty-mount, no 2.3.x backport), (2) fleets pulling images whose manifests carry desc.urls from third parties, (3) nodes moving to Kubernetes 1.37 that want the containerd pairing the test grid exercises, (4) GPU-heavy fleets that want the gzip win. Everyone else: let the LTS carry you to April 2028 and pick this up in 2.5 (December 2026).

Contributors

658 commits, 77 contributors. Release-critical work: Samuel Karp (deprecation removals, thermal masking), Hsiang-Kao Chen (mount manager), @1seal (header hardening), Jan-Philip Gehrcke (klauspost gzip), Maksym Pavlenko (erofs warm cache, release owner), and Derek McGowan as co-release-owner for shipping the first release of the new cadence on schedule.