Cilium Service Mesh: The Sidecar-Free eBPF Mesh on Kubernetes 1.37

Sources

Cilium is the default answer today when platform teams list their CNI requirements: eBPF datapath, kube-proxy replacement, identity-based network policy, and Hubble flow visibility without touching a single application container. But the pitch usually stops there. The interesting question for a platform engineer in late 2026 is no longer "which CNI?" — it is "do we still need a separate service mesh, or can the CNI absorb it?"

This guide takes the second question seriously. We walk the full sidecar-free mesh stack you can assemble with Cilium 1.20 on Kubernetes 1.37: the eBPF datapath for L3/L4, the per-node Envoy for L7, ztunnel for in-path mTLS, Gateway API v1.6 for north-south traffic, Hubble for observability, and ClusterMesh for multi-cluster. Every manifest here is complete and runnable. Nothing is a sketch.

We also spend a lot of time on the parts vendor blogs skip: what breaks when you delete kube-proxy, why L7 policy and ztunnel mTLS fight each other, the exact upgrade sequence for Gateway API CRDs, and which workloads should never enroll in the mesh. By the end you should be able to decide — with numbers and commands, not vibes — whether Cilium Service Mesh replaces the sidecar tax in your platform, or whether you genuinely still need Istio's waypoint model.

TL;DR

Why Sidecars Lost the Argument

The sidecar model charged every pod a fixed tax: CPU shares, memory floor (~50–100 MB per proxy), startup coupling, and an upgrade campaign every time the mesh control plane revved. The last one is the killer at fleet scale — a sidecar mesh upgrade means a rolling restart of every pod in the fleet, coordinated with HPA churn and deployment windows. An eBPF mesh upgrades the DaemonSet, and the fleet keeps serving.

Cilium's answer is architectural, not cosmetic. Instead of putting a proxy next to every workload, it puts the datapath functions under them:

The result is that the "mesh" is no longer a thing you install next to your platform. It is a set of capabilities your CNI already ships, and a set of CRDs you apply selectively. The rest of this guide is the operating manual for those capabilities.

Architecture: What Runs Where

Before manifests, get the component map right. A full Cilium Service Mesh installation on Kubernetes 1.37 consists of:

The Four Planes

  • Agents (DaemonSet cilium): one per node; compiles and attaches eBPF programs, enforces policy, programs service load balancing (BPF LB / Maglev), and redirects L7-matched flows to the node-local Envoy via TPROXY.
  • Operator (Deployment cilium-operator): IPAM, Node Init, ClusterMesh API server management, BGP control plane reconciliation, and (in 1.20) the ztunnel enrollment reconciler that registers SPIFFE identities with SPIRE in spire CA mode.
  • Envoy (DaemonSet cilium-envoy): the per-node L7 engine. Ingress, Gateway API, and L7 CiliumNetworkPolicy rules all compile down to xDS config streamed to this shared proxy. One process per node, not one per pod.
  • Hubble (in-agent) + Relay (Deployment) + UI: flow observation on every node, aggregated by a single relay, queryable by CLI or the Hubble UI deployed as its own chart.

And the data path a packet actually takes:

Packet walk, pod-to-pod with L7 policy (same node vs cross-node)

  Pod A (source)                       Pod B (destination)
 ┌──────────────┐                     ┌──────────────┐
 │  app container│                    │  app container│
 └──────┬───────┘                     └──────▲───────┘
        │ send()                             │ deliver()
 ┌──────▼───────────────────────────────────┴───────┐
 │  Pod A's netns (veth/netkit)                     │
 │   ├─ eBPF (from-container): policy check L3/L4   │
 │   └─ L7 rule matched? ── TPROXY ──► Envoy (node) │
 │                                      │ xDS route │
 │                                      ▼           │
 │  Node kernel: BPF LB / Maglev pick backend       │
 │   ├─ same node: deliver to Pod B netns directly  │
 │   └─ cross node: encapsulate (VXLAN/GENEVE) or   │
 │      route (native), [optional IPsec/WG, or     │
 │      ztunnel HBONE 15008 for mTLS]               │
 └──────────────────────────────────────────────────┘

Three things to internalize from this diagram, because they drive every trade-off later:

  1. The kernel path is the default. Only flows explicitly selected by an L7 rule take the Envoy hop. Everything else is wire-speed in-kernel forwarding.
  2. Envoy is per node, so L7 capacity is per node. A single hot pod with 20k RPS of HTTP policy does not get its own proxy; it shares the node's Envoy with every other L7-matched flow on that node. Capacity-plan Envoy DaemonSet resources accordingly.
  3. ztunnel changes the picture at the pod netns edge. When a namespace is enrolled, iptables rules inside each pod's network namespace redirect supported TCP flows to the node-local ztunnel, which tunnels them via HBONE (mTLS over HTTP/2 CONNECT, port 15008). This is why ztunnel mTLS and L4/L7 policy interact in non-obvious ways — policy that used to see the real 5-tuple now sees the tunnel.

Prerequisites & System Requirements

Cilium 1.20 requires Linux kernel >= 5.10 (or the equivalent backport, e.g. 4.18 on RHEL 8.10), AMD64 or AArch64 nodes, and iptables/netfilter kernel options present for masquerading, tunneling, and the L7/FQDN proxy features. Managed offerings lower the bar further: GKE with the Cilium datapath, EKS with the AmazonLinux 2/Bottlerocket AMIs, AKS with Azure CNI powered by Cilium — each ships a kernel that meets the requirements.

Verify before you install:

# Kernel and arch
uname -r            # expect >= 5.10, or 4.18 on RHEL 8.10
uname -m            # x86_64 or aarch64

# Required netfilter options (module names per docs.cilium.io System Requirements)
for opt in CONFIG_NETFILTER_XT_SET CONFIG_NETFILTER_XT_MATCH_COMMENT \
           CONFIG_NETFILTER_XT_TARGET_MARK CONFIG_NETFILTER_XT_TARGET_TPROXY; do
  grep -q "$opt=[my]" "/boot/config-$(uname -r)" && echo "$opt OK" || echo "$opt MISSING"
done

# Kubernetes version context (guide targets 1.37)
kubectl version --short 2>/dev/null || kubectl version --output=yaml | head -6

Kernel gotcha: the container image ships its own clang+LLVM >= 18.1 toolchain for runtime compilation where needed, so you do not need clang on the host — but the kernel config options above are non-negotiable. On minimal node images (Flatcar, custom Talos configs), verify TPROXY support specifically: with the default bpf.tproxy=false, L7 redirection uses iptables TPROXY, and nodes missing those modules will silently blackhole Gateway API traffic that "should work."

Reference Installation: Helm Values for a Mesh-Ready Cluster

The Cilium Helm chart is the source of truth for a production install. The values below enable the full mesh stack on a fresh cluster: kube-proxy replacement, L7 Envoy, Hubble with Relay, Gateway API controller, and ztunnel mTLS. Apply it with the 1.20.2 chart:

# cilium-values.yaml — Cilium 1.20.2, mesh-ready reference install
kubeProxyReplacement: true          # replace kube-proxy with BPF LB
k8sServiceHost: <API_SERVER_IP>    # required when kube-proxy is removed
k8sServicePort: 6443

# --- Datapath ---
# bpf.datapathMode=auto probes each host for netkit (kernel >= 6.8)
# and falls back to veth on the rest of a mixed fleet (new in 1.20)
bpf:
  datapathMode: auto               # default remains "veth"
  tproxy: false                    # default: iptables-based TPROXY for the L7 proxy

routingMode: tunnel                # or "native" with direct routing
operator:
  replicas: 2

# --- L7 / Envoy ---
l7Proxy: true                      # required for L7 policy + Ingress + Gateway API
envoy:
  enabled: true                    # dedicated cilium-envoy DaemonSet

# --- Gateway API ---
gatewayAPI:
  enabled: true                    # requires Gateway API CRDs >= v1.6.1 (see below)

# --- Hubble ---
hubble:
  enabled: true
  relay:
    enabled: true
  metrics:
    enabled:
      - dns
      - drop
      - tcp
      - flow
      - icmp
      - http

# --- ztunnel mTLS (Beta in 1.20) ---
encryption:
  enabled: true
  type: ztunnel                    # was "ipsec" or "wireguard" for transparent-only
  ztunnel:
    ca:
      type: internal               # 'spire' for the SPIFFE/SPIRE identity stack

Two values here are capacity decisions, not defaults. operator.replicas should be 2+ in any cluster you care about (the operator runs the ztunnel enrollment reconciler and IPAM; one replica is a SPOF). And because Envoy is shared per node, size the Envoy DaemonSet's resources against your real per-node L7 RPS rather than shipping the chart defaults — the node-local proxy is now the L7 budget for every pod scheduled on that node.

Install:

# Gateway API CRDs FIRST — Cilium 1.20 requires v1.6.1 minimum.
# Standard channel first:
kubectl apply --server-side -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.6.1/config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.6.1/config/crd/standard/gateway.networking.k8s.io_gateways.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.6.1/config/crd/standard/gateway.networking.k8s.io_httproutes.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.6.1/config/crd/standard/gateway.networking.k8s.io_referencegrants.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.6.1/config/crd/standard/gateway.networking.k8s.io_grpcroutes.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.6.1/config/crd/standard/gateway.networking.k8s.io_backendtlspolicies.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.6.1/config/crd/standard/gateway.networking.k8s.io_tlsroutes.yaml
# Experimental channel (ListenerSets, TCPRoute/UDPRoute in this cycle):
kubectl apply --server-side -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.6.1/config/crd/experimental/gateway.networking.k8s.io_listenersets.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.6.1/config/crd/experimental/gateway.networking.k8s.io_tcproutes.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.6.1/config/crd/experimental/gateway.networking.k8s.io_udproutes.yaml

# Then Cilium itself:
helm upgrade --install cilium cilium/cilium --version 1.20.2 \
  --namespace kube-system \
  --values cilium-values.yaml --wait

# Generate and apply the ztunnel CA/bootstrap secrets (see next section for full script)
bash generate-ztunnel-secrets.sh

# Confirm the stack:
cilium status --wait
cilium hubble enable   # no-op if Helm already enabled it; patches ConfigMap if needed

Do not skip the secrets step with ztunnel. The ztunnel integration requires a pre-created Kubernetes secret (cilium-ztunnel-secrets) containing bootstrap and CA keys before the agents will come up. This follows the same injection pattern as IPsec key rotation, and unlike the chart's certgen mode used for Hubble/ClusterMesh certs, it is on you to create them. The full generation script is below.

generate-ztunnel-secrets.sh — complete, from the official docs pattern

#!/usr/bin/env bash
# SPDX-License-Identifier: Apache-2.0
set -eu

workdir=$(mktemp -d)
cd "$workdir"

# == Bootstrap identity (secures ztunnel <-> Cilium xDS/cert server) ==
openssl genrsa -out bootstrap-private.key 2048

cat > openssl.conf <<'EOF'
[ req ]
distinguished_name = req_distinguished_name
x509_extensions = v3_ca
prompt = no

[ req_distinguished_name ]
O = cluster.local

[ v3_ca ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names

[alt_names]
DNS.1 = localhost
EOF

openssl req -x509 -new -nodes -key bootstrap-private.key -sha256 -days 3650 \
  -out bootstrap-root.crt -config openssl.conf

# == CA (signs ephemeral in-memory workload certs issued to ztunnel) ==
openssl genrsa -out ca-private.key 2048

cat > ca-openssl.conf <<'EOF'
[ req ]
distinguished_name = req_distinguished_name
x509_extensions = v3_ca
prompt = no

[ req_distinguished_name ]
O = cluster.local

[ v3_ca ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign
EOF

openssl req -x509 -new -nodes -key ca-private.key -sha256 -days 3650 \
  -out ca-root.crt -config ca-openssl.conf

kubectl --namespace kube-system create secret generic cilium-ztunnel-secrets \
  --from-file=bootstrap-private.key=bootstrap-private.key \
  --from-file=bootstrap-root.crt=bootstrap-root.crt \
  --from-file=ca-private.key=ca-private.key \
  --from-file=ca-root.crt=ca-root.crt

echo "ztunnel secrets created in kube-system/cilium-ztunnel-secrets"

The Control Plane Is Not One Thing: Istio Ambient vs Cilium Mesh

Both Istio ambient and Cilium Service Mesh run a per-node proxy named ztunnel. That is not a coincidence — Cilium 1.19+ builds on the upstream ztunnel project that originated in Istio's ambient work, with joint engineering between Isovalent and Microsoft. The difference is what the rest of the stack looks like around that proxy, and it matters for your migration path.

DimensionCilium Mesh (1.20)Istio Ambient
L4 datapatheBPF in-kernel; ztunnel for HBONE mTLS on enrolled namespaces (Beta)ztunnel per node (Rust); all in-mesh TCP via HBONE
L7 datapathPer-node Envoy, used only for L7-selected flowsOptional waypoint proxies (Envoy deployments), destination-enforced
Control planeCilium agents + operator; xDS to node Envoyistiod; separate xDS streams to ztunnel and waypoints
mTLS scopeNamespace label opt-in (io.cilium/mtls-enabled=true); TCP only; in-cluster onlyNamespace label (istio.io/dataplane-mode=ambient); TCP; plaintext fallback policy-controlled
L7 policy APICiliumNetworkPolicy L7 rules; upstream KNP/KCNP tiersAuthorizationPolicy (L4 at ztunnel, L7 at waypoint)
North-southGateway API (HTTP, gRPC, TLS, TCP, UDP routes), Ingress controllerIngress Gateways + Gateway API; waypoint-mediated ingress
Multi-clusterClusterMesh (Global Services, MCS-API stable in 1.20) — incompatible with ztunnel todayEast-west and multi-network gateways; no ClusterMesh equivalent
Day-2 upgradesDaemonSet rollout; L7 flows through proxies disrupted during upgrade (documented impact)ztunnel DaemonSet + waypoint deployments; istiod canary
ObservabilityHubble flows, per-node collection, relay aggregation, metricsistiod metrics, ztunnel access logs, SkyWalking/OTel integration

How to read this table: Istio ambient still owns the richest L7 feature set (VirtualService-style routing, retries, fault injection — all waypoint features). Cilium wins on datapath efficiency and on being the CNI you already operate. If your platform's L7 needs are "Gateway API + a handful of L7 policy rules," Cilium alone is enough. If your platform sells per-team VirtualService routing as a product, you will run Istio ambient next to Cilium — and the docs-supported configuration for that coexistence is kubeProxyReplacement: false, or true with socketLB.hostNamespaceOnly: true and cni.exclusive: false.

mTLS Without Sidecars: ztunnel Deep-Dive

Transparent encryption (IPsec/WireGuard) has always been Cilium's answer to "encrypt east-west without touching pods." But it encrypts between nodes: same-node pod-to-pod traffic never left the kernel unencrypted view, and the mutual-authentication layer (Beta since 1.14) did an out-of-band mTLS handshake that could drop the first packet while identities were proven. ztunnel replaces that dance with an in-path design.

When you label a namespace io.cilium/mtls-enabled=true, the Cilium agent:

  1. Enrolls every existing pod in that namespace (except ztunnel's own) — and every future pod, automatically.
  2. Programs iptables rules inside each pod's network namespace, redirecting supported TCP flows to the node-local ztunnel.
  3. Sends pod metadata to ztunnel via the ZDS protocol; ztunnel holds each new connection until it establishes a TLS 1.3 mTLS session over HBONE to the destination node's ztunnel, then proxies the bytes.

Because the tunnel is established inline, there is no first-packet drop, and traffic between pods on the same node is encrypted too — both of the old design's weaknesses. The trade is a per-connection handshake cost and the hard limitations below.

ztunnel Hard Limitations (1.20, Beta)

  • TCP only. UDP and other protocols are not redirected to ztunnel. Your DNS (UDP), QUIC, and video streams stay plaintext inside the mesh.
  • Both endpoints must be enrolled. Enrolled-to-non-enrolled communication is not supported. A partial rollout across your service graph will break — plan enrollment as a graph-wide unit per call chain, or don't start.
  • Namespace-granularity only. No pod-level enrollment.
  • ClusterMesh incompatible. Enabling ztunnel and ClusterMesh together is a validation error in 1.20. Choose per-cluster: cross-cluster encryption via IPsec/WireGuard, or in-cluster mTLS via ztunnel.
  • Policy interaction. Traffic is encrypted inside the pod netns before it hits Cilium's L4 policy engine on the node, so L4 policies see the tunnel; only rules targeting ztunnel's HBONE port (15008) behave as before. L7 policies on ztunnel-enrolled workloads need explicit design (see the policy section).
  • Host-networked pods are skipped. Anything without a network namespace path (node-local daemons in hostNetwork mode) cannot enroll.
  • iptables required. No iptables-free environments (some minimal runtimes).

Enroll and verify:

# Enroll a namespace
kubectl label namespace payments io.cilium/mtls-enabled=true

# Verify enrollment (three independent checks)
kubectl get namespaces -l io.cilium/mtls-enabled=true
kubectl -n kube-system exec ds/cilium -- cilium-dbg statedb dump | jq '."mtls-enrolled-namespaces"'

# Prove encryption on the wire: HBONE port 15008 traffic on the node interface
kubectl -n kube-system exec -ti ds/cilium -- bash
apt-get update && apt-get -y install tcpdump
tcpdump -i eth0 port 15008    # cilium_vxlan instead of eth0 when tunneling

CA backends: encryption.ztunnel.ca.type=internal (default) runs ztunnel mTLS with keys Cilium itself manages — no SPIRE required. ca.type=spire switches to SPIFFE/SPIRE: the operator runs a namespace enrollment reconciler that registers SPIFFE identities for every ServiceAccount in labeled namespaces, and SPIRE issues and rotates the workload certs. Metrics exist for both paths: cilium_operator_ztunnel_enrollment_ops_total from the reconciler, plus ztunnel's own enrollment and connection-health metrics — the 1.20 additions that make the Beta actually operable.

Gateway API: North-South Without the Annotation Hell

Ingress with per-controller annotations was the industry's shared bad habit. Gateway API replaces it with role-oriented resources: the platform team owns GatewayClass and Gateway, application teams own HTTPRoutes in their namespaces, and everything is portable across conformant implementations. Cilium's implementation (v1.6.1, Standard channel, core conformance) runs on the same node-local Envoy — no separate ingress controller deployment to operate.

What 1.20 adds is the breadth that finally makes Gateway API a credible total replacement for Ingress NGINX:

Gateway API v1.6 capabilities in Cilium 1.20

  • ExternalAuth (GEP-1494): delegate authentication per-HTTPRoute to an external authorizer (Authelia, oauth2-proxy, Keycloak-backed) before traffic ever reaches the app. 200 forwards, 302 redirects (SSO login), 401/403 blocks at the gateway; identity headers injected downstream.
  • TCPRoute / UDPRoute: databases, brokers, DNS, game servers — L4 north-south through the same Gateway model (GEP-2644/2645).
  • ListenerSets: application teams attach their own listeners to a platform-owned shared Gateway via namespace selector — the delegation model Ingress never had.
  • CORS filter, redirect codes 303/307/308, HTTP Server header control (CiliumGatewayClassConfig modes OVERWRITE / APPEND_IF_ABSENT / PASS_THROUGH).

A complete, runnable north-south stack — shared Gateway, delegated ListenerSet, ExternalAuth-protected dashboard, and an L4 TCPRoute for the database:

# --- 1. Platform-owned Gateway with ListenerSet delegation ----------------
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: shared-gateway
  namespace: gateway-system
spec:
  gatewayClassName: cilium
  allowedListeners:                      # 1.20 ListenerSets: delegate listener creation
    namespaces:
      from: Selector
      selector:
        matchLabels:
          gateway-access: "true"
  listeners:
  - name: base-http
    protocol: HTTP
    port: 8081
---
# --- 2. App namespace opts in via label ----------------------------------
apiVersion: v1
kind: Namespace
metadata:
  name: hr
  labels:
    gateway-access: "true"
---
apiVersion: gateway.networking.k8s.io/v1
kind: ListenerSet
metadata:
  name: hr-listeners
  namespace: hr
spec:
  parentRef:
    name: shared-gateway
    namespace: gateway-system
  listeners:
  - name: dashboard
    hostname: hr.example.com
    protocol: HTTP
    port: 80
---
# --- 3. ExternalAuth-protected HTTPRoute (GEP-1494) ------------------------
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: hr-dashboard
  namespace: hr
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: ListenerSet
    name: hr-listeners
    sectionName: dashboard
  hostnames: ["hr.example.com"]
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    filters:
    - type: ExternalAuth
      externalAuth:
        protocol: HTTP
        backendRef:
          name: authelia
          port: 80
        http:
          path: /api/authz/ext-authz
          allowedHeaders: [cookie]
          allowedResponseHeaders: [Remote-User, Remote-Email, Remote-Name, Remote-Groups]
    backendRefs:
    - name: hr-dash
      port: 8080
---
# --- 4. L4 north-south: TCPRoute for the payments database -----------------
apiVersion: gateway.networking.k8s.io/v1
kind: TCPRoute
metadata:
  name: payments-db
  namespace: payments
spec:
  parentRefs:
  - name: shared-gateway
    namespace: gateway-system
    sectionName: tcp-db          # a TCP listener defined on the Gateway
  rules:
  - backendRefs:
    - name: payments-mongo
      port: 27017

The win: one Gateway object, one Envoy DaemonSet, zero annotation archaeology. App teams get self-service listeners and routes without write access to the Gateway, and the same manifests run against any conformant implementation.

The catches:

L7 Policy: CiliumNetworkPolicy and the KCNP Tiering

East-west L7 policy in Cilium is a CiliumNetworkPolicy with an http rules block — it compiles to Envoy config on the node-local proxies and is enforced transparently. The classic pattern: allow the API gateway's calls to the payments service only on /charge, and DNS resolution everywhere.

Pattern 1 — L7 allowlist (from the official Layer 7 policy language)

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: payments-ingress-l7
spec:
  description: "Allow the API gateway to call /v1/charge on payments, JSON only"
  endpointSelector:
    matchLabels:
      app: payments
  ingress:
  - fromEndpoints:
    - matchLabels:
        io.kubernetes.pod.namespace: gateway
        app: api-gateway
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: "GET"
          path: "^/v1/charge$"
        - method: "POST"
          path: "^/v1/charge$"
          headers:
          - 'Content-Type: application/json'
  # DNS resolution for the clients, via the DNS proxy:
  - fromEndpoints:
    - matchLabels:
        io.kubernetes.pod.namespace: kube-system
        k8s-app: kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: UDP
      rules:
        dns:
        - matchPattern: "*"

Read the semantics carefully, because they differ from every L4 intuition you have: path is an extended POSIX regex (anchor with ^...$), method likewise, and headers is a list of exact "Name: value" strings. A violation is not a packet drop — the proxy crafts an application-level denial (HTTP 403 for HTTP, REFUSED for DNS). And per the docs' own warning: if an L4 rule on the same port/protocol exists in the policy, its L7 sibling has no effect; the union of L7 rules per port is what matters. Test in audit mode first — 1.20's policy correlation now names the exact policy that would have denied a flow:

Two 1.20 changes you should build into your policy architecture on day one:

  1. Upstream Kubernetes ClusterNetworkPolicy (KCNP). The network-policy-api subgroup consolidated AdminNetworkPolicy and BaselineAdminNetworkPolicy into a single cluster-scoped ClusterNetworkPolicy with tiers: an Admin tier that outranks namespaced policy, and a Baseline tier that namespaced policy can override. Cilium 1.20 implements it. This is the guardrails primitive platform teams have wanted: hard-deny rules no namespace owner can re-open, and a cluster-wide floor that individual teams can only tighten.
  2. Identity aggregation. Selecting world or remote-node used to fan out to one BPF policy-map entry per matching identity — thousands of entries on meshed clusters, with a real risk of filling the map. 1.20 inserts a single wildcard entry per semantic group instead. The new cluster-mesh entity ("allow from every meshed cluster") rides the same mechanism: fromEntities: [cluster-mesh] replaces fragile per-cluster enumerations, and the mesh-wide rule is now the cheaper one to enforce at scale.
# Cluster-wide guardrail: nothing may talk to the PCI namespace,
# and no namespace policy can override this. (upstream KCNP, alpha API)
apiVersion: policy.networking.k8s.io/v1alpha2
kind: ClusterNetworkPolicy
metadata:
  name: deny-to-pci
spec:
  tier: Admin
  priority: 10
  subject:
    namespaces:
      matchLabels:
        kubernetes.io/metadata.name: pci
  ingress:
  - name: deny-all
    action: Deny
    from:
    - namespaces: {}      # ...except explicitly allowed by a later, lower-priority policy

Removed capability, check your policies: Envoy Go extensions (proxylib) and with them Kafka-aware L7 policies were removed in 1.18 (deprecated earlier). If your repo still contains .spec.ingress[].toPorts[].rules.kafka, .rules.l7 or .rules.l7proto blocks, those policies must be cleaned before upgrading to 1.20 — the upgrade guide is explicit that they block the upgrade path.

Observability: Hubble as the Mesh Telemetry Plane

A mesh you cannot see is a liability. Hubble gives you per-flow visibility (L3/L4 verdicts, DNS, HTTP metadata when L7 policy is active) from eBPF events on every node, aggregated through Relay, with Prometheus metrics for the aggregate view. Enable it in the Helm values (as above), then use it:

# Flows: who talked to whom, and what policy decided
hubble observe --namespace payments --protocol http --since 5m

# Drops with the responsible policy named — 1.20 fills in policy correlation
# for AUDIT verdicts too, so you can preview enforcement before flipping it on:
hubble observe --type policy-verdict --verdict AUDIT --print-policy-names

# Sample output (1.20 policy correlation):
# INGRESS AUDITED BY deny-alliance (CiliumNetworkPolicy) (TCP Flags: SYN)

# Metrics on the relay:
kubectl -n kube-system port-forward svc/hubble-relay 4245:80
hubble status   # connected flows/s

The 1.20 policy correlation for audit verdicts is the feature that changes how you roll out policy: audit mode used to show "this flow would be dropped" with no policy attribution, which made it hard to trust as a pre-enforcement preview. Now the verdict names the exact policy that would have denied or allowed the flow — flip enforcement on only when the audit stream shows the policies you expect.

Multi-Cluster: ClusterMesh and MCS-API

ClusterMesh links up to 255 clusters by default: global services, mesh-wide policy entities, and service affinity, with the classic CiliumClusterwideNetworkPolicy and Global Service annotations. The 1.20 change is the graduation of the upstream Multi-Cluster Services API support to stable: a portable, vendor-neutral way to export services across clusters.

# The entire app-team surface of MCS-API: one object, both clusters.
apiVersion: multicluster.x-k8s.io/v1beta1
kind: ServiceExport
metadata:
  name: web
  namespace: default
---
# Consumers reach it via the standard MCS DNS name:
#   web.default.svc.clusterset.local
# Cilium's operator reconciles ServiceImport, the clusterset VIP,
# and the derived Service backing cross-cluster backends.

1.20 moved the implementation to the multicluster.x-k8s.io/v1beta1 CRDs and installs them automatically; v1alpha1 remains supported during the window. But remember the ztunnel collision: MCS-API/ClusterMesh and ztunnel cannot be enabled together in 1.20 — so your multi-cluster encryption story is IPsec/WireGuard transparent encryption (node-to-node, no in-path mTLS), not ztunnel. If in-cluster mTLS is the priority, single-cluster ztunnel; if cross-cluster services are the priority, ClusterMesh + transparent encryption. This is a genuine architectural fork, and it is documented as a validation error, not a graceful either/or.

kube-proxy Replacement: The Flag That Deserves Respect

kubeProxyReplacement: true is what lets eBPF own the service load-balancing datapath — Maglev consistent hashing, socket-LB for in-cluster service traffic, and direct server return for NodePort. It is the configuration most production Cilium clusters run, and the one with the most edge cases.

You should run it

When you want the full datapath performance profile, DSR for external traffic, and a consistent eBPF-owned LB — and your nodes are a supported, homogeneous-enough fleet.

Respect these edges

Socket-LB vs NodePort semantics: in-cluster connections to NodePort services are now balanced at the client pod (egress) rather than at the target node — meaning client NetworkPolicy must allow egress to the service backends, and backend policy must allow ingress from the client pod. Old rules written for kube-proxy semantics will silently drop.

1.20 behavior unification: the same immediate-load-balance behavior now applies with SocketLB disabled or socketLB.hostNamespaceOnly=true.

firewalld conflicts: firewalld on the node fights the BPF programs for control of the host network. Pick one owner of host networking.

Upgrades reset L7 connections: documented, expected: any traffic flowing via user-space proxies (L7 policy, Ingress/Gateway API) is disrupted when the Cilium pod (and its Envoy) restarts. Clients must reconnect. Plan mesh upgrades as traffic-affecting windows even though L3/L4 keeps flowing.

# Post-install sanity for kube-proxy replacement
cilium status --wait
kubectl -n kube-system exec ds/cilium -- cilium-dbg status | grep -E 'KVStore|kubeProxy|Device'

# Maglev: consistent hashing — check the configured table size and per-node state
kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf lb list | head

# 1.20 datapath mode probe result (netkit auto):
kubectl -n kube-system exec ds/cilium -- cilium-dbg status | grep Device
#   Device Mode: netkit [Configured: auto]
kubectl -n kube-system exec ds/cilium -- cilium-dbg metrics list | grep datapath_config
#   cilium_feature_datapath_config configured_mode=auto operational_mode=netkit 1.000000

Mermaid: The Full Data Path with mTLS

The complete request journey — Gateway API at the edge, eBPF in the kernel, ztunnel HBONE between nodes, Envoy for L7, Hubble watching all of it:

sequenceDiagram
    participant U as Client
    participant GW as Gateway (Envoy, node-local)
    participant AZ as ExternalAuth Service
    participant A as Pod A (payments ns, ztunnel-enrolled)
    participant ZA as ztunnel (node 1)
    participant ZB as ztunnel (node 2)
    participant B as Pod B (payments ns, ztunnel-enrolled)
    participant H as Hubble

    U->>GW: HTTP GET hr.example.com
    GW->>AZ: authz check (cookie)
    AZ-->>GW: 200 OK + Remote-User headers
    GW->>A: forward (L7 policy on node Envoy)
    A->>ZA: TCP (iptables redirect in pod netns)
    ZA->>H: flow event
    ZA->>ZB: HBONE CONNECT, mTLS TLS 1.3 (port 15008)
    ZB->>H: flow event
    ZB->>B: deliver (decrypted in pod netns)
    B-->>A: response (reverse path, same tunnel)

Cost Model: Sidecar Tax vs Node-Local Mesh

Numbers beat adjectives. Model both options for a 50-node cluster, 1,500 pods, 30 pods/node — proxy sizing taken from the ambient-era measurements the community has converged on (Envoy sidecar/waypoint at 50–100 MB; node-local Envoy and ztunnel comparable per process, but shared):

Line itemSidecar mesh (1,500 pods)Cilium mesh (node-local)
Proxy processes1,500 (one per pod)~100 (Envoy + ztunnel per node × 50)
Memory floor (50 MB/proxy)75 GB across the fleet~5 GB (2 processes × 50 MB × 50 nodes)
CPU baselinePer-pod share, always reservedConcentrated on L7-active nodes; idle elsewhere
Upgrade blast radiusEvery pod in the fleet restartsDaemonSet rollout; L7 flows reconnect
Startup couplingProxy init containers delay pod readinessNone — pods never wait for a proxy
Per-pod added latencyTwo extra hops (sidecar in/out)Zero for L3/L4; one hop only for L7-selected flows

The honest caveat: the node-local model moves the capacity conversation from per pod to per node. A node running 30 L7-heavy pods needs an Envoy sized for the sum of their RPS. Right-size the DaemonSet resources (see the Helm values above) and autoscale nodes on Envoy CPU, and the model still wins by an order of magnitude — but "it's the CNI's problem now" is not a capacity plan.

Upgrade Runbook: 1.19 → 1.20

Cilium supports upgrades between consecutive minor versions only, and expects the latest patch of your current minor first. The runbook that avoids the known traps:

# 0. Preflight — do these BEFORE touching Helm
#    a) Kafka/L7 Go-extension policy blocks? (removed in 1.18, will hard-fail)
grep -rn "rules:" policies/ | grep -E "kafka|l7proto" && echo "CLEAN FIRST"
#    b) CiliumNodeConfig on v2alpha1? (removed in 1.20)
kubectl get ciliumnodeconfigs.cilium.io -A -o jsonpath='{range .items[*]}{.apiVersion}{"\n"}{end}' | sort -u
#    c) TLSRoute users: back up now
kubectl get tlsroutes.gateway.networking.k8s.io -A -o yaml > tlsroutes-backup.yaml

# 1. Latest patch of current minor
helm upgrade cilium cilium/cilium --version 1.19.7 -n kube-system --reuse-values --wait

# 2. Gateway API CRDs to v1.6.1 — Experimental channel for TLSRoute v1alpha2
#    (see full CRD commands in the installation section above)

# 3. Cilium 1.20.2
helm upgrade cilium cilium/cilium --version 1.20.2 \
  -n kube-system --reuse-values --wait

# 4. Post-upgrade verification
cilium status --wait
cilium connectivity test --force
kubectl -n kube-system rollout status ds/cilium ds/cilium-envoy
hubble observe --since 10m --verdict DROPPED | head   # watch for surprise drops

# 5. Watch the config-drift metric 1.20 added
#    cilium_configDrift* — keys the agent hasn't applied yet = restart needed
kubectl -n kube-system exec ds/cilium -- cilium-dbg metrics list | grep -i drift

Deprecated in 1.20 — schedule the migration now: Mutual Authentication (the pre-ztunnel feature, Beta since 1.14) is deprecated with removal planned. hubble.preferIpv6 moves to top-level preferIpv6. dnsProxy.preCache goes away in 1.21. cilium-dbg bgp commands and the local REST BGP API are deprecated in favor of the hive shell (cilium-dbg shell -- bgp/peers --format detailed). The ClusterMesh cert validity default is now one year — if you use the Helm generation mode, certs renew only when the chart re-renders, so upgrade at least once a year or switch to cronJob/certmanager modes.

Edge Cases That Will Bite Someone on Your Team

Who Should Skip This Entirely

The Verdict

Cilium Service Mesh in 1.20 is the first release where the honest answer to "do we need a separate mesh?" is "usually not." mTLS that covers same-node traffic, Gateway API with ExternalAuth and L4 routes, tiered cluster-wide policy via upstream KCNP, policy-correlated audit trails, and Hubble — all on the DaemonSet upgrade model, all on one Envoy per node.

What keeps it from being a blanket recommendation is the same thing that has always separated CNI-as-mesh from a real mesh: the L7 control plane. Cilium's L7 story is Gateway API plus CiliumNetworkPolicy — powerful, standards-aligned, and deliberately narrower than Istio's per-workload routing surface. If your platform's product is rich per-service traffic management, you will still run Istio ambient next to Cilium, in the documented coexistence mode. If your platform's product is paved roads with sane defaults — the platform-monkey thesis — then the CNI you already pay for just absorbed your mesh budget, and ztunnel's Beta label is the only reason not to start the rollout planning today.

Plan it as: transparent encryption now (stable), Gateway API north-south now (GA), L7 policy where the audit stream proves it (1.20 correlation makes this safe), ztunnel in one non-ClusterMesh cluster this quarter (Beta, metrics are finally there), and a re-evaluation of the Istio question when ztunnel goes stable and the ClusterMesh incompatibility is resolved.

Mermaid: Component Architecture

The control-plane and data-plane view of a full 1.20 mesh deployment — who talks to whom, and which components share a node:

flowchart TB
    subgraph K8s["Kubernetes API Server (v1.37)"]
        CRDs["CRDs: CiliumNetworkPolicy / CiliumClusterwideNetworkPolicy
ClusterNetworkPolicy (KCNP, v1alpha2) / CiliumNodeConfig (v2)
Gateway API v1.6.1 / CiliumEnvoyConfig"] end subgraph OPS["Platform Operators (kube-system)"] OP["cilium-operator (2 replicas)
IPAM · ztunnel enrollment reconciler · MCS-API · BGP"] HR["hubble-relay"] CM["clustermesh-apiserver
(not with ztunnel!)"] end subgraph NODE1["Node 1"] AG1["cilium agent
(eBPF programs: policy, BPF LB, TPROXY)"] EN1["cilium-envoy
(L7 + Gateway API)"] ZT1["ztunnel
(HBONE mTLS, 15008)"] P1["pod A (mtls-enabled ns)"] P2["pod B"] end subgraph NODE2["Node 2"] AG2["cilium agent"] EN2["cilium-envoy"] ZT2["ztunnel"] P3["pod C (mtls-enabled ns)"] end K8s -->|watch & reconcile| AG1 K8s -->|watch & reconcile| AG2 K8s --> OP OP -->|SPIFFE registration (spire CA mode)| K8s AG1 -->|xDS| EN1 AG2 -->|xDS| EN2 AG1 -->|ZDS: pod metadata| ZT1 AG2 -->|ZDS| ZT2 P1 -->|iptables redirect (netns)| ZT1 P2 -->|in-kernel L3/L4 (no L7 rule)| AG1 ZT1 -->|HBONE mTLS :15008| ZT2 ZT2 --> P3 AG1 -.->|flow events| HR AG2 -.->|flow events| HR CM -.->|global services / MCS-API| AG1 CM -.-> AG2

The Demo App: Complete Runnable Manifests

Everything below assumes this three-tier demo — the same payments/gateway namespaces used in the policy sections. Apply it and every command in this guide runs as written:

# demo-app.yaml — namespaces, deployments, services
apiVersion: v1
kind: Namespace
metadata:
  name: gateway
---
apiVersion: v1
kind: Namespace
metadata:
  name: payments
  labels:
    io.cilium/mtls-enabled: "true"     # ztunnel enrollment (requires ztunnel install)
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
  namespace: gateway
spec:
  replicas: 2
  selector:
    matchLabels: {app: api-gateway}
  template:
    metadata:
      labels: {app: api-gateway}
    spec:
      containers:
      - name: gateway
        image: ghcr.io/cilium/json-mock:http-v1.3.12
        ports: [{containerPort: 8080}]
        resources:
          requests: {cpu: 100m, memory: 64Mi}
          limits: {cpu: 500m, memory: 128Mi}
---
apiVersion: v1
kind: Service
metadata:
  name: api-gateway
  namespace: gateway
spec:
  selector: {app: api-gateway}
  ports: [{port: 8080, targetPort: 8080}]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments
  namespace: payments
spec:
  replicas: 3
  selector:
    matchLabels: {app: payments}
  template:
    metadata:
      labels: {app: payments}
    spec:
      containers:
      - name: payments
        image: ghcr.io/cilium/json-mock:http-v1.3.12
        ports: [{containerPort: 8080}]
        resources:
          requests: {cpu: 100m, memory: 64Mi}
          limits: {cpu: 500m, memory: 128Mi}
---
apiVersion: v1
kind: Service
metadata:
  name: payments
  namespace: payments
spec:
  selector: {app: payments}
  ports: [{port: 8080, targetPort: 8080}]
# Smoke test: connectivity, then prove mTLS and L7 policy on the wire
kubectl apply -f demo-app.yaml
kubectl -n gateway run test --image=cilium/cilium:1.20.2 --command -- sleep 3600
kubectl -n gateway exec test -- curl -s http://payments.payments:8080/ | head -3

# mTLS proof: capture on the source node, HBONE port 15008
kubectl -n kube-system exec -ti ds/cilium -- bash -c \
  "apt-get -y install tcpdump >/dev/null 2>&1; tcpdump -i eth0 port 15008 -c 5"

# L7 policy proof: apply the L7 policy from the previous section, then
hubble observe --namespace payments --protocol http --since 2m -o compact

Traffic Engineering: Maglev, Weights, and Traffic Distribution

Two 1.20 features turn Cilium's BPF load balancer from "evenly split" into a real traffic-engineering surface. Both matter to platform teams running canaries and maintenance windows.

Maglev weights via EndpointSlice annotations. For selectorless Services backed by manually managed EndpointSlices, annotate each slice with service.cilium.io/weight (0–65535, relative). A 70/30 canary becomes a two-line annotation change, and weight 0 is a connection-preserving drain: existing connections keep flowing, but no new ones are steered there — the clean maintenance-window primitive that mesh-free platforms have lacked:

apiVersion: v1
kind: Service
metadata:
  name: payments-stable
  namespace: payments
  annotations:
    service.cilium.io/lb-algorithm: maglev
spec:
  type: ClusterIP
  ports: [{name: http, port: 8080, protocol: TCP, targetPort: 8080}]
  # no selector: backends come from the EndpointSlices below
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: payments-stable-primary
  namespace: payments
  labels:
    kubernetes.io/service-name: payments-stable
  annotations:
    service.cilium.io/weight: "70"        # 70% of new connections
addressType: IPv4
ports: [{name: http, protocol: TCP, port: 8080}]
endpoints:
  - addresses: ["10.0.0.11"]
  - addresses: ["10.0.0.12"]
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: payments-stable-canary
  namespace: payments
  labels:
    kubernetes.io/service-name: payments-stable
  annotations:
    service.cilium.io/weight: "30"        # canary share; "0" = drain
addressType: IPv4
ports: [{name: http, protocol: TCP, port: 8080}]
endpoints:
  - addresses: ["10.0.1.21"]
  - addresses: ["10.0.1.22"]

Traffic distribution with the standard Kubernetes field. Cilium 1.20 honors the upstream trafficDistribution values PreferSameZone and PreferSameNode (beyond the older PreferClose) — keeping traffic local for latency and for cross-zone transfer costs, with graceful fallback when no local backend is healthy. Because it is the standard field, the same Service spec behaves consistently on any conformant implementation:

apiVersion: v1
kind: Service
metadata:
  name: payments
  namespace: payments
spec:
  trafficDistribution: PreferSameZone
  selector: {app: payments}
  ports: [{protocol: TCP, port: 8080, targetPort: 8080}]
  type: ClusterIP

The Monitoring Stack: Prometheus + Grafana for Mesh Health

Hubble metrics feed the standard Prometheus scrape of the agents and relay. The alerting rules below cover the four failure modes that actually page platform teams: agent down, endpoint regeneration storms, policy drops, and ztunnel enrollment failures:

cilium-mesh-alerts.yaml — PrometheusRule for the mesh control plane (note the raw wrapper: the templates below are Prometheus syntax that Nunjucks must not parse)

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: cilium-mesh-alerts
  namespace: kube-system
  labels:
    release: prometheus            # match your Prometheus operator's ruleSelector
spec:
  groups:
  - name: cilium-datapath
    rules:
    - alert: CiliumAgentDown
      expr: cilium_agents_up == 0 or absent(cilium_agents_up)
      for: 5m
      labels: {severity: critical}
      annotations:
        summary: "Cilium agent is down on {{ $labels.k8s_node }}"
    - alert: CiliumEndpointRegenerationStalled
      expr: rate(cilium_endpoint_regenerations_total{outcome="failure"}[5m]) > 0.1
      for: 15m
      labels: {severity: warning}
      annotations:
        summary: "Endpoint regeneration failures on {{ $labels.k8s_node }}"
    - alert: CiliumPolicyDropsHigh
      expr: rate(hubble_drop_total[5m]) > 10
      for: 10m
      labels: {severity: warning}
      annotations:
        summary: "Sustained policy drops — possible rollout regression"
  - name: ztunnel
    rules:
    - alert: ZtunnelEnrollmentFailures
      expr: rate(cilium_operator_ztunnel_enrollment_ops_total{status="error"}[10m]) > 0
      for: 15m
      labels: {severity: warning}
      annotations:
        summary: "ztunnel namespace enrollment is failing — pods not getting mTLS"
    - alert: ZtunnelConnectionIssues
      expr: rate(ztunnel_connection_errors_total[10m]) > 0.5
      for: 10m
      labels: {severity: warning}
      annotations:
        summary: "ztunnel is failing a material share of HBONE connections"

Dashboard queries that matter for a mesh rollout (paste into any Grafana panel against the Hubble metrics):

# 1. mTLS coverage: share of flows on the HBONE port vs total east-west
sum(rate(hubble_flows_processed_total{protocol="TCP"}[5m]))

# 2. Policy verdicts over time (enforcement regressions show here first)
sum by (verdict) (rate(hubble_verdicts_total[5m]))

# 3. Config drift: unapplied ConfigMap keys = agent restart needed (new in 1.20)
cilium_configdrift_keys_total{status="pending"}

# 4. Envoy L7 load per node — the shared-proxy contention signal
sum by (k8s_node) (rate(envoy_http_downstream_rq_total[5m]))

A note on the metric names above: scrape them once and confirm the exact series names your chart version exposes before committing the rules to your alerting repo — Cilium's metric names have shifted across minors (the 1.20 upgrade guide documents several renames, e.g. per-node IPAM metrics replacing cilium_operator_ipam_ips). Treat alert names as code: version them, test them in audit, and grep the exposition endpoint (kubectl -n kube-system exec ds/cilium -- cilium-dbg metrics list) when a panel goes flat.

Troubleshooting: The Five Failures You Will Actually Hit

1. "Gateway API created the LB, but connections time out"

Symptom: the Gateway's Service gets an external IP; curls hang. Cause: L7 traffic never reaches Envoy — the node lacks the iptables/netfilter modules for TPROXY (bpf.tproxy=false default). Fix: verify CONFIG_NETFILTER_XT_TARGET_TPROXY on the node, or switch to the beta eBPF TPROXY (bpf.tproxy=true — remembering it is incompatible with netkit datapath mode).

2. "I labeled the namespace but pods aren't encrypted"

Symptom: tcpdump -i eth0 port 15008 shows nothing. Causes, in order of likelihood: (a) ztunnel not actually enabled — check kubectl -n kube-system describe cm cilium-config | grep enable-ztunnel -A2; (b) the destination namespace is not enrolled (both endpoints must be); (c) host-networked pods — they are skipped by design; (d) UDP flow — ztunnel is TCP-only. Fix: enroll both ends, re-check the StateDB table (cilium-dbg statedb dump | jq '."mtls-enrolled-namespaces"').

3. "L4 policy stopped matching after ztunnel enrollment"

Symptom: previously-working CiliumNetworkPolicies drop or allow unexpectedly on enrolled namespaces. Cause: traffic is encrypted inside the pod netns before Cilium's L4 engine sees it — policy now observes the tunnel, and only rules targeting the HBONE port (15008) behave classically. Fix: redesign the policy set for enrolled namespaces around the documented interaction, validate with hubble observe --verdict DROPPED, and use audit mode with policy correlation before enforcing.

4. "Upgrade to 1.20 broke TLSRoutes"

Symptom: TLSRoute objects disappear from kubectl get after the CRD update. Cause: the v1.6.1 Standard channel's TLSRoute CRD no longer serves v1alpha2, so existing objects are unreadable by the API server. Fix: restore from backup and install the Experimental channel TLSRoute CRD, which still includes v1alpha2. This is why the runbook backs up TLSRoutes before any CRD change.

5. "Node shows veth on a 6.8+ kernel with datapathMode=auto"

Symptom: cilium-dbg status reports Device Mode: veth [Configured: auto] on modern nodes. Cause: bpf.tproxy=true is set — auto mode deliberately reverts to veth when netkit and eBPF-TPROXY conflict. Fix: decide which feature you want: netkit (drop bpf.tproxy=true) or eBPF TPROXY (accept veth). The 1.20 behavior is documented, not a bug.

Performance: What the Datapath Actually Buys You

Claiming "eBPF is faster" without mechanisms is marketing. The measurable mechanisms in this stack:

What to measure before/after: p50/p99 of pod-to-pod TCP round trips (careful: ztunnel enrollment adds the TLS handshake to first-connection latency by design), Envoy CPU per 1k RPS of L7-selected traffic, and node boot-to-schedulable time. Baseline them on your workloads — the numbers that matter are yours.

Migrating Off Ingress NGINX: The Actual Playbook

The Cilium 1.20 release notes make an aggressive pitch: "If you are still running Ingress NGINX, now is the time to let the CNI you already run take on that traffic management too." Here is what that migration actually looks like, including the parts that hurt.

The mechanical translation

Every NGINX annotation has a Gateway API equivalent, but they are not 1:1 — a rewrite-based path routing becomes a URLRewrite filter, a TLS-secret annotation becomes certificates.refs, and canary weights become weighted backendRefs (or, for selectorless Services, the Maglev slice weights from the traffic section). The docs ship a full "Migrating from Ingress to Gateway" walkthrough with the annotation-to-field mapping tables — start there, not from scratch.

# Before: NGINX Ingress with the usual annotation pile
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: payments
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls: [{hosts: [payments.example.com], secretName: payments-tls}]
  rules:
  - host: payments.example.com
    http:
      paths:
      - path: /v1(/|$)(.*)
        pathType: ImplementationSpecific
        backend: {service: {name: payments, port: {number: 8080}}}
---
# After: HTTPRoute on the shared Cilium Gateway
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: payments
  namespace: payments
spec:
  parentRefs: [{name: shared-gateway, namespace: gateway-system}]
  hostnames: ["payments.example.com"]
  rules:
  - matches:
    - path: {type: PathPrefix, value: /v1}
    filters:
    - type: URLRewrite
      urlRewrite:
        path: {type: ReplacePrefixMatch, replacePrefix: /}
    backendRefs: [{name: payments, port: 8080}]

What gets better

• One Envoy DaemonSet instead of ingress-controller replicas to size and upgrade
• Namespace-scoped routes — app teams own their objects, RBAC falls out naturally
• Portable manifests: same HTTPRoute runs on Envoy Gateway, Traefik, Istio
• L7 policy and ingress share one datapath and one Hubble view (drops at the edge are visible with policy correlation)

What hurts

• Every chart that templates ingressClassName: nginx and annotations needs a values refactor
ImplementationSpecific path types must be re-decided explicitly as Exact/PathPrefix/RegularExpression
• Custom snippets / Lua / server-block configs have no Gateway API equivalent — they become separate routes, CiliumEnvoyConfig (admin-only, minimal validation), or app code
• You are now on Gateway API release cadence as well as Cilium's — track CRD versions deliberately

Rollout pattern that works: run both controllers in parallel — Gateway API's Gateway and NGINX's IngressClass can coexist — and cut over host by host via DNS weights. Keep NGINX until every annotation-driven behavior is verified on the Gateway in Hubble (the flow-level comparison between the two paths is your acceptance test), then decommission the controller and reclaim its resource requests.

References & Further Reading