Platform Monkey ctx

The Inference Gateway Pattern: KV-Cache-Aware Routing and Flow Control for Multi-Model GPU Fleets

· By
Sources

On October 20, 2026, llm-d shipped v1.0 — a stable, versioned configuration API for distributed LLM inference on Kubernetes, with multi-tenant quality of service, zero-downtime upgrades, and agent-aware serving as the headline features. Two weeks of perspective later, the release holds up as something bigger than a version bump: it is the moment the inference gateway — the layer that routes and flow-controls requests inside your GPU fleet — stopped being a research proposal and became an operating dependency. Google Cloud builds the GKE Inference Gateway on the llm-d Endpoint Picker. AWS runs disaggregated inference on SageMaker HyperPod with it. IBM Research used it to serve a 753B-parameter model to 3,000 concurrent coding agents on 544 H100 GPUs, at 5–10× lower cost per token than commercial APIs.

This article is the architectural deep-dive on that layer: the Inference Gateway pattern — an L7 gateway coupled with a scheduling brain that picks the specific GPU pod for every request and queues traffic in front of the fleet when the fleet is full. We will cover the exact mechanics: the Gateway API Inference Extension contract, the llm-d Endpoint Picker's Filter–Score–Pick scheduling pipeline, the three-tier flow-control dispatch loop, the header protocol on the wire, and complete manifests for a multi-model fleet. We will also cover where this pattern breaks — and it does break, in specific, documented ways that the marketing material underplays.

Three Gateways, Three Problems

The term “AI gateway” is doing quadruple duty in the industry right now, and most of the confusion in this space comes from conflating three different layers with three different jobs:


                    NORTH-SOUTH: API MANAGEMENT          SOUTH: FLEET SCHEDULING
  ┌────────────┐   ┌───────────────────────────────┐   ┌──────────────────────────────┐
  │  Client    │──▶│  AI GATEWAY (LiteLLM-class)   │──▶│  INFERENCE GATEWAY (this     │
  │  apps,     │   │  - API keys, budgets          │   │  article)                    │
  │  agents    │   │  - Provider routing/fallback  │   │  - Per-request pod selection  │
  │            │   │  - Guardrails, cost telemetry │   │  - KV-cache affinity routing  │
  └────────────┘   │  - North-south, multi-cloud   │   │  - Queueing + flow control   │
                    └───────────────────────────────┘   │  - South, one cluster        │
                                                        └──────────────┬───────────────┘
                                                                       │ per-pod, L7
                                                        ┌──────────────▼───────────────┐
                                                        │  SERVING TOPOLOGY (LWS/P-D)   │
                                                        │  - Multi-host model sharding  │
                                                        │  - Prefill/decode disaggregation
                                                        │  - GPU placement (DRA, MIG)   │
                                                        └───────────────────────────────┘

The AI gateway layer — LiteLLM, Portkey, Helicone — manages API keys, budgets, provider fallbacks, and cost attribution for traffic crossing your trust boundary. We covered it in a separate deep-dive. The serving topology layer — LeaderWorkerSet, prefill/decode disaggregation, cell-based fleets — decides how many hosts a model spans. Both assume someone else solves the question in between: given one fleet of GPUs and thousands of concurrent requests, which specific pod gets this request, and what happens to the requests that don't fit?

That is the inference gateway's job, and it is the layer this article is about. It is not an API management proxy. It is a scheduler at the request level, deployed as a standard Kubernetes Gateway API extension, with real-time telemetry from every model server: queue depth, KV-cache utilization, prefix-cache state, and per-session affinity. The GAIE project's own definition is precise: an Inference Gateway is “a proxy/load-balancer which has been coupled with an Endpoint Picker,” providing optimized routing and load balancing for self-hosted AI workloads on Kubernetes.

Why Round-Robin Wastes Your Fleet

A Kubernetes Service load-balances inference requests the same way it load-balances REST calls: roughly round-robin across ready endpoints. For LLM serving this is actively harmful, for reasons rooted in the mechanics of the workload:

The measured consequences of fixing this are not subtle. Tesla and Red Hat's KServe case study measured 3× higher output throughput and 2× lower time-to-first-token from prefix-cache-aware routing versus round-robin on Llama 3.1 70B. Google's production numbers on GKE: time-to-first-token cut by more than 35% and prefix-cache hit rate doubled from 35% to 70% after switching from a plain Kubernetes Service to the llm-d Endpoint Picker. The IBM fleet study found precise prefix matching alone delivered 79% higher throughput and 67% lower TTFT than approximate matching on the same hardware.

None of these gains come from the GPUs. They come from not wasting the work the GPUs already did.

The Anatomy of an Inference Gateway

The pattern has four moving parts, standardized across the Gateway API Inference Extension (GAIE) and the llm-d implementation:

Note the organizational split, because it matters for what you deploy in 2026: the InferencePool API and the conformance test suite live in kubernetes-sigs (current release v1.6.2, September 2026), while the full-featured EPP — flow control, KV-aware scheduling, the latency predictor — and the InferenceObjective API moved to the llm-d/llm-d-router repository (llm-d Router v1.0.0). GAIE retains only a lightweight reference EPP (LWEPP) for conformance testing. The split was proposed and executed in GAIE issue #2430. In practice: install the InferencePool CRD from GAIE, install the router from the llm-d OCI charts, and the two meet at the CRD.


  Client (OpenAI-compatible SDK)
      │
      ▼
┌─────────────────────────────────────────────────────────────────────┐
│  GATEWAY (Envoy / Istio / GKE ALB)                                  │
│  - terminates TLS, matches HTTPRoute rules                          │
│  - backendRef points at an InferencePool, NOT a Service             │
│  - ext_proc filter: parks the request, opens gRPC stream to EPP     │
└───────────────┬─────────────────────────────────────────────────────┘
                │ ext-proc (FULL_DUPLEX_STREAMED gRPC)
                ▼
┌─────────────────────────────────────────────────────────────────────┐
│  ENDPOINT PICKER (EPP)                        llm-d.ai/v1           │
│                                                                     │
│  1. Parse request (model, body, headers, objective)                 │
│  2. FLOW CONTROL: classify by FlowKey = (fairness-id, priority)    │
│     hold in per-flow queues while pool is saturated (TTL-bounded)  │
│  3. SCHEDULER: Filter -> Score -> Pick over candidate pods          │
│     - prefix-cache affinity, KV utilization, queue depth,          │
│       session affinity, predicted latency (TTFT/TPOT regression)     │
│  4. Return pod address + port to the proxy                          │
└───────────────┬─────────────────────────────────────────────────────┘
                │ chosen pod IP:port
                ▼
┌─────────────────────────────────────────────────────────────────────┐
│  INFERENCE POOL (label-selected pods, e.g. 5x H100 vLLM replicas)   │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐  │
│  │ vLLM pod │ │ vLLM pod │ │ vLLM pod │ │ vLLM pod │ │ vLLM pod │  │
│  │ :8000    │ │ :8000    │ │ :8000    │ │ :8000    │ │ :8000    │  │
│  │ /metrics │ │ /metrics │ │ /metrics │ │ /metrics │ │ /metrics │  │
│  └──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘  │
│   EPP data layer polls each pod's Prometheus metrics + watches       │
│   the Kubernetes API for pool membership changes                    │
└─────────────────────────────────────────────────────────────────────┘

The Request Path, Wire by Wire

The only ext-proc body mode the EPP supports is FULL_DUPLEX_STREAMED: the proxy streams the request headers and body to the EPP over a long-lived gRPC stream, and the EPP's decision comes back as an ext-proc response carrying the selected endpoint. The EPP's request handler parses the payload into an internal structure — out-of-the-box parsers cover the OpenAI HTTP formats and vLLM's gRPC API — and the rest of the pipeline is protocol-agnostic.

Classification happens through HTTP headers, which upstream services (your API gateway, an identity provider, an Envoy authz filter) inject after stripping any client-supplied copies:

curl -X POST "http://${GATEWAY_IP}/v1/chat/completions" \
  -H 'Content-Type: application/json' \
  -H 'x-llm-d-inference-objective: premium-traffic' \
  -H 'x-llm-d-inference-fairness-id: tenant-a' \
  -H 'x-llm-d-slo-ttft-ms: 1200' \
  -d '{"model": "Qwen/Qwen3-32B", "messages": [{"role": "user", "content": "Hello"}]}'

The full header protocol (canonical names, llm-d Router v0.9.0+): x-llm-d-inference-objective names the InferenceObjective to apply; x-llm-d-inference-fairness-id is the tenant key for fairness (missing values fall into a shared default-flow bucket); x-llm-d-slo-ttft-ms and x-llm-d-slo-tpot-ms declare latency targets used by admission and ordering plugins; x-llm-d-model-name-rewrite overrides the target model server-side. Legacy x-gateway-* names are accepted as deprecated aliases — if both are sent, the canonical x-llm-d-* value wins deterministically.

Responses carry the EPP's error contract in x-llm-d-request-dropped-reason. The prefix convention is worth internalizing for client retry logic: rejected-* values mean no inference work was done (cheap to retry); evicted means GPU cycles were already burned (back off accordingly).

OutcomeHTTPx-llm-d-request-dropped-reasonMeaning
Band/global queue capacity exceeded429rejected-saturatedQueue limits (maxRequests/maxBytes) met — load shedding
In-flight eviction429evictedDispatched negative-priority request terminated to reclaim capacity
TTL expiry (pool has endpoints)429rejected-ttl-expiredRequest queued past its TTL before dispatch
Empty pool / scale-to-zero503rejected-no-endpointsNo serving endpoints exist in the pool
Client disconnect while queued503rejected-context-cancelledCaller went away; slot freed
Graceful drain (rollout)503rejected-shutting-downEPP draining during shutdown — retryable, not a fault
Internal error500rejected-internalFlow controller or scheduler bug — page on rate

One caveat the docs are admirably honest about: this entire contract applies only while the EPP is reachable. If the EPP process dies, Envoy never gets a decision, and behavior is governed by the InferencePool's failureMode — covered below, and it is a bigger deal than it looks.

InferencePool: The Contract Between Gateway and Fleet

The InferencePool (group inference.networking.k8s.io, version v1 in GAIE 1.6.x) is deliberately small. It does two jobs: endpoint discovery for the EPP, and EPP wiring for the Gateway controller. One pool maps to one logical model deployment; the model servers scale independently of the EPP that schedules them.

apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
  name: qwen32b-pool
  namespace: llm-d-fleet
spec:
  # Pods matching this selector (same namespace) are scheduling candidates.
  # Scale the deployment; the EPP's candidate list follows automatically.
  selector:
    matchLabels:
      llm-d.ai/inference-serving: "true"
      app.kubernetes.io/name: qwen3-32b
  # Which port on each pod serves inference traffic (vLLM default: 8000).
  targetPorts:
    - number: 8000
  # Optional: "http" (default) or "kubernetes.io/h2c" for HTTP/2 cleartext.
  appProtocol: http
  # The Gateway controller uses this to configure the proxy's ext_proc
  # filter so every pool-directed request consults the EPP first.
  endpointPickerRef:
    group: inference.networking.k8s.io
    kind: Service
    name: qwen32b-pool-epp
    port: 9002
    # FailClose (API default): proxy rejects requests when the EPP is down.
    # FailOpen (llm-d recipe default): proxy bypasses the EPP and routes
    # directly to a pod - availability over pool protection.
    failureMode: FailClose

The failureMode choice is the most consequential one-liner in the whole pattern, so make it deliberately. With FailClose (the API default), an EPP outage fails requests at the proxy — your fleet is down with the router. With FailOpen (what the llm-d base recipe actually ships, for availability reasons), the proxy silently bypasses the dead EPP and routes requests directly to pods: traffic keeps flowing, but with no flow control, no fairness, no saturation gating — and requests can land on already-saturated backends during the outage window. A busy multi-tenant fleet that suddenly loses tenant fairness is usually a louder incident than a brief availability dip; a single-tenant internal fleet usually prefers FailOpen. llm-d v1.0 mitigates the middle case with router high availability — the EPP runs with a hot standby that takes over scheduling if the active instance fails — but the standby takes seconds to promote, and the failureMode governs what happens inside that window.

Endpoint Scheduling: Filter, Score, Pick

Once a request is admitted, the EPP's scheduler picks a pod through a plugin pipeline. Every scheduling cycle runs one or more SchedulerProfiles; each profile is a list of sequential filters, a set of weighted scorers, and a picker:

Scores combine by weighted sum. The docs' worked example: scorer A (weight 2.0) returns 0.8 and scorer B (weight 1.0) returns 0.5 → the endpoint's final score is 0.8×2.0 + 0.5×1.0 = 2.1. This is the exact config surface — note the real weights from llm-d's flow-control guide, which put 3× the weight on prefix-cache affinity over queue depth:

apiVersion: llm-d.ai/v1
kind: EndpointPickerConfig
plugins:
- type: queue-scorer
- type: kv-cache-utilization-scorer
- type: prefix-cache-scorer
# Optional: switch the scorer from approximate to precise prefix matching,
# backed by the EPP's event-driven KV-cache index:
#   prefixMatchInfoProducerName: precise-prefix-cache-producer
# (without it, prefix-cache-scorer falls back to approx-prefix-cache-producer)

schedulingProfiles:
- name: default
  plugins:
  - pluginRef: queue-scorer
    weight: 2
  - pluginRef: kv-cache-utilization-scorer
    weight: 2
  - pluginRef: prefix-cache-scorer
    weight: 3

Prefix routing has two fidelity modes, and the gap between them is a headline number. Approximate matching tracks prefix state from tokenized request bodies with bounded memory; precise matching uses an event-driven KV-cache index (the precise-prefix-cache-producer data producer) that tracks exact block-level cache state across every server. IBM's fleet measurements: switching approximate to precise delivered 79% higher throughput and 67% lower TTFT, with local prefix hit rate rising from 44.46% to 73.18%. Google's Vertex AI production numbers were in the same class — 35% to 70% hit rate after adopting the llm-d EPP. Precise indexing costs EPP memory and a metrics subscription per pod; it is the first knob to turn when context-heavy agentic traffic dominates.

For prefill/decode disaggregated fleets, the disagg-profile-handler runs two profiles — one over prefill-role pods, one over decode-role pods — and returns both decisions to the proxy: the decode endpoint becomes the primary destination, and the prefill endpoint is injected as a header so the decode engine can pull the KV blocks. The v1.0 release also consolidated a coordination layer into the router so a request flows through its encode, prefill, and decode stages as one managed flow rather than being stitched together by the caller.

Flow Control: The Queue in Front of the GPUs

Scheduling picks where; flow control decides whether and in what order requests may reach the fleet at all. In llm-d v1.0 this is the flowControl feature gate on the EPP, and the design idea is simple: move the queue out of the model servers and into the gateway, where it can see priorities, tenants, and pool saturation at once.

Every admitted request is classified into a FlowKey — a tuple of (FairnessID, Priority). The fairness ID comes from x-llm-d-inference-fairness-id (your tenant or workload identifier); the priority comes from the referenced InferenceObjective. Each unique FlowKey gets its own in-memory queue. Dispatch then walks three tiers in strict order:


                        ┌──────────────────────────────────────────┐
   Request ───────────▶ │ ADMISSION: assign FlowKey                │
   (headers)            │ FlowKey = (fairness-id, priority)       │
                        └───────────────┬──────────────────────────┘
                                        │
      TIER 1: PRIORITY BANDS (hardcoded, serviced in strict order)
   ┌────────────────────────────────────┼─────────────────────────────┐
   │                                    │                             │
   ▼                                    ▼                             ▼
┌─────────────────┐         ┌────────────────────────┐   ┌─────────────────┐
│ Band 100        │         │ Band 0                 │   │ Band -10        │
│ premium-traffic │         │ standard-traffic       │   │ best-effort     │
│ maxReq: 500     │         │ maxReq: 200            │   │ maxReq: 50      │
│ queue: tenant-a │         │ queues: tenant-a, -b   │   │ queue: batch-1  │
└────────┬────────┘         └───────────┬────────────┘   └────────┬────────┘
         │ TIER 2: FAIRNESS within band (round-robin policy)          │
         └──────────────┬──────────────┴────────────────────────────┘
                        │ TIER 3: ORDERING within flow (FCFS / EDF /
                        │          SLO-deadline)
                        ▼
              ┌───────────────────────────┐
              │ SATURATION GATE           │
              │ utilization-detector:     │
              │  queue depth + KV cache   │
              │ concurrency-detector:     │
              │  in-flight request count  │
              │ OPEN  -> dispatch to      │
              │         scheduler         │
              │ CLOSED -> hold (HOL block)│
              └───────────┬───────────────┘
                          ▼
                 Filter -> Score -> Pick -> pod

Three details make this production-grade rather than a textbook exercise:

The saturation gate itself is pluggable, and the choice between the two detectors is a real architecture decision:

Both tuning philosophies converge on the same goal — keep a small “healthy buffer” of requests queued locally on the model servers (just enough that continuous-batching engines never starve) while the bulk of waiting happens centrally, where policy can act on it.

Finally, the aggressive tool: in-flight eviction. Gated dispatch can only refuse new work; it cannot recover capacity already committed. With enableEviction: true, when a higher-priority band is blocked at its ceiling, the EPP terminates an already-dispatched negative-priority request (victim order: lowest priority first, most-recently-dispatched first, so the request with the most accrued work survives longest) and returns 429 with x-llm-d-request-dropped-reason: evicted. Capacity returns only when the model server aborts generation and frees its KV blocks — the EPP assumes a bounded engine reclaim time and does not observe it directly, so verify on your engine that aborted requests actually free capacity promptly before enabling this in production. The evicted caller owns retry and needs idempotency. Reserve eviction for batch/background work whose owner can safely re-run it.

The full flow-control configuration surface, with real values from the llm-d guide (adapted to a 5×H100 fleet):

apiVersion: llm-d.ai/v1
kind: EndpointPickerConfig
featureGates:
- flowControl
plugins:
- type: queue-scorer
- type: kv-cache-utilization-scorer
- type: prefix-cache-scorer
- type: round-robin-fairness-policy     # tenant equity within a band
- type: fcfs-ordering-policy            # swap for edf-/slo-deadline- if you
                                        # carry per-request deadlines
- type: concurrency-detector
  parameters:
    maxConcurrency: 132                 # tune just above effective batch size
    concurrencyMode: requests
    headroom: 0.0

schedulingProfiles:
- name: default
  plugins:
  - pluginRef: queue-scorer
    weight: 2
  - pluginRef: kv-cache-utilization-scorer
    weight: 2
  - pluginRef: prefix-cache-scorer
    weight: 3

flowControl:
  saturationDetector:
    pluginRef: concurrency-detector
  maxBytes: "10Gi"          # global queue memory ceiling -> 429 on breach
  maxRequests: "1k"        # global request-count ceiling
  defaultRequestTTL: "60s"  # queued requests expire, never serve stale work
  enableEviction: true      # reclaim capacity from negative-priority work
  priorityBands:
  - priority: 100           # premium: interactive, latency-sensitive
    maxRequests: "500"
    fairnessPolicyRef: round-robin-fairness-policy
    orderingPolicyRef: fcfs-ordering-policy
  - priority: 0             # standard: most product traffic
    maxRequests: "200"
    fairnessPolicyRef: round-robin-fairness-policy
    orderingPolicyRef: fcfs-ordering-policy
  - priority: -10           # negative = SHEDDABLE: batch, background eval
    maxRequests: "50"
    fairnessPolicyRef: round-robin-fairness-policy
    orderingPolicyRef: fcfs-ordering-policy

The bands bind to pools through InferenceObjective resources. Since Router v1.0.0 the stable API is llm-d.ai/v1, where single-pool spec.poolRef was replaced by spec.poolRefs (1–16 pools) and/or spec.poolSelector; the older v1alpha2 is deprecated with removal announced for v1.2:

apiVersion: llm-d.ai/v1
kind: InferenceObjective
metadata:
  name: premium-traffic
spec:
  priority: 100
  poolRefs:
  - name: qwen32b-pool
---
apiVersion: llm-d.ai/v1
kind: InferenceObjective
metadata:
  name: standard-traffic
spec:
  priority: 0
  poolRefs:
  - name: qwen32b-pool
---
apiVersion: llm-d.ai/v1
kind: InferenceObjective
metadata:
  name: best-effort-traffic
spec:
  priority: -10
  poolRefs:
  - name: qwen32b-pool

Operationally, flow control turns the queue into an honest demand signal. The EPP exports llm_d_epp_flow_control_queue_size, llm_d_epp_flow_control_queue_bytes, llm_d_epp_flow_control_pool_saturation, per-band llm_d_epp_flow_control_capacity_utilization_requests/_bytes, queue-duration histograms labeled by fairness_id, priority, and outcome, plus eviction counters (revocations_issued_total, revocations_total) and Jain's fairness index across programs. Two of these deserve alert rules from day one: revocations_total{outcome="timed_out"} sustained non-zero means the EPP is sizing reclamation against capacity that never came back, and a rising queue_duration_seconds p99 with flat pool saturation means your bands are admitting work the scheduler cannot place.

Multi-Model Fleets: One Gateway, Many Pools

Real fleets do not run one model. A chatbot wants a Qwen-class model, a reasoning path wants DeepSeek, and half a dozen LoRA adapters hang off each base model — all behind one OpenAI-compatible endpoint where the only routing signal is a "model" field in the JSON body. Path-based routing cannot see it.

The llm-d answer is the Inference Payload Processor (IPP), itself an ext-proc service: it extracts the model name from the request body, resolves it to a base model (following LoRA-adapter mappings), and stamps the X-Gateway-Base-Model-Name header. Standard Gateway API header matching then routes to the correct InferencePool, and each pool has its own EPP scoring its own pods:


   Client: POST /v1/chat/completions {"model": "food-review-1", ...}
        │
        ▼
┌────────────────────────────────────────────────────────────────┐
│ GATEWAY                                                        │
│   ext-proc #1: IPP (payload processor)                         │
│     body model "food-review-1" -> base "Qwen/Qwen3-32B"       │
│     sets header X-Gateway-Base-Model-Name: Qwen/Qwen3-32B      │
│                                                                │
│   HTTPRoute rules (header match):                              │
│     Qwen/Qwen3-32B    -> backendRef InferencePool qwen32b-pool │
│     deepseek/DeepSeek-r1 -> backendRef InferencePool r1-pool   │
└───────────────┬───────────────────────────────┬────────────────┘
                │ header match                  │ header match
                ▼                               ▼
┌───────────────────────────┐      ┌───────────────────────────┐
│ qwen32b-pool + EPP #1     │      │ r1-pool + EPP #2          │
│  - Flow control (bands)   │      │  - own bands/queues       │
│  - prefix/session routing │      │  - own prefix index       │
│  - LoRA adapter affinity  │      │  - P/D disagg profiles    │
│  vLLM pods xN (H100 x5)   │      │  vLLM pods xN             │
└───────────────────────────┘      └───────────────────────────┘

The model registry is just labeled ConfigMaps — IPP watches them and resolves both base models and adapters:

apiVersion: v1
kind: ConfigMap
metadata:
  name: qwen-model-mapping
  namespace: llm-d-fleet
  labels:
    inference.llm-d.ai/ipp-managed: "true"
data:
  baseModel: "Qwen/Qwen3-32B"
  # Adapter names must be globally unique across ALL ConfigMaps/pools.
  adapters: |
    - food-review-1
    - travel-assistant

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: model-fleet-route
  namespace: llm-d-fleet
spec:
  parentRefs:
  - name: llm-d-inference-gateway
  rules:
  - matches:
    - headers:
      - type: Exact
        name: X-Gateway-Base-Model-Name
        value: Qwen/Qwen3-32B
    backendRefs:
    - group: inference.networking.k8s.io
      kind: InferencePool
      name: qwen32b-pool
  - matches:
    - headers:
      - type: Exact
        name: X-Gateway-Base-Model-Name
        value: deepseek/DeepSeek-r1
    backendRefs:
    - group: inference.networking.k8s.io
      kind: InferencePool
      name: r1-pool

One Gateway can mix InferencePool backends with ordinary Service backends on different routes, and a single route can carry multiple pool backends with weights — which is how canary rollouts of a new model revision work without a second fleet. Two constraints to respect: every model and adapter name must be globally unique across all pools (IPP resolves by name alone), and do not create pool-level catch-all routes while running header-based routes on the same gateway — the catch-all swallows the header matches.

The Reference Deployment: 5×H100, Two Models, Full Flow Control

Putting the whole pattern together as one deployable unit — a two-model fleet on five H100 nodes with premium/standard/sheddable traffic classes and demand-driven autoscaling. These commands mirror llm-d's own validated guides (GAIE pinned to v1.6.2, Router chart v1.0.0):

# 1. InferencePool CRD set (inference.networking.k8s.io/v1)
kubectl apply -f \
  https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.6.2/v1-manifests.yaml

# 2. Namespace + model pull secret
kubectl create namespace llm-d-fleet
kubectl create secret generic llm-d-hf-token --from-literal=HF_TOKEN=$HF_TOKEN \
  -n llm-d-fleet

# 3. The router: EPP + Envoy sidecar + InferencePool, from OCI charts.
#    Layer your values (flow control config from the section above) on top
#    of the shared base values; provider.name selects the Gateway
#    implementation wiring (istio / gke / envoy).
helm install fleet-router \
  oci://ghcr.io/llm-d/charts/llm-d-router-gateway \
  --version v1.0.0 \
  -f base.values.yaml \
  -f fleet-flow-control.values.yaml \
  --set provider.name=istio \
  --set httpRoute.create=false \
  --set httpRoute.inferenceGatewayName=llm-d-inference-gateway \
  -n llm-d-fleet

# 4. Model servers: your vLLM Deployments (or LeaderWorkerSets for
#    multi-host tensor parallelism) with the pool selector labels.
kubectl apply -n llm-d-fleet -f model-servers.yaml

# 5. Objectives + routes + autoscaler
kubectl apply -n llm-d-fleet -f objectives.yaml routes.yaml scaledobject.yaml

For the router sizing, take the project's own base values seriously: the EPP request spec is 4 CPU / 8Gi memory (16Gi limit), and the Envoy sidecar gets the same — because flow-control queues hold request payloads in memory, the EPP is sized like a cache, not like a controller. The base recipe also pins Envoy to --concurrency 8 and --log-level warn: the chart's default trace-level logging burns a meaningful share of sidecar CPU at high QPS, and an unpinned Envoy spawns worker threads equal to hardware concurrency, oversubscribing the cgroup.

Autoscaling closes the loop. Because flow control surfaces unmet demand as a metric, KEDA can scale the model server Deployment on what is actually waiting rather than on GPU utilization — which, during decode-bound generation, cannot distinguish a lightly loaded server from a saturated one. The recommended shape is two triggers, unmet demand plus active concurrency:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: qwen32b-scaler
  namespace: llm-d-fleet
spec:
  scaleTargetRef:
    name: qwen3-32b-vllm        # the model server Deployment
  minReplicaCount: 2            # warm floor; dampens flap
  maxReplicaCount: 12
  cooldownPeriod: 300
  triggers:
  # Unmet demand: requests held in flow-control queues (requires the
  # flowControl gate; series absent otherwise). AverageValue = per-replica
  # queued-request target; the HPA scales so the aggregate stays under it.
  - type: prometheus
    metadata:
      serverAddress: http://prometheus.monitoring:9090
      query: sum(llm_d_epp_flow_control_queue_size)
      threshold: "1"
  # Active concurrency: in-flight requests across the pool.
  - type: prometheus
    metadata:
      serverAddress: http://prometheus.monitoring:9090
      query: sum(llm_d_epp_request_running)
      threshold: "16"

Both thresholds are per-replica AverageValue targets — the aggregate must exceed threshold × current replicas to justify more capacity — and both are deployment-specific: a larger model or a smaller GPU saturates at lower concurrency, so re-derive them under representative load rather than copying. The min/max bounds and cooldown damp the loop because model server cold starts (weight load) take minutes, not seconds. Below saturation, flow control queues stay empty and the concurrency trigger carries the scaling signal alone; only past the band ceiling does unmet demand surface as queue depth.

What 544 H100s Prove About the Pattern

The strongest public validation of this architecture is IBM Research and Red Hat's September 2026 deployment of GLM-5.2 — a 753B-parameter mixture-of-experts model (~39B active) — across 544 NVIDIA H100 GPUs in a prefill/decode disaggregated topology, serving real agentic coding traffic:

The through-line: none of these numbers came from new hardware. They came from the inference gateway refusing to recompute work the fleet had already done — and from flow control that let the same fleet absorb agent bursts without evicting the interactive traffic sharing it.

Production Realities

Now the part the launch posts soft-pedal. Every one of these is documented in the project's own design docs — which is to the project's credit — but each will bite an operator who deploys the pattern as a black box:

Who Should Skip This Pattern

The Decision Table

Signal in your fleetPattern to reach for
Multi-turn/agent traffic, high context reuse, TTFT is the painInference gateway with prefix-cache scoring (precise producer) + session affinity
Multiple tenants or traffic classes sharing one fleetInference gateway with flow-control priority bands + fairness policy
Several models / LoRA adapters behind one OpenAI endpointInference gateway with IPP + per-model InferencePools and header-matched routes
Model bigger than one host (multi-host TP/EP)Serving topology layer (LeaderWorkerSet / disaggregation) under an inference gateway
Mostly external providers, key/budget/cost control painAI gateway layer — not this pattern
Batch/offline only, no interactive trafficBatch queue + gang scheduler — skip the gateway

References & Further Reading