GPU capacity planning for LLM inference is broken in most orgs, and the failure looks identical everywhere: a fleet of H100s provisioned for peak traffic sits at 5–15% utilization while a queue somewhere quietly breaches its SLO. The instinctive fix — buy more GPUs or shard harder — treats the symptom. The real problem is that capacity planning for inference is a memory problem disguised as a compute problem, and almost every tool in your stack treats it as the opposite.

This guide is the planning playbook that falls out of what production fleets actually do in 2026: Cloudflare packing 13 models onto a single GPU at ~400% memory over-commit, Character.AI redesigning attention before buying more hardware, and Kubernetes graduating DRA's prioritized device lists to stable in 1.36 so a claim can say “give me an H100, fall back to an A100 if none are free.” The order of operations matters: architecture first, partitioning second, elastic fleet last. Doing it in reverse is how you end up with a $200k/month fleet serving a product that never needed it.

TL;DR

  • Capacity for inference is KV-cache memory, not FLOPs. A GPU is “full” when its KV pool can’t hold more concurrent conversations, long before its compute saturates. Plan in tokens-resident, not tokens/second.
  • DRA finally makes GPUs a schedulable resource. The device-plugin integer model is gone: DRA went GA in 1.34, the gate locked on in 1.35, and 1.36 made prioritized fallback lists stable — ordered preference (“H100, else A100”) instead of one hardcoded device model.
  • Time-slicing and MIG are the bin-packing tools. The NVIDIA DRA driver exposes GPU sharing (time-slicing, MPS) and MIG slicing directly through DeviceClasses, so a 10GB inference pod can claim 1/7th of an H100 instead of a whole card.
  • Over-commit is a real technique with a real price. Cloudflare’s Omni runs 13 models per GPU by intercepting CUDA allocations into unified memory — and pays ~156ms cold-swap latency when a swapped-out model is called. Over-commit long-tail models; never over-commit your flagship.
  • Scale on queue size, not GPU utilization. Google’s own GKE guidance ranks queue size and batch size above GPU utilization because utilization doesn’t map to latency. A static 16-GPU fleet costs $1,532/day at Lambda rates; queue-driven elasticity at the same SLO costs $958/day — a $17,237/month difference (Decimal-verified, 2026-09-28 rates).
  • When NOT to pack: latency-critical interactive flagship serving, training jobs, and any fleet under 8 GPUs — the orchestration tax exceeds the savings. Numbers below.

The problem: GPUs are scheduled as integers, capacity is a memory curve

The Kubernetes device-plugin model that most fleets still run treats a GPU as an opaque countable integer: a pod either gets nvidia.com/gpu: 1 or nothing. That model has no way to express sharing, per-workload configuration, capability constraints, or topology requirements. Meanwhile the workload’s actual constraint is a memory curve: how many tokens the KV cache can hold for the model’s context length, which sets the concurrency ceiling of the card.

Run the numbers for a single H100 80GB serving a 32B dense GQA model in bf16. We computed the per-token KV cost and pool size for our vLLM review with Decimal-verified math; the shape of the result is what matters here:

Per-token KV bytes (dense GQA, 32B-class, 64 layers):
  8 KV heads x 128 head_dim x 64 layers x 2 (K+V) x 2 bytes
  = 262,144 bytes/token = 256 KiB/token

Usable H100 80GB memory pool at utilization 0.92:
  80 GiB x 0.92 ~= 73.7 GiB

  - model weights (32B x 2 bytes)      ~64.0 GiB   * does not fit; use fp8/AWQ
  - 8B-class model instead (16 GiB)     ~16.0 GiB
  ---------------------------------------
  KV pool for 8B model                 ~57.7 GiB

Concurrent conversations (8K context, 8K gen):
  57.7 GiB / (16,384 tokens x 256 KiB) ~= 14 concurrent streams