vLLM 0.30.0: The Engine Restart Problem Gets a Daemon, and Your Watermark Policy Gets a Trust Hole

September 22, 2026 · By Yuval Glasman

Sources

vLLM 0.30.0 went out at 05:20 UTC this morning — 762 commits from 315 contributors, 104 of them first-timers — and it is one of those releases where the headline features are genuinely architectural rather than cosmetic. Three of them matter to anyone operating inference fleets: a persistent per-GPU weight-cache daemon that makes engine restarts a CUDA IPC mapping instead of a disk reload (PR #54921), native Gumbel-max text watermarking with a keyed PRF and detection primitives (PR #54053, design in RFC #53916), and HiSparse, a host-resident tier for sparse-MLA decode that spills KV pages to pinned host memory under GPU pressure (PR #53781). There is also a breaking-changes list that will bite automation: scale-out endpoints are now opt-in, GPTQ activation ordering is gone, and a YaRN alignment silently shrinks max_model_len on several vendor checkpoints.

This is the release where vLLM stops pretending that a model reload from disk is an acceptable unit of operational failure recovery, and starts shipping the memory-management machinery that disaggregated serving stacks needed all along. If you run vLLM behind a gateway as one of several model backends (see our LLM router and gateway guide for the surrounding topology), read the breaking changes before you bump the tag.

What's Actually In It

The changes that matter to platform engineers, by blast radius:

Change	Type	PR	Who it hits
Fast Start: per-GPU weight-cache daemon, `--load-format ipc_cache`	Architecture	#54921	Fleets with frequent engine restarts / autoscaling churn
Gumbel-max watermarking + detection, keyed PRF	Feature	#54053	Anyone with AI-content-provenance obligations
HiSparse host-resident KV tier for sparse-MLA decode	Architecture	#53781	Sparse-MLA model operators (DeepSeek-V4 lineage)
MRV2 dual-batch overlap, eager + FULL CUDA graphs	Performance	#50945, #51700	Everyone on Model Runner V2
GC frozen during CUDA-graph capture (capture 12s→2s, init 28.9s→8.2s on H200)	Performance	#54646	Everyone; this is free startup time
Scale-out endpoints (`/render`, `/derender`, `/inference/v1/generate`) now opt-in via `--enable-scale-out`	Breaking	#54579	Anything that automated against scale-out endpoints
GPTQ `g_idx` activation ordering removed	Breaking	#54809	Older GPTQ checkpoints and their kernels
YaRN aligned with Transformers — vendor aliases stop re-scaling `max_model_len`	Breaking	#56446	TeleChat3-36B-Thinking (131072→32768), sarvam-105b (5242880→131072)
`VLLM_PREFIX_CACHE_RETENTION_INTERVAL` and `VLLM_MM_HASHER_ALGORITHM` env vars removed	Breaking	#55353	Deployments driving config via env instead of config fields
`all` Mamba cache mode deprecated, falls back to MRV1	Deprecation	#55041	Hybrid-Mamba operators using `all` mode
`python -m vllm.entrypoints.grpc_server` deprecated → `vllm serve --grpc`	Deprecation	#56746	gRPC serving setups with custom launchers
Validation-error response bodies bounded (~5300x amplification closed)	Security	#54684	Public endpoints — DoS surface reduction

Fast Start: Killing the Disk Reload

The pain is old and well understood: engine restarts are dominated by weight loading from disk. Every crash-loop, every autoscaling event, every config rollout pays the deserialization and quantization tax again. PR #54921 (Siyu Liu, @liusy58, with Michael Qiu of Ant Group) answers it with a persistent weight-cache daemon per GPU that holds post-quantized, TP-sharded weights in GPU memory. A restarting engine maps them over CUDA IPC — zero-copy — instead of touching disk. In zero_copy mode the engine literally shares the daemon's GPU allocations; process_weights_after_loading is skipped entirely, which is where the startup win comes from.

                    one node (TP=2)
  ┌───────────────────────────────────────────────┐
  │                                               │
  │   weight-cache daemon (rank 0)  daemon (rank 1)│
  │        holds quantized, sharded weights         │
  │        in GPU 0 / GPU 1 memory, resident        │
  │            │ socket        │ socket            │
  │            ▼               ▼                    │
  │   engine restart #N ── CUDA IPC ──► maps       │
  │   tensors zero-copy, skips process_weights,     │
  │   serves in seconds instead of minutes          │
  │                                               │
  └───────────────────────────────────────────────┘
   disk is read ONCE, by the daemon itself

Operationally it is two processes. The daemon loads from disk (its own ipc_cache as a load format is explicitly rejected with a ValueError), forms its own TP group, and binds a Unix socket only once the model is fully cached — so engines can't attach to a half-loaded daemon:

# Daemon: persistent, per-GPU, holds weights in VRAM.
# Must load from disk (default --load-format); ipc_cache here is an error.
export MODEL=/models/Qwen3.5-122B-A10B-FP8/
python -u -m vllm.model_executor.model_loader.weight_cache.daemon \
    --model "$MODEL" \
    --tensor-parallel-size 4 \
    --trust-remote-code \
    --weight-cache-socket-dir /var/run/vllm-wcache

# Multi-node TP: the daemon holds its own TP rendezvous group open,
# so it needs a master port distinct from the engine's --master-port:
#   --weight-cache-master-port 19527   (required when --nnodes > 1)

vllm serve "$MODEL" \
  --load-format ipc_cache \
  --tensor-parallel-size 4 \
  --model-loader-extra-config '{"socket_dir": "/var/run/vllm-wcache", "mode": "zero_copy", "fallback": false}' \
  --gpu-memory-utilization 0.15 \
  --max-model-len 8192 \
  --enforce-eager \
  --trust-remote-code \
  --port 34000

The extra config surface is small and worth knowing by heart: socket_dir (where the daemon sockets live), mode (zero_copy default, copy alternative), and fallback (fall back to disk loading when the daemon is unavailable). New this release: FP4 checkpoints are cacheable (#55465), multi-node TP works (#55468), socket folders are keyed by GPU UUID so multi-instance nodes don't collide (#56669), and per-client IPC tensor export means a copy-mode client can no longer release the daemon's zero-copy weights (#56472). Non-CUDA/ROCm platforms get a clear error instead of a confusing one (#56010).

The hidden costs — read before you deploy

VRAM residency is the trade. In zero_copy mode the weights live in the daemon's CUDA IPC allocations permanently. Notice the example above sets --gpu-memory-utilization 0.15 for the engine — the rest is spoken for by the daemon. Your effective per-GPU model budget changed; capacity planning must now count daemon + engine, and the daemon refuses to die politely when you want the VRAM back for a different model.
Sleep mode is incompatible. Zero-copy IPC pins the engine to the daemon's allocations, so CuMemAllocator weight offloading (sleep mode) must not be used with ipc_cache — the loader source states this explicitly. RL/eval loops that rely on sleep-to-swap-models need copy mode or a different strategy.
TP and EP only. The daemon rejects data parallelism (data_parallel_size > 1 raises ValueError — "The weight cache daemon only supports tensor and expert parallelism"). DP fleets attach one daemon per replica topology, which multiplies resident VRAM.
The daemon is single-writer. It acquires a lock on its socket path so a second daemon cannot remove a live socket and hijack the path — good — but it also means daemon lifecycle is now your lifecycle problem: it's a long-running process that needs supervision, restart policy, and a place in your memory budget. This is new operational surface, not a free lunch.

Watermarking: Gumbel-Max, Keys, and the Trust Hole

PR #54053 (by @TQCB, design fully specified in RFC #53916, which leans on the Gumbel-max watermarking literature — e.g. arXiv:2410.20418) makes watermarking an engine-level, not application-level, concern. Enabled via config at startup:

vllm serve MODEL \
  --watermark-config '{"algorithm":"gumbel","key":42}'

The config surface (from vllm/config/watermarking.py) is deliberately narrow: algorithm is gumbel (default) or dual_key_gumbel, the PRF is philox with a 64-bit key, context_width defaults to 4 prior tokens per watermark decision (values above 16 emit a warning — larger contexts are more edit-fragile), and deduplicate_contexts defaults to single_turn with a recommended history of at least 1024 positions. When watermarking is configured, it is enabled for requests by default — and here is the part your security team needs to read: requests can opt out with SamplingParams(watermarking=False) in Python, a watermarking: false field on the OpenAI-compatible and Rust chat/completions APIs, or in sampling_params on the Rust token API. The watermarking docs are blunt about it:

The trust boundary, in the project's own words

"Deployments that require watermarking must restrict this field to trusted callers, or strip and validate it at the ingress boundary, so untrusted clients cannot opt out." Translation: a per-request escape hatch is a per-request bypass. If watermarking is a compliance requirement rather than a nice-to-have, the field must be stripped at your gateway or middleware before it ever reaches the sampler — same discipline as capping max_tokens or pinning temperature for SLA tiers.

Mechanically, Model Runner V2 constructs a Watermarker from the config, and GPUWatermarkSampler invokes it for the final stochastic token selection — after temperature, min-p, top-k and top-p are applied. Detection is decoupled from generation: WatermarkDetector consumes token IDs, and callers are responsible for using the tokenizer and watermark profile that match generation. An example detection server ships in the repo at examples/basic/online_serving/watermark_detection_server.py.

Two footguns worth pinning in runbooks:

Speculative decoding dilutes the signal. Plain gumbel is rejected with speculative decoding before model loading; dual_key_gumbel supports it natively. Even then, accepted draft tokens are not watermarked, so the watermark signal is diluted in proportion to the share of output tokens supplied by accepted drafts (rejected drafts don't dilute — their recovery tokens are watermarked). allow_target_only_watermarking: true forces the combination at the cost of weaker detectability.
Dedup misconfiguration causes repetition loops. The config validator itself warns: gumbel with deduplicate_contexts='none' or a history under 1024 positions "may increase the frequency of degenerate generations, including repetition loops." That's a quality-vs-robustness dial that will be discovered in production, not in staging.

HiSparse: A Host Tier Under the Sparse-MLA KV Cache

Sparse-MLA models (the DeepSeek-V4 lineage) keep an indexer on GPU and attend to a top-k of positions — which means the KV that misses the GPU capacity budget is still valuable, just not continuously. PR #53781 (Matthew Bonanni, @MatthewBonanni) adds HiSparseConnector: a host-resident tier that spills KV pages to pinned host memory only when GPU capacity must be reclaimed, and serves top-k misses from a per-request GPU hot buffer. Unlike the always-host-resident design it supersedes (#46326), it keeps as much KV as possible in the normal device cache and treats host as overflow, not primary.

resident GPU pages
        |
        | spill under GPU pressure
        v
  pinned host pool ── selected top-k misses ──► per-request GPU hot buffer
        ^                                              |
        |                                              v
        +──────────────── prefix reuse ────────── sparse attention

A fused CUDA resolver maps each selected position through:
  1. a normal resident GPU page       (returns first)
  2. an existing hot-buffer row       (updates GPU-side LRU)
  3. the pinned host pool             (gathers ONLY the selected rows)
Resolution, replacement and LRU updates stay on the GPU and
are CUDA-graph-replay compatible.

Wiring it up (exact shape from the repo's own eval configs):

vllm serve nvidia/GLM-5.2-NVFP4 \
  --tensor-parallel-size 4 \
  --max-model-len 8192 \
  --max-num-seqs 32 \
  --max-num-batched-tokens 4096 \
  --enforce-eager \
  --kv-cache-dtype bfloat16 \
  --attention-config '{"hisparse_config":{}}' \
  --kv-transfer-config '{"kv_connector":"HiSparseConnector","kv_role":"kv_both","kv_connector_extra_config":{"host_pool_gib":32}}'

The ownership split is the interesting design decision: the normal KV cache manager keeps allocating and freeing GPU blocks (it owns residency leases); HiSparseCoordinator owns logical host blocks, host-prefix identity, and spill policy; HiSparseRuntime owns the hot views and the GPU LRU. Host blocks use the same token hashes as their resident counterparts, but hashes are published only after contents are durable — an in-flight host copy can't be reused as a prefix hit. NIXL remains the P/D transport, and HiSparse only decides the decoder-side landing policy: imported prefixes that fit resident pools land on GPU; larger ones land in the registered host pool with the indexer still imported to device.

Cost accounting warning

host_pool_gib is the usable host-cache capacity per data-parallel replica, not a node-wide memory budget — and TP ranks hold replicated views of that logical cache. A node running several DP replicas each with host_pool_gib: 32 is committing multiples of that, with physical layout topology-dependent. The design doc is explicit that realized capacity may run slightly smaller because the budget rounds down to complete host blocks. Overcommit host memory here and the OOM killer, not vLLM, will make the capacity decision. Also: the design doc is stamped "Status: experimental" — treat it as such in fleet rollouts.

Model Runner V2: The Quiet Performance Haul

Beneath the headlines, the V2 engine core took a real step. Dual-batch overlap landed in eager mode (#50945) and with FULL CUDA graphs for microbatched steps (#51700) — both from first-time contributor @specture724. MTP and EAGLE3/DFlash/DSpark speculative decoding now run under pipeline parallelism, and every draft-model speculator gets adaptive verification through an online acceptance estimator (#52228).

The most relatable fix: freezing the Python GC during CUDA-graph capture (#54646, njhill) cuts capture time from 12s to 2s and engine init from 28.9s to 8.2s on an H200 — that is a straight 20 seconds off every cold start, on every rollout, for free. And if you run RL loops, the --return-sampling-mask path was compacted on GPU, fixing a roughly 2x RL step-time regression (#54901).

Model-side performance work is deep but narrow: DeepSeek-V4.1-Flash gets its whole KV stored in MXFP8 through the FlashMLA V4.1 record on SM100 (#56893) with DeepGEMM Mega-mHC (#56962); GLM-5.3-Flash gets FlashKDA for chunked prefill at 1.7–3.8x the Triton path (#55737); Kimi K3 drops mixed-batch gather/scatter for 5.2–7.7% E2E throughput (#56159) and gets a 4–6x kernel speedup at small batch from grouped FP8 MLA cache insertion (#55356); Hopper MoE tuning on H20 lands +21% (#54668). New model support includes DeepSeek-V4.1-Flash, DeepSeek-V4-Flash-Vision-Exp, GLM-5.3-Flash, K2-Horizon, Cohere Compass, Bailing V3 VL, Nanbeige4.2 — and, notably, a DeepSeek-V4 CPU backend with AVX512/AMX sparse MLA kernels (#55355).

Breaking Changes: The Migration Tax

What broke	What you had	What you do now
Scale-out endpoint registration	`/render`, `/derender`, `/inference/v1/generate` always registered; `VLLM_ENABLE_SCALE_OUT_ENDPOINTS`	Pass `--enable-scale-out` (`vllm launch render` and `vllm serve --tokens_only` still register them); env var is gone (#54579, #55176)
YaRN `max_model_len` re-scaling	Vendor YaRN aliases multiplied `max_position_embeddings` a second time — inflated context	Aligned with Transformers; TeleChat3-36B-Thinking drops 131072→32768, sarvam-105b drops 5242880→131072; `attn_factor`/`extrapolation_factor` ignored in favor of `mscale`/`attention_factor` (#56446)
GPTQ activation ordering	`g_idx` respected; Marlin/GPTQ/CPU/RDNA3 kernels for it	`g_idx` ignored, kernels deleted — re-quantize affected checkpoints (#54809)
0.29-deprecated env vars	`VLLM_PREFIX_CACHE_RETENTION_INTERVAL`, `VLLM_MM_HASHER_ALGORITHM`, `use_fp4_indexer_cache` alias, `CUDA_VISIBLE_DEVICES` ROCm fallback	Use the config fields; on ROCm use `HIP_VISIBLE_DEVICES` (#55353)
gRPC server launch	`python -m vllm.entrypoints.grpc_server`	`vllm serve --grpc` (#56746)
DCP backend validation	Silent misconfiguration	Attention implementations must declare DCP support; DCP with ROCm standard attention, Triton, FlexAttention or TurboQuant now fails at backend selection (#55780)
Audio pipeline	PyAV default resampler	torchaudio default; decoding backend-selectable via `--media-io-kwargs` `audio_backend`, soundfile-first in `auto` (#52598, #51826)

The YaRN change is the sneaky one: nothing crashes. Your vendor checkpoint that claimed 5M context now derives 131k, and the first symptom is prompt-length validation failures on traffic that worked yesterday. If you serve the affected checkpoints, audit your max_model_len assumptions before upgrading, and budget explicit --max-model-len overrides if the base model's true RoPE scaling supports them.

Security: The Unsexy Fixes That Matter

For anyone exposing vLLM beyond localhost, the security section of the notes is the real reason to schedule this upgrade:

Validation-error response bodies are bounded, closing a roughly 5,300x response amplification vector (#54684) — a request that triggered deep validation errors could previously drag megabytes of echo back per request.
Client-supplied sparse embeddings are bounded before densification (#54632).
Request-controlled video sampling is capped for the GLMGA (#54935) and Qwen-VL (#56729) backends — a client could previously steer server-side decode work.
cache_salt is validated before reaching LMCache "so one request cannot take down the engine" (#51444) — the phrasing tells you what the failure mode looked like.
Credentials are redacted from benchmark logs (#56662) and Rust frontend HTTP method labels are normalized to prevent unbounded Prometheus series cardinality (#56058).

Platform Notes

CUDA 13.0 is the default for wheels and the vllm/vllm-openai Docker images; CUDA 12.9 variants ship as v0.30.0-cu129, with Ubuntu 24.04 variants for both. There's a public CUDA 13.4 Rubin build path (#54640, #56545).
ROCm moves to TheRock on ROCm 10.0 (#55246) with AITER 0.1.21.post2 (#52826).
CPU images move to Ubuntu 25.04 with GCC 15 (#49410), add a dedicated Zen5 image (#50314) and nightly CPU images on Docker Hub (#55163).
Dependency floors moved: Transformers 5.16.1 (#53905), CUTLASS 4.7.1 (#54190), openai>=2.25.0, huggingface_hub>=1.31.0 (#56460). If your image pins any of these, unpin or bump.
Rust frontend grows real surface: TLS for the render server (#54999), HTTP RL weight synchronization (#56567), and the vllm-proto crate published to crates.io (#56365).

Install is unremarkable: pip install vllm from PyPI, or the image tags above. Docs live at docs.vllm.ai. If you run vLLM under Kubernetes, our AI fleet architecture patterns guide covers the surrounding scheduling and capacity topology.

Who Should Actually Upgrade Now

Upgrade now if you operate restart-heavy fleets (autoscaling, crash recovery, frequent config rollouts) on NVIDIA — Fast Start plus the GC-capture fix is a material cold-start reduction, and the security fixes argue for the bump on their own. Do the VRAM capacity math first.
Upgrade deliberately if you serve vendor YaRN-aliased checkpoints (TeleChat3, Sarvam) or GPTQ checkpoints with activation ordering — your context windows and quantized weights change behavior, and that needs a staged rollout, not a tag bump.
Wait for 0.30.x patch settles if you're on a stable 0.29 serving dense models without sparse MLA: the headline features here (Fast Start, HiSparse, MXFP8 KV, DBO) target sparse-MLA-era models and GPU-dense restarts, so the breaking-change tax buys you little. The one exception is public-facing endpoints — the response-amplification fix (#54684) is worth taking early regardless of model.
Watermark adopters: treat this as compliance infrastructure, not a feature — the per-request opt-out means your watermark policy is only as strong as your ingress stripping, and speculative decoding dilutes detectability in proportion to accepted-draft share. Budget for detection tooling, not just generation flags.

Credits

Release highlights worth naming because the architecture came with them: Siyu Liu (@liusy58) and Michael Qiu on Fast Start (#54921); @TQCB on Gumbel-max watermarking (#54053); Matthew Bonanni (@MatthewBonanni) on HiSparse (#53781); njhill on GC-free graph capture (#54646) and the Mamba cache-mode deprecation (#55041); hmellor on the YaRN alignment (#56446); Roderick-Wu on the GPTQ removal (#54809); franciscojavierarceo on scale-out gating (#54579); lzhan011 on the response-body bounding (#54684); and first-timer @specture724 landing dual-batch overlap in both eager and FULL-graph modes (#50945, #51700). 104 of the 315 contributors were new — the ecosystem math that keeps vLLM, a PyTorch Foundation hosted project, ahead of closed engines on integration depth.