Distributed LLM Inference With LeaderWorkerSet: v1.0 Mechanics, the DeepSeek-R1 Pattern, and the Toil

Two 1.0 releases landed in the last three weeks that quietly close out a four-year argument in the inference-infrastructure world. LeaderWorkerSet v1.0.0 shipped on October 16, 2026, and llm-d v1.0.0 followed on October 20. The argument was this: Kubernetes schedules pods, but distributed inference needs groups — gangs of pods that are placed, started, restarted, upgraded, and drained as one unit, because the model sharded across them is one machine. Until now, that group was assembled outside the API: custom operators, Ray clusters bolted onto StatefulSets, init containers waiting on leader DNS, and half a page of YAML origami per model. LWS v1.0 makes the group itself the API object — and its parent project now hosts a second API, DisaggregatedSet, that does the same for prefill/decode-disaggregated serving.

The timing matters. The open-weight class has moved to models that physically cannot run on one node: DeepSeek-R1-class MoE models consume 500 GB+ of memory for weights alone, as the llm-d wide-EP documentation puts it — and the platform default for serving them is no longer “one Deployment, hope for the best.” This guide is the mechanics-first companion to our earlier pieces on serving topologies and the inference gateway pattern: those covered what topologies make sense and how requests route to them. This one covers the layer underneath — the Kubernetes API that stamps those topologies onto a cluster, verified against the LWS v1.0.0 release, the api/leaderworkerset/v1 and api/disaggregatedset/v1 Go types at main, and the llm-d v1.0.0 tree. No benchmark claims here — no GPU cluster was harmed in the making of this article; every number is computed from public specifications and labeled as such.

The Unit Mismatch: Kubernetes Speaks Pods, Inference Speaks Groups

Start with the failure mode that motivates the whole API. Multi-host tensor parallelism wires one model across N nodes with NCCL all-reduce collectives over the interconnect. Every GPU in the group participates in every layer. The group has exactly three useful states: all pods running, no pods running, and broken. There is no meaningful “partially deployed” state — 15 of 16 GPUs running is not 94% of a serving capacity, it is zero serving capacity plus 15 GPUs of stranded electricity and a hung collective.

Stock Kubernetes has no object with that shape:

• Deployment treats replicas as independent and interchangeable. A pod that dies is replaced individually. For an inference group, individually replacing rank 3 means the other 15 ranks sit in a collective timeout, then a model reload, then another collective timeout — cascading restarts at their worst.
• StatefulSet gives stable identity and ordered ops, but one StatefulSet is still one homogeneous chain of pods. A leader-plus-workers structure with different templates for each role does not fit, and ordered pod management scales painfully at group sizes that matter.
• Job gets gang semantics through batch schedulers (Kueue, Volcano), but a Job ends. Serving does not.
• Ray-on-K8s operators work — at the cost of a second scheduler, a second resource model, and a second set of failure modes inside your cluster.

So teams built the group by hand: a StatefulSet for workers with hostname-derived ranks, an init container that polls the leader’s DNS until it resolves, a headless Service with exactly the right selector, a liveness probe tuned to tolerate collective waits, and a custom controller or CronJob to restart the whole stack when one pod wedged. Every shop had its own variant. None of them handled rolling updates at the group level, because none of them had a revision that covered the group. LWS is that revision-bearing object.

LeaderWorkerSet Mechanics: The Group as the API Object

A LeaderWorkerSet (short name lws, group leaderworkerset.x-k8s.io/v1) creates replicas copies of a pod group. Each group is one leader pod plus size - 1 worker pods. The controller materializes the group with plain Kubernetes objects — no new pod-like primitive — which is the design decision that makes the whole thing debuggable with the tools you already have:

LeaderWorkerSet "deepseek-r1" (replicas=2, size=2)
│
├── StatefulSet deepseek-r1                (leaders; one pod per group, ordinal 0..N-1)
│     ├── pod deepseek-r1-0        leaderworkerset.sigs.k8s.io/group-index=0
│     │                           leaderworkerset.sigs.k8s.io/worker-index=0
│     │                           leaderworkerset.sigs.k8s.io/role=leader
│     └── pod deepseek-r1-1        group-index=1, worker-index=0, role=leader
│
├── StatefulSet deepseek-r1-0              (workers of group 0; named for its leader pod)
│     └── pod deepseek-r1-0-1     group-index=0, worker-index=1, role=worker
│                 (annotation leaderworkerset.sigs.k8s.io/leader-name=deepseek-r1-0)
│
├── StatefulSet deepseek-r1-1              (workers of group 1)
│     └── pod deepseek-r1-1-1
│
├── Headless Service deepseek-r1           (Shared subdomain policy)
│     (pod DNS: deepseek-r1-0.deepseek-r1, deepseek-r1-0-1.deepseek-r1, ...)
│
├── ControllerRevisions                   (group-level template history -> rolling update)
│
└── Workload + PodGroup per replica        (only if spec.scheduling is set; alpha)

The naming and identity rules come straight from the API source. The leader pod is named <lws>-<groupIndex> and is the name of the group; workers are <lws>-<groupIndex>-<workerIndex> with the leader always taking worker index 0. Every pod carries the labels leaderworkerset.sigs.k8s.io/name, group-index, worker-index, and role, plus a group-key hash that uniquely identifies the group instance. The admission logic injects three environment variables into every container — LWS_LEADER_ADDRESS (the leader’s DNS address), LWS_GROUP_SIZE, and LWS_WORKER_INDEX — which is the entire integration contract a distributed runtime needs. vLLM’s multi-node startup consumes exactly these three; no service discovery, no configmap, no sidecar.

Startup order and failure handling are where LWS stops being a convenience and becomes a semantics upgrade:

• spec.startupPolicy — LeaderCreated (default) creates the worker StatefulSet immediately after the leader pod object exists; LeaderReady waits for the leader to pass readiness first. For model servers that need the leader’s NCCL store up before workers dial in, LeaderReady removes a class of startup races at the cost of slower cold start.
• spec.leaderWorkerTemplate.restartPolicy — the semantics of a pod restart, in four values. This is the API’s most opinionated knob, and its default changed on the road to v1.0: the field now defaults to RecreateGroupOnPodRestart, not the old StatefulSet-like Default (which is deprecated and aliased by None).
• spec.podTerminationPolicy (new in v1.0.0) — Sequential (default) keeps workers waiting for the leader to be fully deleted before terminating; Parallel terminates the whole group concurrently during rebuilds, rollouts, scale-down, and deletion. Immutable after creation.
• spec.leaderWorkerTemplate.maxGroupRestarts + restartBackoff (v1.0.0) — bounded group recovery, detailed below, because it is the feature that makes the all-or-nothing restart survivable in production.

The restart policies deserve the full table, because picking the wrong one is the difference between a self-healing fleet and a fleet that re-pays model-load costs every time a single GPU hiccups:

Now the bounded-recovery machinery, because an unbounded RecreateGroupOnPodRestart loop is a production incident waiting to happen — one wedged node produces an infinite teardown-and-rebuild cycle that re-pays weight download, model load, and NCCL initialization on every iteration, forever. v1.0.0 adds maxGroupRestarts to cap that loop:

spec.leaderWorkerTemplate:
  restartPolicy: RecreateGroupOnPodRestart
  maxGroupRestarts: 3            # opt-in; unset (nil) = unlimited recreation
  restartBackoff:                # opt-in; exponential, defaults shown
    baseSeconds: 10              # a group must have existed >= min(base*2^count, cap)
    capSeconds: 300              # before it can be recreated; measured from
                                 # leader pod CREATION time, not failure time -
                                 # a group that ran longer than the delay
                                 # recovers immediately on failure

Budget exhausted =>
  1. controller terminates the group
  2. terminal Pod API objects are RETAINED (K8s >= 1.27 required) and carry the
     finalizer leaderworkerset.sigs.k8s.io/group-restart-budget-cleanup
     so cleanup finalizers can still land on them
  3. automatic recreation STOPS
  4. LWS status: Degraded=True, reason ReplicaRestartBudgetExceeded
     (annotations on the LWS: group-restart-counts, group-restart-charged-leaders;
      on the leader pod: group-restart-budget-exhausted)

Manual recovery: annotate the retained leader pod with
  leaderworkerset.sigs.k8s.io/recover=true
=> budget for that revision/group resets, recovery resumes

Three operational details in that block are easy to miss and expensive to learn from incidents. First, the restart budget is charged per revision per group and tracked in annotations on the LWS object itself (group-restart-counts keyed <revision>/<groupIndex>), so an operator can audit how close every group is to its cap with a single kubectl annotate --list. Second, under Hash group identity (below), admitting a replacement leader transfers the dead group’s restart count to the new group key — the budget survives group replacement, which is the only sane behavior for an autoscaled fleet. Third, changing maxGroupRestarts on an already-exhausted group does not resume it; the recover=true annotation is the only resume path. That is deliberate: budget exhaustion is treated as a signal for a human, not a knob for a reconciler to keep turning.

Rollouts operate at group granularity too. spec.rolloutStrategy is RollingUpdate-only for now, with a rollingUpdateConfiguration of maxUnavailable: 1, maxSurge: 0, partition: 0 by default — groups are upgraded one whole group at a time, and the later group does not start until the earlier one is ready. The partition field supports ordinal-partitioned canaries (update groups partition..N-1 first), and the LWS docs note the pattern is explicitly useful for xPyD-style deployments where you stage prefill before decode. The revision hash lives in leaderworkerset.sigs.k8s.io/template-revision-hash on every managed object, and the Available, Progressing, UpdateInProgress, and Degraded conditions roll up group state. HPA integration comes from a scale subresource wired to .spec.replicas/.status.replicas with status.hpaPodSelector selecting leader pods — which has a consequence we will dissect in the autoscaling section: whatever metric you autoscale on must be exposed by (or aggregated onto) the leader pod.

The DeepSeek-R1 Pattern: Multi-Host Tensor Parallelism Without Ray

Run the memory arithmetic that forces the multi-host decision, for the model the dispatch queue named and the llm-d docs call out by name. DeepSeek-R1 is a 671B-parameter MoE with Multi-head Latent Attention; its FP8 weights are 671 GB (1 byte per parameter), and its MLA KV cache runs ~576 bytes per token per layer across 61 layers — 34.3 KiB per token, all layers in. The numbers below are computed, not benchmarked (Decimal script, GB = 10⁹, policy: 80% of VRAM usable, 70% of post-weights spare reserved for KV):

Weights, FP8:                    671 GB
8xH100 node, total VRAM:         640 GB   -> weights DO NOT FIT on one node
16xH100 (2 nodes), total VRAM: 1,280 GB
  spare after weights:           609 GB
  KV budget (70% of spare):      426 GB
  KV token capacity @ 34.3 KiB/token (MLA, 61 layers):
                                 ~12.1M tokens of KV across the replica

Interpretation: one 16-GPU replica holds ~12M tokens of KV - i.e. ~370
concurrent 32k-context requests of pure KV headroom, before activation
memory and NCCL buffers eat their share. On an 8-GPU node the same model
does not merely serve badly - it cannot be instantiated at all.

That is why the “DeepSeek-R1 pattern” for LWS is the shape every platform team converges on: size: 2 (one leader pod + one worker pod), 8 GPUs per pod, tensor parallel 8 within each node, pipeline parallel 2 across nodes — the same topology the upstream LWS example ships for Llama-3.1-405B, retargeted. And the key v1.0-era detail: no Ray. The upstream example’s header comment says it outright — “basic multi-node inference with vLLM (GPU), no Ray” — because LWS’s injected environment replaces the cluster-bootstrapping layer Ray used to provide. Here is the full manifest, adapted from that example with the model and pinned image swapped to what we verified current at time of writing (vLLM stable v0.32.0, October 18; the upstream example pins v0.28.0 — bump deliberately, not accidentally):

apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
  name: deepseek-r1
spec:
  replicas: 2                      # two independent 16-GPU serving groups
  rolloutStrategy:
    type: RollingUpdate
    rollingUpdateConfiguration:
      maxUnavailable: 1            # one whole group down at a time, max
      maxSurge: 0                  # no spare capacity needed for rollouts
  leaderWorkerTemplate:
    size: 2                        # 1 leader + 1 worker pod = 2 nodes/group
    restartPolicy: RecreateGroupOnPodRestart   # one dead rank => rebuild group
    maxGroupRestarts: 3            # stop the rebuild loop; page a human
    restartBackoff:
      baseSeconds: 30
      capSeconds: 600
    leaderTemplate:
      metadata:
        labels:
          role: leader             # the Service below selects exactly this
      spec:
        containers:
          - name: vllm-leader
            image: vllm/vllm-openai:v0.32.0
            env:
              - name: HF_TOKEN
                valueFrom:
                  secretKeyRef: { name: llm-d-hf-token, key: HF_TOKEN }
            command:
              - vllm
              - serve
              - deepseek-ai/DeepSeek-R1
              - --port
              - "8080"
              - --tensor-parallel-size
              - "8"               # TP within the node (NVLink domain)
              - --pipeline-parallel-size
              - "2"               # PP across the two nodes (RDMA domain)
              - --nnodes
              - $(LWS_GROUP_SIZE)  # injected by LWS: 2
              - --node-rank
              - $(LWS_WORKER_INDEX) # 0 on the leader
              - --master-addr
              - $(LWS_LEADER_ADDRESS) # headless-service DNS of the leader pod
            resources:
              limits:
                nvidia.com/gpu: "8"
                memory: 1124Gi
                ephemeral-storage: 800Gi   # weight-download scratch
              requests:
                ephemeral-storage: 800Gi
                cpu: 125
            ports:
              - containerPort: 8080
            readinessProbe:
              tcpSocket: { port: 8080 }
              initialDelaySeconds: 15
              periodSeconds: 10
            volumeMounts:
              - { mountPath: /dev/shm, name: dshm }   # NCCL/transport shm
        volumes:
          - name: dshm
            emptyDir: { medium: Memory, sizeLimit: 15Gi }
    workerTemplate:
      spec:
        containers:
          - name: vllm-worker
            image: vllm/vllm-openai:v0.32.0
            env:
              - name: HF_TOKEN
                valueFrom:
                  secretKeyRef: { name: llm-d-hf-token, key: HF_TOKEN }
            command:
              - vllm
              - serve
              - deepseek-ai/DeepSeek-R1
              - --port
              - "8080"
              - --tensor-parallel-size
              - "8"
              - --pipeline-parallel-size
              - "2"
              - --nnodes
              - $(LWS_GROUP_SIZE)
              - --node-rank
              - $(LWS_WORKER_INDEX)   # 1 on the worker
              - --master-addr
              - $(LWS_LEADER_ADDRESS)
              - --headless           # worker: join, don't serve the API
            resources:
              limits:
                nvidia.com/gpu: "8"
                memory: 1124Gi
                ephemeral-storage: 800Gi
              requests:
                ephemeral-storage: 800Gi
                cpu: 125
            volumeMounts:
              - { mountPath: /dev/shm, name: dshm }
        volumes:
          - name: dshm
            emptyDir: { medium: Memory, sizeLimit: 15Gi }
---
apiVersion: v1
kind: Service
metadata:
  name: deepseek-r1-leader
spec:
  ports:
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080
  selector:
    leaderworkerset.sigs.k8s.io/name: deepseek-r1
    role: leader            # only leaders run the OpenAI API server
  type: ClusterIP

Walk the wiring, because each line encodes a decision:

• --nnodes $(LWS_GROUP_SIZE) — the group size is not hardcoded, so resizing the group later (more nodes per replica for a bigger model, or a wide-EP layout) is a template change, not a script rewrite.
• --node-rank $(LWS_WORKER_INDEX) / --master-addr $(LWS_LEADER_ADDRESS) — rank assignment comes from the API, derived from the injected labels. The leader is rank 0 and the headless-service DNS name it is reachable at is deterministic (deepseek-r1-0.deepseek-r1 in Shared subdomain mode).
• The Service selects role: leader — the single most common wiring mistake in hand-rolled multi-host serving is a selector that includes workers, which then round-robins API traffic to pods that do not serve the API. LWS’s role label exists precisely to make this selector trivial.
• ephemeral-storage: 800Gi — weights land on local disk before load; the upstream example sizes this for exactly that, and a node group without the scratch space will crash-loop on ErrImagePull’s quieter sibling: full disk.
• /dev/shm as Memory emptyDir — NCCL and the vLLM transports need shared memory far beyond the default 64MB; this is the same requirement as single-node serving, now repeated per pod.
• TP8/PP2, not TP16 — tensor parallel stays inside the NVLink domain where all-reduce is cheap; pipeline parallel (an order of magnitude less collective traffic) crosses the network boundary. This is the standard placement of parallelism domains onto hardware domains, and LWS’s exclusive placement + subgroups (below) are the API knobs that enforce it.

For the MoE-shaped variant of the same pattern — which the llm-d docs recommend specifically for MLA models like R1 — see the subgroups section: data-parallel attention plus expert-parallel MLPs avoids replicating the KV cache across TP ranks, which is the hidden cost of the plain-TP layout above.

Scheduling the Gang: Exclusive Placement, Kueue TAS, and Native PodGroups

A 16-GPU group is only as fast as its slowest interconnect link, and only as schedulable as its hardest-to-place member. LWS attacks placement from three directions — one stable, one via Kueue, one brand-new and alpha.

Exclusive placement (stable). Annotate the LWS (or set the new spec.placementPolicy field, which replaces the annotations and rejects conflicting values) with a topology domain, and LWS injects pod affinity/anti-affinity so that each group’s pods land inside one domain and no other group’s pods share it — a 1:1 group-to-domain mapping:

# Legacy annotation form (still used by the upstream examples):
metadata:
  annotations:
    leaderworkerset.sigs.k8s.io/exclusive-topology: cloud.google.com/gke-nodepool

# v1 field form - changing the effective placement creates a new revision:
spec:
  placementPolicy:
    type: ExclusiveTopology
    topology: topology.kubernetes.io/zone   # any node-label key: zone, rack,
                                             # block, or nvidia.com/gpu.clique
  # subGroup placement is set alongside spec.leaderWorkerTemplate.subGroupPolicy

Kueue integration (stable, and quota-aware). For capacity management, LWS workloads flow through Kueue with two annotations on the leader and worker templates that bind the whole group into one admission unit — the upstream vllm-kueue.yaml example wires it exactly like this:

apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
  name: vllm
  labels:
    kueue.x-k8s.io/queue-name: default      # LocalQueue for quota admission
spec:
  replicas: 1
  leaderWorkerTemplate:
    size: 2
    restartPolicy: RecreateGroupAfterStart   # see the restart table: a queued,
                                             # partially-placed gang must not
                                             # be churned by the restart policy
    leaderTemplate:
      metadata:
        annotations:
          kueue.x-k8s.io/podset-group-name: "vllm-multi-host"
          kueue.x-k8s.io/podset-required-topology: "cloud.google.com/gce-topology-block"
      spec: ...
    workerTemplate:
      metadata:
        annotations:                          # BOTH templates, same values
          kueue.x-k8s.io/podset-group-name: "vllm-multi-host"
          kueue.x-k8s.io/podset-required-topology: "cloud.google.com/gce-topology-block"
      spec: ...

The podset-required-topology value makes Kueue’s Topology-Aware Scheduling place the whole group inside one topology block, all-or-nothing. Two hard operational edges to plan around, both verified in our Kueue 0.20 upgrade guide: while a LWS is Kueue-managed, leaderWorkerTemplate.size is immutable (the LWSImmutableGroupSize beta gate fixed a quota bypass where group size could be raised past reserved quota) — resharding a model across more nodes means delete-and-recreate, not in-place edit; and spec.replicas (the group count) stays mutable, so horizontal scale-out remains live.

Native gang scheduling (alpha, and honest about it). v1.0.0 ships the first native integration with the Kubernetes Workload-Aware Scheduling APIs: LWS owns the Workload and runtime PodGroup objects, sets each member pod’s spec.schedulingGroup, and lets the kube-scheduler do all-or-nothing admission — no second scheduler, no Kueue required. It is opt-in with spec.scheduling, and the empty block is the shortest useful opt-in:

# Replica level (the default): one PodGroup per replica, gang.minCount = size
spec:
  scheduling: {}                  # leader + its workers admitted together;
                                 # different replicas remain independent

# Role level: separate PodGroups for leader and workers within each replica
spec:
  scheduling:
    replica:
      leader: {}                  # setting either role selects this level
      worker: {}

# Whole-LWS level: ONE PodGroup for the entire LeaderWorkerSet
spec:
  scheduling:
    schedulingPolicy:
      gang: {}                    # replicas * size pods across all replicas;
                                 # during a rollout, gang.minCount counts
                                 # not-yet-replaced replicas at their OLD size

The alpha restrictions are worth more than the feature summary, because they define whether you can use it at all: spec.scheduling must be set at creation and the level is immutable; startupPolicy: LeaderReady is rejected (workers are not created until the leader is ready, so the gang could never form — a validation rule that saves you a deadlock); it cannot be combined with the exclusive-topology annotation in this alpha; leader and workers must share a priorityClassName; and schedulingPolicy/schedulingConstraints/disruptionMode/resourceClaims are immutable after creation (only gang.minCount follows replicas and size). One subtlety from the docs saves a debugging session during LWS upgrades: while the webhook and controller briefly run different LWS versions mid-rollout, a leader admitted by the new webhook can land in a different PodGroup than the workers the old controller creates — LWS detects this and recreates the replica (deleting its leader pod) as long as no worker is scheduled yet, recording a RecreateGroup event. If you see that event storm during a controller upgrade, it is the system healing itself, not a bug in your workload.

Subgroups: Hierarchical Collectives for Expert-Parallel Fleets

Between “every pod in the group” and “every pod on its own” there is a third topology that pure leader/worker APIs miss: sub-groups, where a large group is partitioned into fixed-size chunks with their own identity, their own placement domains, and their own failure isolation. LWS’s subGroupPolicy exists because real multi-node serving racks are hierarchical — NVLink inside a node, NVSwitch or IB blocks between them — and collectives want to follow that hierarchy:

spec:
  leaderWorkerTemplate:
    size: 16                        # e.g. 16 nodes = one wide-EP group
    subGroupPolicy:
      subGroupSize: 2               # 8 sub-groups of 2 pods each; IMMUTABLE,
                                   # must divide size (or size-1 with the
                                   # leader as the extra pod in group 0)
      subGroupPolicyType: LeaderWorker   # LeaderWorker | LeaderExcluded
# Pod labels injected:
#   leaderworkerset.sigs.k8s.io/subgroup-index, subgroup-key (hash),
#   annotation subgroup-size
# Placement per sub-group:
#   spec.placementPolicy.subGroup:
#     type: ExclusiveTopology
#     topology: topology.kubernetes.io/zone   # each sub-group its own zone,
#                                             # other sub-groups kept out

The two policy types differ in exactly one way: LeaderWorker includes the leader pod in the first sub-group; LeaderExcluded (only valid when size - 1 divides evenly by subGroupSize) keeps the leader out of all sub-groups — the shape you want when the leader is a coordinator that must not consume a collective slot. The llm-d docs name the canonical use: hierarchical collective communication and targeted fault isolation across multi-node groups.

This is also where the DeepSeek-R1 pattern gets its MoE-native upgrade. The wide expert-parallelism architecture serves exactly the “very large MoE models like DeepSeek-R1” case: attention layers run data-parallel (one KV copy per token — the docs are blunt that MLA’s KV cache “replicates when sharded with tensor parallelism”, which is the hidden tax of the plain-TP layout in the YAML above), while the expert MLP layers run expert-parallel so every accelerator holds a slice of the experts. The per-token routing is sparse — DeepSeek selects 8 of 256 experts — so the dispatch/combine collectives (all-to-all, on DeepEP-class kernels) move far less data than TP all-reduces and tolerate network-rack-class interconnects. vLLM’s flags for the layout are --data-parallel-size plus --enable-expert-parallel, per the llm-d vLLM integration page. The deployment consequence: one wide-EP group spans several nodes, so it is one LWS group; each DP rank can expose its own API-server port (8000–8007 for eight ranks per pod in the docs’ example), and the InferencePool lists every rank port so the router scores individual podIP:port endpoints rather than whole pods — prefix-cache affinity lands on the rank that holds the prefix, not on whichever rank the pod’s internal balancer picks.

Group Identity and Autoscaling: Why Serving Fleets Want Hash Mode

Everything above assumed groupIdentity: Ordinal (the default): leaders managed by a StatefulSet, groups named 0..N-1, scale-down always removing the highest ordinal. That is exactly the wrong behavior for an autoscaled serving fleet. HPA scales a fleet down when utilization drops; Ordinal scale-down removes the highest-numbered group regardless of health. If group 3 is wedged and group 7 is fine, scaling from 8 to 7 replicas deletes healthy group 7 and keeps the wedged group — you paid for a rebuild to shed capacity you did not want to lose. Hash identity fixes the ordering problem by changing the manager:

The mechanics behind the “health-aware scale-down” row are the cleverest part of the design: each Hash-mode leader carries a leaderworkerset.sigs.k8s.io/group-ready readiness gate that the LWS pod controller clears only once the group’s worker StatefulSet is ready — so a leader is Ready only when its whole group is. The leader Deployment’s ReplicaSet then ranks scale-down victims using standard pod-health semantics, which now encode group health. And PostTermination replacement admission counts capacity honestly: gated leaders are admitted oldest-first up to G - T (gate count minus tearing-down groups), keyed off the old group’s pods being fully deleted — not the leader object, which the ReplicaSet can delete in the background seconds before its workers release capacity. The docs spell out the failure mode this avoids: counting the leader object alone lets a replacement land where its predecessor’s GPUs are still occupied. Two serving-specific consequences: treat group identity as ephemeral and never cache a leader’s DNS address across a replacement; and note that a waiting replacement appears as a SchedulingGated pod and counts in status.replicas — capacity dashboards must not read that as serving capacity.

DisaggregatedSet: Prefill/Decode as a Kubernetes API

LWS solved “one group.” Disaggregated serving — the DeepSeek production pattern where compute-bound prefill and bandwidth-bound decode run on separate, differently-shaped pools — needs two coupled groups that scale at different ratios, roll out in lockstep, and exchange KV cache. Two independent LWS objects get you two independent revision histories: a decode pool on revision A paired with a prefill pool on revision B works right up until the KV-cache transfer format between them changes, and then it fails per request. DisaggregatedSet (disaggregatedset.x-k8s.io/v1, short names dset/disaggset, KEP-766) is the parent API that fixes the coupling by construction:

DisaggregatedSet "r1-disagg" (slices=2, revision B)
│
├── slice 0
│   ├── LeaderWorkerSet r1-disagg-0-B-prefill   <- child LWS, one per role
│   │     └── prefill pods (llm-d.ai/role=prefill)
│   ├── LeaderWorkerSet r1-disagg-0-B-decode
│   │     └── decode pods  (llm-d.ai/role=decode, routing sidecar injected)
│   └── [opt-in] ComputeDomain for (slice 0, revision B)     (DRA, alpha)
│
├── slice 1
│   └── ... same topology, independent failure/restart domain
│
└── per-role status: desiredReplicas, readyReplicas, updatedReplicas,
    minAvailableReplicas; conditions Available, Progressing,
    MinimumReplicasAvailable, SubRolesAssigned

Child LWS naming: <ds>-<slice>-<revision>-<role>
Pod env injected:  DISAGGREGATEDSET_NAME, _ROLE, _SLICE, _REVISION
Labels consumed by the llm-d router: disaggregatedset.x-k8s.io/{name,role,slice,revision}

The API’s core vocabulary, from the api/disaggregatedset/v1 types: roles are the functional phases (prefill, decode — the release notes say 2–10 are supported), each with its own leaderWorkerTemplate inlined and its own replica count; slices are complete copies of the whole multi-role topology — scaling slices adds or removes full end-to-end serving units, never a lone prefill pod stranded without decode partners. Placement comes in three grades: None, ExclusiveSlice (all roles of one slice co-located in one topology domain — mandatory when KV moves between roles over NIXL RDMA — while other DisaggregatedSets may still share the domain), and ExclusiveTopology (full 1:1 domain-to-slice mapping across all DisaggregatedSets in the cluster). Role scaling is Static or External — the latter creates a DisaggregatedSetRoleScaler named <ds>-<role> that an external autoscaler (KEDA is the documented example) drives, ignoring the inline replicas, which is how a fleet survives a 10:1 prompt:completion burst without over-provisioning decode.

v1.0.0 made DisaggregatedSet genuinely production-grade, and two of its changes are upgrade hazards worth quoting precisely from the release notes. First, the rollout default flipped: DisaggregatedSet roles without a rollingUpdateConfiguration now roll out with maxSurge: 0, maxUnavailable: 1 — like LeaderWorkerSet — instead of the old maxSurge: 1, maxUnavailable: 0. Teams that relied on bringing the new revision up before draining the old must now set maxSurge: 1/maxUnavailable: 0 explicitly or their rollouts will trade capacity for safety after the upgrade. Second, a stale-object trap: v1.0.0 removes support for slice-less LeaderWorkerSets generated for DisaggregatedSets created under v0.9.x — if such a set has not rolled out under v0.10.x or v0.11.x first, install one of those, complete one template rollout, then upgrade. On the positive side of the ledger, v1.0.0 adds sub-roles (independently scalable partitions of one role with their own rollout budgets, routing labels, and Ordinal/Hash group identity — KEP-960), fractional rollout windows that advance roles proportionally to their replica counts so new batches start before earlier ones are fully Ready (PR #907), and opt-in scaling during rollouts via spec.scalingPolicy.duringRollout: AdvanceRollout (KEP-1022).

Here is the smallest honest P/D manifest — note the example’s own caveat in the upstream repo: the NIXL connector and prefill/decode wiring are deployment-specific; this is a starting point, not turnkey:

apiVersion: disaggregatedset.x-k8s.io/v1
kind: DisaggregatedSet
metadata:
  name: vllm-disagg
spec:
  slices: 1
  placementPolicy:
    type: ExclusiveSlice            # KV crosses roles over RDMA: co-locate them
    topology: topology.kubernetes.io/zone
  roles:
    - name: prefill
      spec:
        leaderWorkerTemplate:
          size: 1
          workerTemplate:
            metadata:
              labels:
                role: prefill
                llm-d.ai/role: prefill        # router-role detection
                llm-d.ai/engine-type: vllm
                llm-d.ai/inferenceServing: "true"
            spec:
              containers:
                - name: vllm
                  image: vllm/vllm-openai:v0.32.0
                  command:
                    - sh
                    - -c
                    - "python3 -m vllm.entrypoints.openai.api_server --port 8080
                       --model deepseek-ai/DeepSeek-R1
                       --kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_producer\"}'"
                  resources:
                    limits: { nvidia.com/gpu: "1" }
                  ports:
                    - containerPort: 8080
                  readinessProbe:
                    tcpSocket: { port: 8080 }
    - name: decode
      spec:
        leaderWorkerTemplate:
          size: 1
          workerTemplate:
            metadata:
              labels:
                role: decode
                llm-d.ai/role: decode
                llm-d.ai/engine-type: vllm
                llm-d.ai/inferenceServing: "true"
            spec:
              containers:
                - name: vllm
                  image: vllm/vllm-openai:v0.32.0
                  command:
                    - sh
                    - -c
                    - "python3 -m vllm.entrypoints.openai.api_server --port 8080
                       --model deepseek-ai/DeepSeek-R1
                       --kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_consumer\"}'"
                  resources:
                    limits: { nvidia.com/gpu: "1" }
                  ports:
                    - containerPort: 8080
                  readinessProbe:
                    tcpSocket: { port: 8080 }

The request path that this topology implies — which the llm-d stack drives through the inference gateway (Gateway API Inference Extension) — is where the topology pays for itself. The sequence, verified against the llm-d disaggregated-serving design doc:

Client
  │
  ▼
Proxy (Envoy, standalone mode or GAIE InferenceGateway)
  │  forwards request metadata to EPP
  ▼
EPP: disagg-profile-handler
  │  1. decode-profile: filter/score/pick  -> D endpoint
  │  2. decider: how much of this prompt is UNCACHED on D?
  │       small uncached suffix ─────────────► return D only (skip disagg;
  │                                            decode prefills locally)
  │       large uncached suffix:
  │  3. prefill-profile: filter/score/pick ─► return P + D endpoints
  ▼
Proxy ──► routing sidecar (in the DECODE pod), header:
          x-llm-d-prefiller-host-port = <prefill address(es)>
  │
  ▼ (vLLM nixlv2 two-phase protocol)
sidecar ──► P: request { max_tokens=1, kv_transfer_params=remote-decode-metadata }
P        ──► P:  run prefill, hold KV blocks in GPU memory
P        ──► sidecar: response carrying KVTransferParams
sidecar ──► D (local): request + KVTransferParams (+ do_remote_prefill)
D        ──► P: PULL KV blocks over NIXL RDMA (GPUDirect; GPU-registered memory,
  │            no staging buffers, no CPU copy)
P        ──► P:  release KV cache
D        ──► D:  run decode loop ──► sidecar ──► Proxy ──► Client

Three details in that flow carry disproportionate production weight. The decider makes disaggregation conditional per request — a prompt whose prefix is already cached on the chosen decode pod skips the prefill pool entirely, which is the mechanism that makes P/D and prefix-cache-aware routing compose instead of fight. The lazy handshake (decode and prefill workers establish NIXL peer connections on first pairing via a background ZMQ metadata exchange, with no central coordinator) means scale events do not need a connection-reconfiguration step — but also that the first request after any pairing pays the setup. And the fallback policy is a real decision: with kv_load_failure_policy=recompute, a decode pod that fails a remote KV pull (say, its prefill partner was scale-downed mid-transfer) recomputes prefill locally instead of failing the request — trading latency for availability, per request.

Rollouts are the coupling problem again, and the answer is a router plugin. During a DisaggregatedSet rollout, old and new revisions run side by side; prefill and decode from different revisions may use incompatible KV-cache formats or transfer protocols, so requests must not pair across revisions. The LWS docs’ llm-d integration page documents the disaggregatedset-rollout-screener EPP plugin: a revision receives traffic only when it has at least one Ready pod in every required role (no traffic to a revision whose prefill pods are still loading weights, however ready its decode pods are), and in max-role mode the screener computes each revision’s traffic share from the role with the most Ready pods across eligible revisions — six ready decode pods in A and four in B gives B a 40% share, counted in pods, not GPUs. The share hand-off between one router (simplest: both profiles run after one revision choice) or per-role routers (a coordinator copies the x-llm-d-disagg-revision header from prefill’s response into the decode request) is plumbing; the design point is that the API gives the router revision labels (disaggregatedset.x-k8s.io/revision) that make the pinning possible, and LWS’s own rollout pacing (groups not updated until their predecessors are ready) keeps the share math monotonic.

The Accelerator Layer: DRA ComputeDomains for NVL72-Generation Racks

Multi-host TP assumes the interconnect is Ethernet-or-better and lets NCCL sort out the topology. The GB200/GB300 NVL72 generation inverts that assumption: NVLink now spans racks (Multi-Node NVLink), and workloads that share GPU memory across pods — cuda_ipc, NCCL/NVSHMEM collectives, NIXL NVLink KV-cache transfer — must join a shared IMEX domain, which on Kubernetes is an NVIDIA DRA ComputeDomain object served by the k8s-dra-driver-gpu. LWS v1.0.0 automates that wiring (KEP-1259, AutoComputeDomain feature gate, alpha), which is the difference between a rack-native serving stack and a Kubernetes deployment that happens to occupy an NVL72:

# Controller install with the gate on (Helm):
#   helm upgrade --install lws oci://registry.k8s.io/lws/charts/lws \
#     --namespace lws-system --create-namespace \
#     --set featureGates.AutoComputeDomain=true

apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
  name: deepseek-r1
  annotations:
    leaderworkerset.sigs.k8s.io/auto-compute-domain: "enabled"
    leaderworkerset.sigs.k8s.io/exclusive-topology: nvidia.com/gpu.clique
spec:
  replicas: 2
  leaderWorkerTemplate:
    size: 4
    workerTemplate:
      spec:
        containers:
          - name: vllm
            image: nvcr.io/nvidia/vllm:latest
            resources:
              limits: { nvidia.com/gpu: "4" }
          - name: routing-proxy
            image: envoyproxy/envoy:v1.31-latest   # CPU-only sidecar: claim
                                                   # injection skips it
# LWS creates one ComputeDomain per replica; a restarted/replaced group gets
# a NEW domain. The lws-compute-domain-channel ResourceClaim is injected only
# into containers requesting nvidia.com/gpu > 0.
# DisaggregatedSet variant: disaggregatedset.x-k8s.io/auto-compute-domain:
# "enabled" -> one domain per (slice, revision), shared across roles.

The operational constraints in the docs are the load-bearing part. An IMEX channel connects GPUs only within the same physical NVLink clique (nvidia.com/gpu.clique), and the driver enforces at most one active ComputeDomain per node in driverManaged mode even when the node has free GPUs — so a pod that cannot join its group’s domain is a pod that cannot run, full node or not. Pair the opt-in with placement constraints so a domain’s members co-locate on one clique and separate domains land on distinct nodes; rolling updates with maxSurge > 0 need spare clique nodes, or you set maxSurge: 0 and drain-before-reuse when every clique node is occupied. And the disable path is genuinely hazardous: managed ComputeDomains deliberately carry no owner reference (ownership is tracked in a compute-domain-owner-uid label so garbage collection cannot delete a domain while pods drain), so if you turn the feature gate off and delete the workload, its domains are not garbage collected — they sit in Terminating held by the compute-domain-protection finalizer until the gate is re-enabled or you remove the finalizer by hand after the pods are gone. The docs’ runbook order is the only safe one: remove the annotations, wait for the managed domains to be deleted, then touch the gate.

Production Realities

Everything above is the brochure. This is the part that decides whether the architecture survives contact with your on-call rotation.

• RDMA is the entry fee for disaggregation; NIXL’s TCP fallback is a dev toy. The llm-d docs do not soften this: disaggregated serving requires high-performance RDMA interconnects for KV transfer, and without RDMA NIXL falls back to TCP, which “is not efficient and should only be used for testing and development.” A P/D split over plain Ethernet is a latency regression dressed up as an architecture diagram. Budget the fabric before you budget the GPUs.
• The NixlConnector has documented sharp edges. The same docs page warns of known issues around TP-ratio direction and stale agent caching after prefill pod restarts — i.e., the exact failure mode a RecreateGroupOnPodRestart fleet produces routinely. Heterogeneous-parallelism P/D (different TP per role) plus group restarts is the configuration where those bugs live; canary it, and keep the decode-side recompute fallback enabled until you have evidence it never fires.
• Group restarts are expensive; budget them like deploy events, not pod events. A 16-GPU group rebuild re-pays weight download (unless cached — the volumeClaimTemplates support exists precisely for per-pod persistent weight/KV scratch), model load, NCCL initialization, and KV warm-up — across all 16 GPUs. restartBackoff plus maxGroupRestarts converts the infinite version of that loop into a bounded one, but the arithmetic of your fleet’s effective capacity must assume each group restart is minutes of zero throughput. Size replicas so that maxUnavailable: 1 group rebuilds still meet SLO, and treat the Degraded condition as a page, not a log line.
• HPA scales groups, and groups are big. The scale subresource changes spec.replicas — the quantum of horizontal scale is one entire group. For the R1 pattern that is 16 H100s per step, and for a PD fleet the two roles scale on different quanta (a prefill “step” and a decode “step” are different machines entirely). Autoscaling signal design is accordingly non-trivial: HPA reads leader-pod metrics via status.hpaPodSelector, so either the leader aggregates group metrics (the API comments suggest exactly that pattern) or you front the fleet with KEDA + External role scaling on the DisaggregatedSet. Also mind the Hash-mode caveat: a SchedulingGated replacement counts in status.replicas but serves nothing.
• Upgrade sequencing is not optional. The v1.0.0 notes carry an “urgent upgrade” section for a reason: the DisaggregatedSet rollout-default flip (maxSurge 1/0 → 0/1), the slice-less-LWS removal for v0.9.x-era sets, and the version-skew PodGroup splits during the LWS controller’s own rolling upgrade (self-healing via RecreateGroup events, but only for replicas with no scheduled workers). Read that section before the upgrade, not during the incident.
• Alpha features are load-bearing alpha. Native gang scheduling (no exclusive-topology combination, immutable scheduling level, LeaderReady rejected) and AutoComputeDomain (controller exits at startup if the gate is on but the ComputeDomain API is absent; kustomize users must add the RBAC component explicitly) both change behavior fast between minors. Pin the chart version, read the upgrade notes, and treat either gate as a dependency with an upgrade cost — because it is one.
• The observability contract is vLLM’s /metrics, and it is good. The EPP’s scheduling signals are scraped straight from vLLM’s Prometheus endpoint — queue depth from vllm:num_requests_waiting, running requests from vllm:num_requests_running, KV pressure from vllm:kv_cache_usage_perc, block geometry from vllm:cache_config_info. That means your fleet’s routing quality degrades exactly as much as your metrics pipeline does; scrape-interval drift is routing degradation.
• This stack is proven at fleet scale — by other people, with caveats. The reference data point is IBM Research’s deployment of GLM-5.2 across 544 H100s with llm-d, which llm-d’s v1.0.0 release and blog cite as its flagship case; the v1.0.0 release notes additionally name AWS and Google production numbers. What the public material does not give you is a like-for-like comparison against a well-tuned monolithic stack at your traffic shape. Run your own P/D-vs-aggregated A/B on your own ISL:OSL ratio — the llm-d docs themselves scope the win to medium-large models with long inputs, and single out 10:1 ISL:OSL-class workloads; short-prompt fleets get the complexity without the prize.

Who Should Skip This

• Teams serving models that fit one node. If the model fits in one pod’s GPUs with sane KV headroom, LWS is structural overhead: the group is one pod, size: 1, and a Deployment plus the inference gateway does everything LWS would, with less YAML. The group abstraction earns its keep the first time size > 1.
• Teams without RDMA fabric. DisaggregatedSet’s headline feature moves KV between roles over RDMA. On Ethernet-only clusters the honest architecture is aggregated serving — multi-host TP via LWS if the model needs it, P/D never.
• API-first organizations. If your inference spend is below the self-host break-even line we computed in the vLLM self-hosting review (~23–47% sustained GPU utilization depending on how much of the memory floor you actually achieve), you do not need distributed inference infrastructure at all — you need utilization, and this stack does not fix utilization.
• Teams that cannot upgrade Kubernetes. LWS tests against Kubernetes 1.34–1.37; the native gang scheduling and DRA paths additionally need the Workload-Aware Scheduling APIs and DRA served in-cluster. On a fleet pinned to older minors for compliance reasons, assume the alpha features are off the table and plan around Kueue-only scheduling.
• Anyone who needs stable ordinal identity in the serving path. Hash identity is the right default for autoscaled serving, but it is ephemeral by design — leader DNS names change on every group replacement, volumeClaimTemplates are rejected, partition canaries are rejected. If your stack caches pod DNS or reattaches PVCs to ordinal successors, that is training-workload machinery, and it belongs in Ordinal-mode LWS or a plain StatefulSet — not in the serving fleet.

The Decision Table

References & Further Reading

• kubernetes-sigs/lws — LeaderWorkerSet & DisaggregatedSet APIs (source of all API mechanics, labels, annotations, and examples cited here; v1.0.0 released 2026-10-16)
• LWS concepts documentation — startup, rollout, group identity, gang scheduling, placement, volume claim templates
• DisaggregatedSet concepts documentation — roles, slices, placement, revision gating with llm-d
• Automatic ComputeDomain management (NVIDIA MNNVL) — DRA integration, scopes, and lifecycle hazards
• llm-d — distributed inference stack co-designing these APIs; P/D disaggregation design and wide expert parallelism
• IBM Research: serving open models on 544 H100 GPUs with llm-d — the flagship fleet-scale deployment write-up
• vLLM: Parallelism and Scaling · Data Parallel Deployment · Expert Parallel Deployment
• Gateway API Inference Extension — the routing layer that pairs with LWS-managed fleets
• Platform Monkey companions: serving topologies after Beam (why multi-host is now routine), the inference gateway pattern (EPP/InferencePool mechanics), GPU scheduling for LLMs and the Kueue 0.20 upgrade guide (quota and TAS edges), and the vLLM self-hosting review (the utilization break-even)