Platform Monkey ctx

Karpenter v1.15.0 review: node repair grows up — from forced eviction to budgeted, rebootable disruption

· By
Sources

Karpenter v1.15.0 landed on 10 October 2026 — core at 05:42 UTC and the AWS provider fifty-three minutes later — and the headline is architectural rather than a feature-list: node auto-repair stopped being an eviction hammer and became a first-class voluntary disruption method. If you run GPU fleets on Karpenter with the EKS Node Monitoring Agent, this is the release that changes how your unhealthy nodes die — and, for the first time, how some of them get to live.

We read the release notes, the three design docs that motivated the change, the v1.15.0 source of both kubernetes-sigs/karpenter and aws/karpenter-provider-aws, and the official disruption documentation before writing this. Everything below cites file paths and PR numbers from those sources. What follows is the skeptical version: what actually shipped, what it costs you, and who should not flip the gate.

The pain: a circuit breaker that failed in both directions

Node auto-repair shipped in PR #1768 as an alpha, opt-in mechanism that forcefully terminates nodes a cloud provider declares unhealthy. The Making Node Repair Voluntary RFC is unusually blunt about the old design: the standalone node.health controller deleted NodeClaims directly, gated only by a single hardcoded allowedUnhealthyPercent = "20%" circuit breaker. It did not pre-spin a replacement. It did not respect PodDisruptionBudgets. It did not honor terminationGracePeriod. It gave an operator no lever to stop it mid-incident.

The RFC calls the breaker "the wrong shape" because it fails in both directions. In a detector false-positive wave — the exact scenario a large accelerator fleet lives in — the breaker trips and all repair freezes cluster-wide, including the one genuinely broken node. When the flap settles and the fraction drops back under the line, the breaker re-opens and Karpenter terminates every still-flagged node at once. A single latched threshold cannot distinguish a bad rollout (where replacement re-inherits the fault), a flaky detector, a zonal partition, and genuine hardware failure. The design doc's failure table is the best argument for the rewrite we have read in any 2026 release notes.

The community evidence is pinned in the RFC itself: issue #2811 (an over-termination incident asking for "1 at a time, once per 5m" pacing), #2042 (repair ignoring PDBs — a graduation blocker), and #2134 / #2321 (single-node NodePools where 20% rounds up to "always repair" or "never repair"). We verified all four issue states at time of writing: #2042 is closed-completed by this release; #2134 and #2321 are closed-not-planned — the "configurable breaker" ask was superseded by the budget machinery; #2811 remains open for the per-NodePool repair-policy configuration follow-up.

The fix: repair as a fourth disruption reason

From v1.15.0, repair is a voluntary disruption method — the same class as consolidation and drift — and it rides the disruption-budget machinery as a new reason. In the code (pkg/apis/v1/nodepool.go), DisruptionReason gains Unhealthy alongside Underutilized, Empty, and Drifted, so your existing disruption budgets can now pace it per NodePool:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-train
spec:
  disruption:
    budgets:
      # Repair at most one node at a time; never let it eat the
      # budget that consolidation and drift also draw from.
      - nodes: "1"
        reasons: ["Unhealthy"]
      # Everything else (Underutilized/Empty/Drifted) stays on the
      # default pacing during business hours.
      - nodes: "10%"

Three semantics from the source are worth pinning because they are not obvious from the YAML. First, budgets with reasons only apply to the listed reasons; a budget without reasons applies to all. Second, when multiple budgets are active, Karpenter uses the most restrictive value. Third — the detail that makes the whole design actually work — NotReady nodes do not count against the Unhealthy budget. The comment in pkg/controllers/disruption/helpers.go explains why: NotReady is precisely repair's trigger, so counting unhealthy nodes against the repair budget would let a health wave zero the very budget that fixes them — the freeze failure mode, resurrected. Only nodes already being deleted or actively rebooting consume repair budget. The 20% breaker still exists (repairUnhealthyThreshold = "20%" in pkg/controllers/disruption/repair.go), but it is now a self-clearing "correlated failure" tripwire — if more than 20% of a pool goes unhealthy, repair assumes the problem is systemic (bad AMI, AZ outage) rather than per-node, emits a NodeRepairBlocked warning event on the node, NodeClaim, and NodePool, and waits — it no longer latches until a human intervenes.

The full control flow, as documented and as implemented:

Node reports unhealthy condition
        |
        v
[Feature gate] NodeRepair=true?  --no--> legacy behavior off; node left alone
        |yes
        v
Match RepairPolicy: condition type + status + reason regex
(toleration must have elapsed; e.g. Ready=False waits 30m,
 AcceleratedHardwareReady=False XID-family waits 10m)
        |
        v
karpenter.sh/do-not-repair: "true" on node?  --yes--> Blocked event; skip
        |
        v
Budget check: Unhealthy budget for this NodePool > 0?
(NotReady nodes do NOT consume the Unhealthy budget;
 nodes being deleted or rebooting DO)
        |
        v
>20% of pool unhealthy?  --yes--> NodeRepairBlocked event; defer
(correlated-failure tripwire, self-clears)
        |
        v
Action from policy: ReplaceNode or RebootNode
  Replace: pre-spin replacement, verify reschedulability,
           then cordon/drain (PDB-honoring, TGP-bounded), delete
  Reboot:  fence + bounded drain, then in-place instance reboot,
           2 attempts per 24h per NodeClaim, then escalate to replace

The AWS policy table: reason regexes and XID families

The abstract RepairPolicy type grew from three fields in v1.14 (ConditionType, ConditionStatus, TolerationDuration) to eight in v1.15 — adding ReasonRegex, TerminationGracePeriod, Priority, and Action. The AWS provider's concrete set (pkg/cloudprovider/cloudprovider.go, landed in aws#9680) is where this becomes operationally real, because the Node Monitoring Agent embeds the NVIDIA XID code in the condition reason, and different XID families warrant different handling:

Condition (status=False)Reason matchTolerateDrain boundAction
Readyany (.*)30m10mReplace
Ready (Unknown)any30m10mReplace
AcceleratedHardwareReadyXID 46/48/54/62/63/95/109/110/136/140/143/155/156/158 — reboot-clearable transient GPU faults10m5mReboot* or Replace
AcceleratedHardwareReadyXID 64/74/79/119/120 — fatal / uncorrectable10m5mReplace
AcceleratedHardwareReadyfallback (no regex match)30m10mReplace
StorageReadyany30m10mReplace
NetworkingReadyany30m10mReplace
KernelReadyany30m10mReplace
ContainerRuntimeReadyany30m10mReplace

*Reboot requires the RebootForRepair AWS feature gate (alpha, off by default, landed in aws#9716); without it, the same policies replace the node on the same timing. Note the deliberate split: a transient GPU fault tolerates only 10 minutes and gets rebooted (preserving a scarce GPU instance), while an unrecognized fault waits 30 minutes before replacement — the policy set says "we don't know what this is, so give it time before we burn the instance." That is a mature policy table, not a marketing one. Note also what the docs warn explicitly: when a node matches several conditions at once, Karpenter takes the most disruptive action among them.

The per-condition drain bound matters more than it looks. TerminationGracePeriod on a policy bounds the drain even when neither NodePool nor NodeClaim set one — the RFC's words for the alternative are "the unbounded (~19-day) drain hang that a nil TerminationGracePeriod would otherwise permit." A zero value means forceful: skip the drain entirely, for conditions the kubelet cannot evict through (wedged kernel, lost heartbeat).

Reboot as a node action: the in-place path

PR #3345 adds reboot as a first-class node action, driven by a dedicated controller (pkg/controllers/nodeclaim/reboot/controller.go) that carries a committed reboot through drain → issue → observe to a terminal RebootSucceeded or RebootFailed. The lifecycle is durable in the NodeClaim's Rebooting condition, with phases as the condition reason: RebootRequested → RebootIssued → RebootSucceeded/RebootFailed. On the AWS side, aws#9676 implements CloudProvider.Reboot() via ec2:RebootInstances — a new IAM permission your controller role needs before the gate does anything.

The mechanics are careful in a way worth spelling out because they differ from what most people assume "reboot" means:

Who is this for? The design motivation is scarce or slow-to-provision capacity: a p5-class instance that took hours to obtain and would take hours to replace. For everyone else, replacement remains the default action and reboot is the opt-in exception — which is the correct polarity for an alpha.

Terminate-first: disruption for fleets with no room to grow

The quieter structural change is PR #3342 (terminate-first repair) and its sibling #3312 (terminate-first drift), both behind alpha gates (TerminateFirstRepair requires NodeRepair; TerminateFirstDrift standalone). The RFC describes the deadlock: voluntary disruption replaces a node by launching its replacement first, which requires spare capacity. Fleets pinned to on-demand capacity reservations sized to exactly N accelerators have none — so drift stalls and repair cannot fire, and "a large reserved GPU fleet can sit un-rolled for weeks."

Terminate-first is the second strategy: delete first, then let ordinary reactive provisioning launch the replacement into the slot that freed. Karpenter derives the choice automatically from configuration you already supplied — a static NodePool at its limits, or a scheduling simulation that can only place the replacement back into the same full reservation. No new API. The karpenter_voluntary_disruption_terminate_first_decisions_total counter records why the replacement could not be staged: no-reserved-capacity or static-at-limit. The blast-radius story is honest: the budget bounds the delete side, PDBs bound how much of a workload is down at once, and if the slot is not free yet the launch retries on its normal cadence — there is no guarantee the replacement lands instantly, because none is possible on a structurally full fleet.

What else shipped: spread awareness, DRA, metrics

Three more changes deserve operator attention:

Hidden costs, breaking changes, and the migration vector

The upgrade guide marks the repair change Breaking even though the gate is alpha and off by default. The full bill:

The migration vector for a typical EKS GPU fleet: upgrade CRDs and controller together, fix the two metric-label alerts, leave NodeRepair off until you have decided your per-pool Unhealthy budget, then enable the gate pool-by-pool with a tight budget (nodes: "1", reasons: ["Unhealthy"]) and watch karpenter_voluntary_disruption_decisions_total{reason="unhealthy"} plus the NodeRepairBlocked events for a week before loosening. If you run reserved-capacity fleets, evaluate TerminateFirstRepair only after NodeRepair itself has been sober for a while — it is the more aggressive primitive.

Who should skip this release's headline feature

Credits and what to watch next

The repair arc in this release is overwhelmingly @DerekFrank's work (the RFC, the voluntary-disruption integration, terminate-first repair and drift, the legacy flag, and the metric discipline), with @Sarthug99 on reboot-as-action, @Sebestien on reason-aware policy matching and candidate escalation, @ryan-mist on the disruption-queue integration, @jigisha620 on the VPA prediction groundwork, and @GaneshBannur on the repair-veto RFC. The AWS provider policy set came from @Sebestien (aws#9680) and @Sarthug99 (reboot, aws#9676). Watch issue #2811 for per-NodePool repair-policy configuration — the natural next step, and the one that makes the policy table operator-tunable rather than provider-fixed.

Our verdict: this is the most defensible 1.x minor Karpenter has shipped — a rewrite of its least-safe subsystem that closes a graduation blocker (#2042), replaces a latched breaker with pacing that cannot starve itself, and adds GPU-fleet capabilities (reason-regex policies, in-place reboot, terminate-first) that AWS's own managed node groups are only now matching. The alpha gating is conservative and the breaking surface is small and loudly documented. Upgrade for the CRD-and-metrics hygiene even if you leave every new gate off; turn the repair gate on only with a deliberate per-pool budget and a week of watching the unhealthy decision stream first.

References & Further Reading