Platform Monkey ctx

Kubernetes Cost Visibility in 2026: OpenCost, VPA InPlace, and the Right-Sizing Math That Actually Saves Money

· By
Sources

September 19, 2026. Cloudflare bought itself back more than 100TB of RAM this week without adding a single server — by shrinking hash-ring data structures and fixing struct layout in Rust. It is the most visible entry yet in a genre of engineering that platform teams know well: capacity work that shows up as a real, defensible line in the budget. The Kubernetes version of the same discipline is less glamorous and more common: your pods are requesting resources they never use, and every one of those requests is money that the scheduler counts as spent.

Here is the uncomfortable mechanism: in Kubernetes, a resource request is not a suggestion — it is a reservation. The scheduler subtracts requests from node allocatable when deciding where a pod can fit, and your cost model bills you for max(request, usage). A pod that requests 2 CPU and averages 0.4 CPU doesn’t “cost a little more than it should” — it burns capacity as if it were fully loaded, and forces the cluster to scale out to hold the fiction. Multiply by a few hundred over-provisioned workloads and you get the silent 30–50% waste that every FinOps survey keeps re-finding.

The tooling to see and fix this has quietly matured to the point where the excuses are gone. OpenCost (the CNCF incubating project Kubecost originally open-sourced before IBM bought the company in 2024) shipped its Public Pricing 1.0.0 data module in v1.121.2 (September 11) — nearly 200,000 committed price entries across AWS, Azure, GCP, IBM, Oracle, and Alibaba — and followed with v1.121.3 (September 16) adding GPU public pricing for GCP and an IBM Cloud Usage Reports integration. VPA 1.7.x can now apply recommendations without evicting your pods on Kubernetes 1.33+ via in-place resize, and Goldilocks v4.16.2 (September 15) exposes a /metrics endpoint and regression coverage for InPlace mode. This guide puts the three layers together — visibility (OpenCost), recommendation (VPA/Goldilocks), and actuation (in-place resize) — with the actual math you should run before you claim a saving, because the difference between a request and a reservation is exactly where the money is.

The One Formula That Explains Kubernetes Cost

Everything in Kubernetes cost allocation reduces to a single decision written down in the OpenCost specification: workload cost = max(request, usage) × price. The spec is explicit that this “effectively assigns costs that have been directly reserved or allocated by kube-scheduler.” Usage only counts when it exceeds the request — burst above request is real capacity consumption too — but under-requested, over-consumed pods are billed for what they actually take.

The two faces of the same formula — and where the waste lives:

  allocation cost = max(request, usage) x price      <-- reconciles to the bill
  usage cost      =          usage      x price      <-- what the pod really consumed

  request >> usage  ──▶  request-overhang waste (the classic right-sizing target)
  usage  >> request ──▶  noisy-neighbor / throttle risk (the under-provisioning trap)

  ┌────────────────────────────────────────────────────┐
  │  pod requests 2 CPU / 8 GiB                          │
  │  pod uses     0.4 CPU / 2.1 GiB                      │
  │                                                      │
  │  ████░░░░░░░░░░░░░░  request (reserved capacity)      │
  │  ██░░░░░░░░░░░░░░░░  usage                          │
  │      ▲▲▲▲▲▲▲▲▲▲▲▲▲  the scheduler bills this whole    │
  │                      bar regardless of utilization    │
  └────────────────────────────────────────────────────┘

Run the numbers on a real, current price. OpenCost’s committed public pricing data lists an AWS m6i.2xlarge (8 vCPU / 64 GiB) in us-east-1 at $0.384/hour on-demand (v1.121.3 data file). That works out to roughly $0.048 per vCPU-hour and $0.006 per GiB-hour on that instance family. Now the waste becomes concrete: a pod requesting 2 CPU / 8 GiB that peaks at 0.4 CPU / 2.1 GiB is allocated $0.144/hour of capacity while consuming $0.032/hour of resources. The $0.112/hour difference is pure request-overhang — $81.91/month for that one pod, $40,953/month across a fleet of 500 lookalikes. The bill reconciles to the request, not the usage, and that is the whole game.

Two consequences before we touch any tooling:

Layer 1 — Visibility: OpenCost Without the Guesswork

OpenCost is the CNCF incubating cost-allocation engine behind Kubecost, separated out as a spec plus a Go implementation. The install story changed recently and is now Helm-only — the standalone manifests are gone:

helm repo add opencost https://opencost.github.io/opencost-helm-chart
helm repo update

# Chart 2.5.31 ships OpenCost 1.121.x. The UI and MCP server are on by default.
helm upgrade --install opencost opencost/opencost \
  --namespace opencost --create-namespace \
  --set opencost.prometheus.internal.enabled=true \
  --set opencost.prometheus.internal.serviceName=prometheus-server \
  --set opencost.prometheus.internal.namespaceName=prometheus-system

kubectl -n opencost port-forward service/opencost 9090:80
# UI:       http://localhost:9090
# API:      http://localhost:9003  (exporter apiPort)

The pricing layer is what used to be OpenCost’s weak point — for years it leaned on live provider pricing APIs that rate-limited, changed shape without notice, or required keys for anything useful. The Public Pricing 1.0.0 release (merged August 10, shipped in v1.121.2) changed the architecture: pricing files are now committed to the repository under modules/pricing/public/ with per-provider subdirectories, and a GitHub Action tags and versions the module on every push to that tree. The v1.121.3 follow-up added GCP GPU pricing, AWS marketplace fees, and split basic pricing by provisioning model. What that means operationally:

Once the exporter is running against your Prometheus, the allocation API is a single curl. The default query the OpenCost UI runs is:

curl -G http://localhost:9003/allocation \
  -d window=7d \
  -d aggregate=namespace \
  -d resolution=1m

The parameters that actually matter, from the API docs:

Parameter What it does When you use it
window Time range: 30m, 12h, 7d, month, or explicit UTC timestamps Always. Use a window ≥7d for right-sizing — shorter windows chase noise and miss weekly peaks
aggregate Grouping: namespace, controller, pod, container, label, comma-combined Chargeback by namespace first; container-level before editing any manifest
shareIdle=true Distributes node idle cost across the pods that could have used it Team chargeback — without it, idle hides in a shared bucket nobody owns
idleByNode=true Keeps idle attributed per-node instead of shared Cluster-autoscaler tuning — finding which node pools are over-provisioned
accumulate=day Sums the window into daily totals (also hour, week) Trend lines and month-over-month deltas

For terminal-first teams, kubectl-cost wraps the same APIs as a kubectl plugin (kubectl krew install cost) — and per its own README, “most of kubectl cost works” against OpenCost’s API, with kubectl cost namespace, pod, controller, and node views. The MCP server (on by default since chart 2.5.31, port 8081) even exposes cost queries to AI agents through a standardized interface — useful for an internal platform portal that lets service owners ask “what did my team spend last week” in plain language.

One deployment note that will bite anyone running sharded Prometheus: point PROMETHEUS_SERVER_ENDPOINT at a global query endpoint (Thanos Query, Cortex, or Mimir), not a single Prometheus pod — the OpenCost README warns that a single-pod endpoint produces incomplete, intermittent export results. The chart has dedicated opencost.prometheus.thanos and .amp blocks for exactly this.

Layer 2 — Recommendation: What VPA Actually Computes

The Vertical Pod Autoscaler’s recommender is the de-facto engine behind most right-sizing tooling — Goldilocks is a namespace-label veneer over VPA-in-recommendation-only mode. Knowing what it actually does with your usage histograms separates real savings from cargo-culting. From the 1.7.1 recommender source defaults:

Knob (recommender flag) Default What it controls
--target-cpu-percentile / --target-memory-percentile 0.9 The recommended request targets the P90 of the usage histogram — not the average, not the peak
--recommendation-margin-fraction 0.15 Then adds a 15% safety margin on top of the P90 target
--pod-recommendation-min-cpu-millicores / -memory-mb 25m / 250MB Floor values so tiny containers don’t get starved by rounding
Memory aggregation 24h interval × 8 days Memory is aggregated in 24-hour blocks kept for 8 days — the default window over which the P90 is taken
Histogram decay half-life (CPU & memory) 24h Old usage decays with a 24-hour half-life — a spike from three days ago counts, much less
OOM response ×1.2, min +100Mi After an OOMKill, memory recommendation bumps 20% and at least 100Mi — OOMs move the floor, not just the target

Read that as a policy: “set requests to P90 of 8 days of usage, plus 15%”. That’s materially more conservative than “average × 1.5” folklore, and it self-documents — the recommendation you see in the VPA object is exactly what those parameters produce. For latency-critical or bursty services, the percentile is the knob to argue about: a P90 target means one bad hour in ten lands above the request, where it either throttles (CPU limits) or gets the node’s shared nothing (no limits). If you have CPU limits set and care about p99s, either raise the percentile or delete the limits — the middle ground is where surprise throttling lives.

Continue the worked example with those defaults. P90 usage of 0.35 CPU / 2.0 GiB, plus the 15% margin, gives a recommended request of 0.4025 CPU / 2.3 GiB — an allocation cost of $0.0331/hour against the original $0.1440/hour. That is a 77% cost reduction per pod, and here is the part most teams miss: it converts directly into node-count reduction. One hundred of these pods at 2 CPU each demand 200 schedulable cores — 25 m6i.2xlarge nodes, $7,008/month. The same hundred pods at VPA’s 0.4025 CPU fit in 6 nodes, $1,682/month. Right-sizing is not a percentage-point optimization; it is the difference between a 25-node fleet and a 6-node fleet, minus whatever headroom you deliberately keep.

Right-sizing chain — from usage histogram to node bill:

  cAdvisor metrics          VPA recommender             scheduler / cluster-autoscaler
  ─────────────────         ──────────────────           ─────────────────────────────
  container_cpu_usage_      P90 of 8d histogram    ──▶    request 0.4025 CPU
  seconds_total             + 15% safety margin           (was 2.0 CPU)
  container_memory_
  working_set_bytes         OOM seen? bump x1.2           100 pods: 40.25 CPU total
  (per container,           (min +100Mi)                  vs 200 CPU before
  24h x 8d windows)
                                                        ┌─────────────────────────┐
      │                                                 │  8 CPU/node (m6i.2xl)   │
      │                                                 │  6 nodes now            │
      ▼                                                 │  25 nodes before        │
  OpenCost allocation API                                 │  $1,682/mo vs $7,008/mo │
  max(request,usage) x price                             └─────────────────────────┘
  proves the before/after

The classic operational trap: VPA and HPA cannot both scale the same resource metric — the known-limitations doc is blunt that VPA-on-CPU plus HPA-on-CPU double-controls the same signal. The supported combination is VPA on memory with HPA on CPU (or HPA on custom/external metrics). Also remember VPA 1.7.x is not compatible with pod-level resources stanzas yet (AEP-7571 is in flight) — if you adopted pod-level resource managers, keep VPA off those workloads until that lands.

Layer 3 — Actuation: In-Place Resize Changes the Risk Model

Here is why right-sizing historically stalled in every org that tried it: applying a new request used to mean evicting and recreating the pod. VPA’s Recreate update mode is a controlled disruption loop, and for stateful or latency-sensitive workloads that trade was correctly rejected. Kubernetes’ in-place resize (KEP-1287) — beta in 1.33, GA since 1.35 — changes the actuation economics: resource requests can now change while the container runs, no restart, subject to each container’s resizePolicy (NotRequired by default, or RestartContainer for runtimes/apps that can’t adapt).

VPA rides this in 1.7.x with three actuation modes, and the differences matter operationally:

updateMode Since Behavior Disruption risk
Off always Recommendations computed and written to the VPA object; nothing applied None — the honest starting point
Initial always Applies recommendations to new pods only, at creation, via the admission webhook None to running pods; new pods start right-sized
Recreate always Evicts pods whose requests drift from the recommendation so controllers recreate them Full pod restarts, budgeted via VPA’s eviction rules
InPlaceOrRecreate VPA 1.6 [GA] Attempts in-place resize first; falls back to recreation if infeasible/deferred >5 min, stuck >1h, QoS class would change, or memory-limit downscale hits a PreferNoRestart policy Small — container runtime performs the resize; fallback still evicts
InPlace VPA 1.7 [alpha] Eviction-free: only in-place updates, defers and retries when the node can’t take the resize; never evicts None by design — but resizes may be deferred indefinitely on packed nodes

The engineering detail worth internalizing: a resize succeeds at the node only if capacity is actually available, and kubelet reports back through the pod’s resize status — ResizeDeferred, ResizeInProgress, ResizeInfeasible, ResizeError. VPA’s updater in InPlace mode treats those as retry signals, not failures, and caches infeasible attempts so it only retries when the recommendation improves. Watch the updater’s metrics (vpa_updater_in_place_updated_pods_total, vpa_updater_failed_in_place_update_attempts_total) to confirm actuation is actually happening — a rightsizing program whose resizes all defer is just Off with extra steps.

The newest toy in 1.7 is CPUStartupBoost (alpha): the admission controller temporarily multiplies CPU at pod startup — startupBoost.cpu.type: Factor, factor: 3, durationSeconds: 10 — then scales back down in-place once the pod is Ready. It exists because right-sized JVMs and other startup-heavy runtimes otherwise take a slow-train to first request; the boost fixes the cold-start tax that P90 recommendations impose. Alpha, gated behind --feature-gates=CPUStartupBoost=true on both admission controller and updater, K8s 1.33+. Interesting, not yet load-bearing.

The Control Loop: Putting It Together With Goldilocks

Goldilocks is the glue that makes VPA recommendations consumable by teams rather than by a platform engineer with kubectl. v4.16.2 (September 15) added a controller /metrics endpoint (previously it was dashboard-or-nothing) and regression coverage for InPlace update modes — the maintainers are visibly tracking VPA’s new modes. The mechanics, from the source: Goldilocks watches namespaces for the label goldilocks.fairwinds.com/enabled=true, then creates a VPA per workload controller (Deployment, StatefulSet, DaemonSet) in recommendation-only mode, defaulting updateMode: Off — the safe read-only posture. Teams opt into more via the label/annotation goldilocks.fairwinds.com/vpa-update-mode, which in 4.16.2 accepts off, initial, recreate, inplaceorrecreate, and inplace (case-insensitive; the deprecated auto alias maps to Recreate).

# 1. Install Goldilocks (chart 11.1.1 / app v4.16.2) with the VPA recommender subchart.
#    vpa.enabled=false by default: BYO VPA or let the chart install recommender-only.
helm repo add fairwinds-stable https://charts.fairwinds.com/stable
helm upgrade --install goldilocks fairwinds-stable/goldilocks \
  --namespace goldilocks --create-namespace \
  --set vpa.enabled=true \
  --set vpa.updater.enabled=false \
  --set vpa.exporter.enabled=false

# 2. Enable recommendations for a team namespace (VPA objects created in Off mode)
kubectl label namespace payments goldilocks.fairwinds.com/enabled=true

# 3. Read what the recommender says (target / lowerBound / upperBound per container)
kubectl get vpa -n payments -o yaml | grep -A4 recommendation

# 4. Pick a mode per workload when you trust the numbers — alpha, opt in deliberately:
kubectl -n payments patch deployment checkout-api --type=merge \
  -p '{"metadata":{"annotations":{"goldilocks.fairwinds.com/vpa-update-mode":"inplaceorrecreate"}}}'

# 5. Prove the change with OpenCost, before and after (share idle for chargeback honesty)
curl -G http://localhost:9003/allocation \
  -d window=7d -d aggregate=namespace -d shareIdle=true \
  | jq '.data[].allocations["payments"].totalCost'

# 6. Watch actuation, not intentions — the updater metrics tell you if resizes land:
#    vpa_updater_in_place_updated_pods_total        (going up = working)
#    vpa_updater_failed_in_place_update_attempts_total (going up = investigate node capacity)

One warning from Goldilocks’ own installation docs that survives every version bump: the full VPA install includes the updater and admission webhook — the components that change pod specs and evict pods — and “an admission webhook can introduce unexpected results in a cluster if not planned for properly.” Goldilocks only needs the recommender. If you install VPA via the Goldilocks chart’s subchart with vpa.enabled=true, you get recommender-only by default (vpa.updater.enabled=false); if you install VPA yourself with ./hack/vpa-up.sh, you get the whole apparatus including the webhook mutating every matching pod. Know which one you installed.

The Sane Rollout Plan (and Who Should Skip This)

Everything above compresses to a sequence that respects the blast radius:

Decision matrix — which actuation mode earns trust, per workload class:

  workload class                 first move          why
  ─────────────────────────      ─────────────────   ─────────────────────────────
  stateless, replicas >= 2       InPlaceOrRecreate   resizes land, fallback exists
  bursty / p99-sensitive         Initial only        P90 target underserves tails;
                                                     fix percentile first
  stateful / single replica      Off + manual        eviction cost > waste; review
                                                     per-release instead
  JVM / fixed-heap runtimes      resizePolicy:       in-place memory downscale is
                                 RestartContainer   wasted on a runtime that
                                                     cannot act on it
  pod-level resources (1.37      skip VPA            VPA 1.7 incompatible (AEP-7571)
  PodResourceManagers users)                         until support lands

Who should skip this article’s advice: if you are running a single node with a handful of dev workloads, the visibility stack costs more attention than it saves — read your invoice instead. If your cluster autoscaler scales to zero nightly and your workloads are entirely spot/reclaimed (e.g. Karpenter consolidation aggressively bin-packs), request right-sizing yields less because consolidation already attacks the same waste — though even there, OpenCost’s allocation view is how you prove Karpenter is doing its job. And if you have no baseline, do not start with VPA in any applying mode — recommendations without a measured baseline are just different guesses.

The Skeptic’s Appendix: Where This Stack Still Falls Short

The Cloudflare lesson generalizes beyond proxies: capacity engineering wins are measured wins — they start with an instrument that says exactly how much is wasted, and they end with a number that reconciles to the bill. On Kubernetes, that instrument is OpenCost, the recommendation engine is VPA, the actuation is in-place resize, and the math is one formula: max(request, usage) × price. Everything else is plumbing. Deploy the plumbing, trust the formula, and make the scheduler’s opinion of your pods match reality.