Prometheus 3.16: PromQL Gets a Cost Model — Estimation, Hard Limits, and the Counter-Reset Arc

Prometheus 3.16.0 was cut on November 4, 2026 and published to GitHub releases on November 5 — roughly six weeks after our 3.15.0 analysis, and two weeks after the 3.15.1 security patch. It is the newest stable release of the CNCF-graduated project at the time of writing. On paper it reads like a standard minor: an experimental PromQL function here, a stable-by-default promotion there, and the usual long tail of bugfixes. The real story is narrower and more consequential: 3.16 is the release where upstream Prometheus finally grew a query cost model — estimation before execution, hard limits during execution, and actual-cost reporting after it. That is a governance primitive the vanilla server has never had. Until now, the only built-in guard rails were --query.max-samples (a ceiling on peak samples held in memory) and --query.timeout — both of which abort work after most of it has already happened, and neither of which tells you anything before you run the query.

The gap has been filled by derivatives for years. Run Grafana Mimir as your backend — the architecture we covered in our long-term storage guide — and you get per-tenant query limits in the control plane. Buy Grafana Cloud or Amazon Managed Service for Prometheus and the vendor sets those ceilings for you. Run the single binary that the whole ecosystem is named after, and you got nothing: a single global --query.max-samples shared by dashboards, recording rules, and alerting rules alike. PR #20028 (@roidelapluie, merged October 20) closes that gap behind a feature flag, and it is the feature this release will be remembered for. The rest of 3.16 — the start-timestamp counter-reset work, a cluster of genuinely scary TSDB durability fixes, and the NHCB-to-classic conversion flag for remote write — is what makes the upgrade worth scheduling rather than skipping.

The Pain: One Blunt Ceiling, Applied After the Work Is Done

The industry pain here is not exotic. Every platform team that runs self-managed Prometheus knows the incident shape: a Grafana dashboard with a dozen panels, each refreshing every 30 seconds, one of them running rate() over a high-cardinality label set. Or an engineer pastes a "quick" ad-hoc range query into the expression browser, walks away, and the query grinds the storage for four minutes until --query.timeout kills it. The failure mode of the pre-3.16 tools is that they intervene late: --query.max-samples counts peak samples resident in memory, so a query is aborted only after the engine has already selected the series and read most of the data. Nothing answers the questions an operator actually asks: how expensive will this query be, and can I set a hard budget per query that rejects it the moment it crosses the line?

The 3.16 model splits that into three separate capabilities, all behind one flag:

PRE-3.16: the only knobs intervene after most of the work

  query ─▶ execute ─▶ peak samples > --query.max-samples? ─▶ abort late
                        │                                    (nothing learned
                        └─ nothing tells you the cost         beforehand)
                          before you run the query

3.16 (--enable-feature=query-cost) adds three stages:

  1. ESTIMATE   GET /api/v1/query_cost        never executes the query;
                index walk + density          returns seriesTouched and
                sampling (up to 50 chunks     samplesRead estimates;
                per selector)                 informational only — it
                                              NEVER rejects a query

  2. ENFORCE    during execution, against     query_max_series
                the ACTUAL running cost —     query_max_samples_read
                reject the moment a limit     query_max_duration
                is crossed (rules too)         (surfaces as a timeout)

  3. REPORT     cost=true on the query        estimated vs actual in
                endpoints                     the response, with
                                              peakSamples under actual

The architectural honesty is in the separation: the estimate is never used to reject a query, and the enforcement is never based on the estimate. The feature-flag documentation is explicit that estimation can over- or under-count and is offered as information; the limits are enforced against the actual running series and sample counts, continuously, during evaluation.

The Mechanics: Three Endpoints, Three Limits, One Feature Flag

Enablement is a single flag: --enable-feature=query-cost. Clients can detect it through the /api/v1/features endpoint, where it is reported as query_cost in the api category. With the flag on, three things happen.

First, two new API endpoints appear — /api/v1/query_cost and /api/v1/query_range_cost — which accept the same parameters as the instant and range query endpoints, but return an estimate instead of a result:

curl 'http://localhost:9090/api/v1/query_cost?query=up'
# or with a real PromQL expression, URL-encoded for you:
curl -G 'http://localhost:9090/api/v1/query_cost' \
  --data-urlencode 'query=sum(rate(node_cpu_seconds_total{mode="idle"}[5m]))'

{
  "status": "success",
  "data": {
    "estimate": {
      "seriesTouched": 42,
      "samplesRead": 5040
    }
  }
}

Second, the instant and range query endpoints grow a cost=true parameter that attaches an estimated-versus-actual comparison to the response — you get the result and the bill:

"cost": {
  "estimated": {
    "seriesTouched": 42,
    "samplesRead": 5040
  },
  "actual": {
    "seriesTouched": 40,
    "samplesRead": 4980,
    "peakSamples": 320
  }
}

Third — and this is the part with teeth — the flag activates enforcement of three reloadable limits in the global: section of the configuration file:

global:
  # All three are enforced DURING execution against actual running cost,
  # never against the estimate. 0 / 0s means "no limit" (the defaults).
  # They apply to every query the engine evaluates — recording and
  # alerting rules included.
  query_max_series: 100000
  query_max_samples_read: 50000000
  query_max_duration: 2m

The per-request overrides are deliberately one-directional: a client may only lower a ceiling for its own request. Ask for a higher value than the operator set and the API rejects the request with HTTP 400 (bad_data) instead of silently clamping. That is the correct design — the operator's ceiling is a hard fence, and query authors can opt into a smaller budget for testing, but nobody escapes the fence by asking nicely.

The estimator itself is worth understanding before you trust its numbers, because the docs are refreshingly blunt about its error bars. Estimation enumerates matching series in the index (which scales with selector cardinality and block count) and samples chunk data to measure density — up to 50 chunks per selector's window, and up to 50 series for native-histogram sizing. Samples are counted in the engine's own units: a float sample is one unit, a native histogram is weighted by its in-memory size. Estimates include the extra lookback windows that extended range selectors consume — anchored selectors extend by one lookback delta, smoothed range selectors by two. If remote read is configured, density sampling is unavailable and the estimate falls back to assuming the global scrape_interval with float-sized samples, with a warning per matching selector.

The Hidden Costs Nobody Will Tell You About

The most consequential operational fact is buried in the feature-flag prose: these limits apply to every query the engine evaluates, not only API requests. Recording rules and alerting rules are subject to them. A rule whose query exceeds a limit fails to evaluate like any other query error — meaning an aggressively tight query_max_series will break your recording-rule layer at the next config reload, and the failure will look like rule evaluation errors, not like a governance policy working as intended. There is a second, subtler interplay: while the flag is enabled, query_max_duration replaces --query.timeout as the timeout of rule queries, and the command-line flag quietly stops governing them. If your team tunes --query.timeout and then sets query_max_duration from a different pull request, you have two overlapping timeout knobs with precedence most engineers will not remember.

The rollout that respects these mechanics is staged: enable the flag with all three limits at 0 (pure observability), mine the cost=true data and the /api/v1/query_cost endpoint to build an actual distribution of query costs — recording rules included — and only then tighten the ceilings to cut off the tail. Going straight to tight limits is the classic way to turn a governance feature into a self-inflicted outage.

Two more accounting changes ride along in 3.16 and will confuse anyone diffing query statistics across versions: subquery samples are now included in the query TotalSamples and TotalSamplesPerStep statistics, and SamplesRead now counts reads from subqueries with an @ modifier in range queries. A fix in the same release keeps materialized subquery samples from being double-counted in the total. The net effect: dashboards that chart query stats will shift after this upgrade even when the queries are identical. Re-baseline them; don't file a bug.

And the boundary that remains: these limits are global. One ceiling per Prometheus server, no per-tenant, per-user, or per-team scoping. If your actual problem is that team A's dashboards starve team B's alerting, vanilla 3.16 still cannot express that — that is the per-tenant limits story in the Mimir architecture, and it is a real reason teams stay on the managed path rather than self-hosting the single binary.

Anchored and Smoothed Graduate: Stable by Default, With a Rule-Group Trap

The second headline is a promotion: PR #19697 (@roidelapluie, merged September 15) enables the extended range selector modifiers — anchored and smoothed — by default. They are no longer experimental flags; they are stable, always-on PromQL grammar, with the semantics specced in the April 2025 design proposal in the project's proposals repository.

The motivation is the oldest complaint in PromQL: rate() and increase() extrapolate to range boundaries, which produces smooth numbers that are wrong in predictable ways when scrapes are irregular or series are short-lived. The two modifiers take opposite positions on the trade-off. anchored refuses to extrapolate at all — it anchors the calculation to the samples that actually exist: the most recent sample within the lookback delta at the beginning of the range (or the first sample inside the range if none qualifies), and the last sample inside the range at the end. What you get is the direct difference between sample values, nothing invented. smoothed goes the other way: it linearly interpolates values at the range boundaries using samples on both sides, producing an estimate that is robust against irregular scrapes and missing samples — but it needs a sample after the evaluation interval to do that interpolation, which is where the trap lives.

# anchored: no extrapolation, direct difference between actual samples
# (increase over floats returns integers with anchored)
increase(http_requests_total[5m] anchored)

# smoothed: linear interpolation at window boundaries, needs one sample
# beyond the evaluation instant; step() resolves the query step as a range
rate(http_requests_total[step()] smoothed)

groups:
  - name: smoothed-rates
    query_offset: 1m        # >= one scrape interval; more for missed scrapes
    interval: 30s
    rules:
      - record: job:http_requests:rate5m_smoothed
        expr: rate(http_requests_total[5m] smoothed)

One portability consequence nobody will spell out for you: these are now parse-time grammar. A recording rule that uses smoothed will not evaluate on a pre-3.16 Prometheus — the rule file fails to parse on the older engine. If your rule files are evaluated by mixed-version components (a fleet mid-upgrade, a secondary ruler, a downstream tool that vendors the PromQL engine), adopting the modifiers is a fleet-wide commitment, same class of constraint as the XOR2 one-way door we covered in the 3.15 analysis — annoying but honest, and documented.

Start Timestamps: The Counter-Reset Arc Continues

The extended-selector promotion is one move in a longer campaign: making counters exact by carrying explicit reset information instead of inferring it from data dips. 3.16 adds three more pieces. PR #18966 (@vpranckaitis, merged October 13) uses start timestamps to detect counter resets when float series are queried with anchored/smoothed selectors, behind the use-start-timestamps feature flag. Note the honest boundary in the feature-flag docs: full start-timestamp processing in functions like rate(), irate(), increase() and start_timestamp() still "doesn't currently work with extended range selectors" — what #18966 adds under the same flag is the narrower reset-detection capability. Precision over promise.

PR #19066 (@69iqqq, merged September 11) adds a per-query escape hatch at the HTTP layer: the X-Prometheus-Use-Start-Timestamps header overrides the engine-level start-timestamp setting for a single request — the kind of knob a frontend or testing harness uses to A/B a rule expression with and without ST processing without restarting anything.

And for series whose sources never emit start timestamps at all, there is now --enable-feature=st-synthesis: Prometheus synthesizes start timestamps for cumulative metrics (counters, classic and native histograms) by tracking previous values, detecting resets, and subtracting the initial reference point — the same strategy the OpenTelemetry collector's metricstarttimeprocessor uses, and the docs say exactly that. The flag is marked experimental, which is the correct posture for anything that rewrites ingest-time semantics.

The Durability Cleanup Crew: Three Silent Sample-Loss Fixes

Buried below the features is the part of the release that should drive your upgrade scheduling if you run out-of-order ingestion windows or the newer selected-series and stale-series compaction paths. Three of the TSDB fixes are silent-data-loss classes — no crash, no error log, samples simply not there afterward.

The nastiest is PR #19666 (@duricanikolic, merged September 9): during selected-series or stale-series compaction, a sample that committed after the block was written but before the series was evicted from the head could be silently lost. The fix — retaining any series that received a sample since compaction started — no longer depends on query isolation being enabled, which matters because isolation is not on everywhere.

selected-series / stale-series compaction, pre-3.16:

  block written for series S ──▶ S still in head, awaiting eviction
                │
                └─▶ a NEW sample for S commits after the block write,
                    before eviction from the head
                              │
                              ▼
                eviction removes S from the head; the late sample
                is in neither the block nor the head
                              │
                              ▼
                sample SILENTLY LOST — no error, no log, no metric

3.16 (#19666): a series that received ANY sample since compaction
started is retained; the protection no longer depends on isolation.

The companion fix PR #19672 (same author) makes the append-ID watermark check durable against later cleanup, closing a rarer variant where an unrelated write to the same series erased the evidence needed to protect it. PR #19734 clears up the aftermath: spurious "unknown series references" warnings during WAL replay when the only blocks on disk were produced by these compaction modes — harmless-looking noise that made real corruption harder to spot.

Out-of-order ingestion gets its own loss fix: PR #19737 (@replay, merged October 2) fixes the case where a WAL checkpoint dropped a series record that the out-of-order write-behind log still needed — losing out-of-order samples on the checkpoint path. If you enable out_of_order_time_window for late-arriving telemetry, this fix is your reason to move off 3.15.x.

The rest of the durability list, rapid-fire, because each one is a distinct failure mode: PR #19854 (@aknuds1) stops queries that seek within a head chunk from returning samples committed after the query started — a query-isolation violation, not a durability one, but the same "result you cannot trust" family. PR #19940 preserves the series identities needed to recover exemplars after compaction and restart. PR #20041 (@bwplotka) fixes the WAL watcher permanently skipping checkpoint series garbage collection when it lagged behind head truncation — a slow resource leak disguised as bookkeeping. And PR #19858 now increments prometheus_tsdb_wal_corruptions_total when corruption is found during checkpoint creation, so the canary metric reflects corruption found at compaction time instead of only after a restart-triggered repair — worth wiring into an alert if it is not already there.

Two behavior fixes deserve equal billing with the data-loss items. PR #19882 (@ajaykr0905) restores firing alerts whose for durations are shorter than --rules.alert.for-grace-period after a restart — previously those alerts went pending again and re-fired from scratch, which is exactly the flapping you do not want during the restart of the machine that is telling you things are broken. And PR #19866 (@thampiotr) guards remote write against future-dated exemplars: before the fix, a single producer with a skewed clock could scale remote-write shards down to the minimum and advance Agent-mode WAL truncation beyond the retention window. Clock skew as a data-loss vector is an underrated production story, and this is a clean fix.

Interop: NHCB-to-Classic Conversion, and Agent Mode Learns Metadata

The remote-write side of 3.16 is about translation, and it is a shrewd piece of product thinking. Native histograms are the better storage story, but most receiving systems still cannot ingest them. PR #17205 (@Naman-B-Parlecha, merged October 19) adds an experimental per-endpoint switch: convert_nhcb_to_classic, which converts native histograms with custom buckets (NHCB) into classic _bucket/_count/_sum series after write_relabel_configs are applied. The pitch: keep NHCBs locally for cheaper storage and histogram atomicity, while sending classic histograms to receivers that are not native-capable yet — and use the same switch as a migration lever.

remote_write:
  - url: https://receiver.example.com/api/v1/write
    # EXPERIMENTAL: converts native histograms with custom buckets
    # to classic histograms on the wire. Default is false.
    convert_nhcb_to_classic: true

Read the fine print before flipping it: conversion happens even when send_native_histograms is enabled (that flag governs forwarding of other native histograms); classic series already ingested alongside NHCBs — for example via always_scrape_classic_histograms — are still sent, so the converted series duplicate them and you are expected to drop one copy with write_relabel_configs; gauge histograms convert to just _count and _sum; and conversion state is not persisted across restarts, so bucket-layout or sample-type changes that span a restart will not produce stale markers for retired classic series. "Experimental" is doing real work in that flag name.

Agent mode — the scrape-and-forward deployment — finally gets parity on metadata: PR #19684 (@bwplotka, merged October 20) enables metadata-wal-records in Agent mode, so Remote Write 2.0 metadata flows through the Agent's WAL like it already did in server mode, with the companion fix (#19678) repairing the case where the main feature flag was never passed through to the TSDB. And on the same theme of cleaning up the RW 2.0 path, PR #20080 (@shilohlee98) stops exposing the legacy metadata metrics for Remote Write 2.0 queues — a small breaking change with a real blast radius if your dashboards still reference them:

If you alert or panel on any of those against RW 2.0 queues, they go dark the moment 3.16 lands. Remote Write 1.0 queues keep the metrics. Audit your dashboards before the upgrade, not after the empty graphs.

The Rest, Rapid-Fire

Upgrade Plan, and Who Should Skip This Release

The upgrade itself is a normal minor: no new one-way storage doors this time (XOR2 was the 3.15 event), and the changelog carries the usual deprecation hygiene. What deserves a checklist before you roll:

• Enable query-cost with all limits at zero first. Use /api/v1/query_cost and cost=true to baseline real costs — recording rules included — for a week before tightening query_max_series or query_max_samples_read. Remember the limits bind rules, and query_max_duration overrides --query.timeout for rule queries while the flag is on.
• Audit dashboards for the four legacy RW 2.0 metadata metrics above — they disappear in 3.16.
• Re-baseline query-stat dashboards. Subquery samples now land in TotalSamples/TotalSamplesPerStep; the numbers move even when queries do not.
• Check exporters emitting duplicate exemplar label names — those get rejected now.
• If you run TLS with --web.config.file and never set --web.external-url, expect generated links to flip to https.
• Move fast if you use out-of-order ingestion windows (the WBL checkpoint fix), selected-series or stale-series compaction (the silent-loss trio), HTTP/2 with TLS on Go 1.27+ builds, or restart-prone alerting with for durations under the grace period. Those are correctness and durability fixes, not conveniences.

Who should skip: if your Prometheus is fully managed — Amazon Managed Service for Prometheus or Grafana Cloud — the upgrade cadence is the vendor's problem and per-tenant query limits already exist in their control planes; nothing in 3.16 changes your day. If you run a single-user, single-purpose server with no rule load, the cost-governance story has no audience and the release reduces to "nice bugfixes, upgrade when convenient." And if your actual requirement is per-team query budgets on shared infrastructure, vanilla 3.16 still does not express that — the limits are global-only — so the multi-tenant answer remains a backend like Mimir in front of your metrics.

Who Did the Work

Credit where it belongs, per the release notes: the query cost API and the extended-selector default came from @roidelapluie; @bwplotka landed the Agent-mode metadata work, the WAL-watcher garbage-collection fix, and the remote-write receiver migration to AppenderV2; @duricanikolic authored three of the compaction durability fixes; @replay the out-of-order WBL fix; @aknuds1 the head-chunk isolation fix; @thampiotr the future-timestamp guard; @ajaykr0905 the alert-restore fix; @69iqqq and @vpranckaitis carried the start-timestamp API work; @jjo contributed integral(); @Naman-B-Parlecha the NHCB conversion; @shilohlee98 the legacy metrics cleanup; @Ramya-9353 the exemplar uniqueness rejection; @hexonal the external-URL scheme default; and @dongjiang1989 the HTTP/2 ALPN repair. That is a healthy long-tail contributor mix for a minor release — the governance feature from a core maintainer, the durability cluster from relative newcomers, and the interop glue from across the ecosystem.