Every platform team running vanilla Prometheus eventually hits the same wall: the local TSDB is a single-node database, retention fights with disk space, and the "just add Thanos" answer adds four more services than anyone wanted. VictoriaMetrics — the Apache-2.0 TSDB from the Latvia-based company of the same name — is the most credible "just replace the storage" answer in the CNCF-adjacent market, and this week it shipped v1.153.0 with two security advisories, OIDC single sign-on, and a batch of operational fixes. The repository turned eight years old the same week — created 2018-09-30, now at 17,788 stars and a release cadence that lands a minor every two weeks.
We didn't read about this one: we downloaded the actual v1.153.0 linux-amd64 binary, verified its SHA-256 checksum against the published checksums file, and ran it through a 300,000-sample high-churn ingest, restart cycles, Unix-socket permission tests, and a deliberate probe of every admin endpoint. The result is a two-sided verdict, like most honest ones: the storage engine is genuinely excellent, the release hygiene is better than most vendors', and the operational defaults contain at least three traps that will bite anyone who deploys it the way the quickstart suggests.
Executive Scorecard
Dimension
Score
Why
Reliability
8/10
0.2s clean shutdown, 0.2s storage reopen with all 503,613 rows intact, per-URL persistent disk queues in vmagent; monthly partitions make retention pruning a non-event. Loses points for a 5-second unflushed-data window and single-node being a single point of failure by design
DX
7/10
Drop-in Prometheus remote-write target, MetricsQL is a strict superset, vmui is genuinely useful; but ~1,000 flags, a two-stage data-visibility model (5s flush + 30s latency offset) that looks like data loss until you read the docs, and dashboards that silently show nothing for fresh series
Cost
9/10 community / 4/10 enterprise
Community edition is free and includes the full storage engine; downsampling, multiple retentions, and automatic vmstorage discovery are Enterprise-gated behind a quote-only plans page, or a managed VictoriaMetrics Cloud
Security
5/10
v1.153.0 fixes a memory-exhaustion advisory in the native import path and an OIDC redirect flaw, and adds OIDC SSO — but /api/v1/admin/tsdb/delete_series accepts unauthenticated deletes out of the box, and the docs' own security section assumes you'll front everything with vmauth yourself
Scope note, the honest kind: all numbers below were measured on this review's host — 4 CPU cores, 30 GB RAM, local SSD, single-node binary, default flags unless stated. They are single-run wall-clock numbers from a working benchmark, not a normalized multi-host benchmark suite; treat them as directional, and re-run on your own hardware before making a procurement decision. What they reliably demonstrate is the shape of the system: where it's fast, where it's clever, and where it will surprise your on-call engineers.
Architecture Mechanics: What Actually Runs
The single-node binary is one process with one job: absorb samples, index them, keep queries off the disk where possible. The design choices that matter operationally are all visible in the startup log:
remote writers (Prometheus, vmagent, OTel, Influx, Datadog agent)
|
| /api/v1/write (snappy+protobuf remote write)
| /api/v1/import/prometheus (exposition text, fastest bulk path)
| /api/v1/import (JSON line)
| /api/v1/import/csv, /influx/write, /opentelemetry/v1/metrics
v
+-----------------------------------------------------+
| victoria-metrics-prod (single process) |
| |
| in-memory rows --flush every 5s--> data parts |
| (-inmemoryDataFlushInterval, min 1s) |
| |
| storage layout (per -storageDataPath): |
| data/data/small/YYYY_MM small parts |
| data/data/big/YYYY_MM big parts (compacted) |
| data/indexdb/YYYY_MM inverted index |
| cache/rollupResult query result cache |
| snapshots/, metadata/, tmp/ |
| |
| monthly partitions: retention prune = drop month |
| adaptive limits at boot: |
| -search.maxUniqueTimeseries derived from |
| remaining RAM (925,535 on this 30GB host) |
+-----------------------------------------------------+
|
| PromQL/MetricsQL queries, -search.latencyOffset=30s
| (samples < 30s old excluded by default)
v
/api/v1/query, /api/v1/query_range, /api/v1/series,
/api/v1/export, Graphite API, vmui (web UI)
Three mechanics from that diagram drive everything else in this review. First, ingestion is decoupled from persistence: samples sit in memory for up to 5 seconds (-inmemoryDataFlushInterval, minimum 1s) before a guaranteed disk flush — the docs explicitly note the flush is what survives unclean shutdowns like OOM kills and power loss. Second, query results exclude the last 30 seconds of samples by default (-search.latencyOffset, overridable per query via a latency_offset argument, floor 1ms) — we verify both windows live below, because their combination is the number-one "VictoriaMetrics ate my data" false alarm. Third, data is organized into monthly partitions: on our test host the boot log showed partition "2026_09" has been created and creating a partition "2026_10" — which means retention is enforced by dropping whole months, the same trick ClickHouse users will recognize, and it makes -retentionPeriod=100y-style configs operationally boring (our test ran exactly that).
The cluster version fans the same storage engine out into three services — vmstorage (state), vminsert (write router), vmselect (query router) — with multitenancy as a first-class URL path (/insert///prometheus/api/v1/write), replication via -replicationFactor at vminsert, and per-tenant limits in the Enterprise edition. The single-node binary does not support multitenancy at all: the docs are explicit that when single-node is read through -vmselectAddr, everything belongs to the 0:0 tenant. Teams that need hard tenant isolation on the write path are in cluster territory, full stop.
Where does Prometheus fit? VictoriaMetrics' own guidance — and our testing backs the shape of it — is that the single-node binary comfortably handles ingestion below a million samples per second, while Prometheus' local TSDB remains the right scraper for small estates: it is a single-node database in its own words ("not clustered or replicated... should be managed like any other single node database") with two-hour blocks and an explicit steer toward remote storage for durability. The realistic architectures are therefore: Prometheus scraping + VictoriaMetrics storing (remote write, the common case), vmagent scraping + VictoriaMetrics storing (replaces the Prometheus server entirely, adds per-URL persistent disk queues and relabeling), or VictoriaMetrics cluster (horizontal storage, multitenancy, and the Enterprise features). What does not work — and we've watched teams try — is treating single-node VictoriaMetrics as both scraper replacement and long-term system of record on one box, with no replication and no disk redundancy.
Hands-On: Ingest, Query, and the Two Windows That Look Like Data Loss
We ran the real binary end to end. Setup is exactly as advertised — untar, run with a data path and retention, health endpoint responds immediately:
$ sha256sum victoria-metrics-linux-amd64-v1.153.0.tar.gz
1b495bde563825cf83dc7c0425a9d8fa03e7214858e0949f9177efc6bb1f8bfc
$ ./victoria-metrics-prod -storageDataPath=/tmp/vmtest/data \
-retentionPeriod=100y -httpListenAddr=:8428
# startup log (verbatim, 4-core / 30GB host):
# partition "2026_09" has been created
# creating a partition "2026_10" with smallPartsPath=...
# limiting -search.maxUniqueTimeseries to 925535 according to
# -search.maxConcurrentRequests=8 and remaining memory=1480856372 bytes
# successfully opened storage in 0.107 seconds
# pprof handlers are exposed at http://0.0.0.0:8428/debug/pprof/
$ curl -s http://localhost:8428/health
OK
That pprof handlers are exposed line at startup is your first security clue: this is a component that assumes it lives on a trusted network segment unless you lock it down. More on that in the failure-modes section.
Ingest throughput: the headline claim holds
We generated a hostile payload — 5,000 series with pod labels that churn six times per series (30,000 distinct label sets), 300,000 total samples — and pushed it through the two main ingest paths. On this modest 4-core host:
HTTP 204, zero errors. Docs correctly warn this path is "optimized for performance" and not for lossless streaming ingestion
/api/v1/import (JSON line, same 300k samples)
4.33s
~69,000 samples/s
Same data, 4x slower — format choice is a real decision at scale
~277k samples/s on four cores with default flags is consistent with the project's long-standing reputation here, and it's the single strongest argument for the migration: that is comfortably above what a mid-size fleet scrapes, on one process, with headroom to spare. After the load test the index reported totalSeries: 259,317 and totalLabelValuePairs: 1,555,879 — our label churn created a quarter-million live series, and neither ingest nor the subsequent full-index aggregation query (sum(rate(http_requests_total[5m])) by (dc) fetching 198,211 series in 629ms) degraded. A 5000-series rate() over the raw churn set took 2.7s — heavier, because per-series rollups over hostile label sets are index-bound, and that is where your capacity planning lives.
The two-window gotcha: 5 seconds of flush, 30 seconds of latency offset
Here is the trap we hit on day one, and the one that will generate most of your false "data is missing" incidents. We imported a sample and queried it back one second later: HTTP 204 on import, empty result on query. No error, no log line, no hint. The sample was in the process — /api/v1/export couldn't see it either. The mechanics, verified live:
Two independent mechanisms, both defaults: samples are not guaranteed on disk (or visible to export) until the 5-second -inmemoryDataFlushInterval fires — that flush is what survives an OOM kill, per the docs — and even after flush, instant queries exclude anything newer than -search.latencyOffset (30s) so that late, out-of-order samples from slow scrapers don't tear holes in the last data point. The per-query escape hatch is real but has a floor: latency_offset=0s returns 400: latency_offset=0ms is out of allowed range [1ms ... 3153600000000ms], while latency_offset=1ms works. Prometheus has no equivalent visibility gap for freshly scraped local data — a scraper reads its own head immediately — so teams migrating dashboards will see "missing" panels for the first 30 seconds of every series' life. Operational fix: either tune -search.latencyOffset down fleet-wide (at the documented cost of occasionally incomplete last points for slow/lagging writers), or bake latency_offset=1ms into internal "did the data land" probes — never into customer-facing alerting, where the guard exists for good reason.
Restart, durability, and the umask fix
Graceful shutdown and reopen is where the storage engine shows its quality: SIGTERM shut the process down in 0.203s with the rollup cache saved; restart reopened the same data directory in 0.215s with all partsCount: 6; blocksCount: 498650; rowsCount: 503613 intact and queryable. Data does not go through a risky block-compaction gauntlet on restart the way large Prometheus TSDBs can after an unclean kill.
One v1.153.0 bugfix earned specific verification because it's the kind of thing that breaks real deployments: Unix-domain-socket listeners previously hardcoded 0600 permissions, which meant a reverse proxy running as another user simply could not connect. The fix (#11615) makes socket permissions follow process umask. Confirmed live on v1.153.0:
$ umask 077 && ./victoria-metrics-prod -httpListenAddr=unix:/tmp/vm.sock ...
$ stat -c '%a %n' /tmp/vm.sock
700 /tmp/vm.sock # <- follows umask now
$ umask 022 && ./victoria-metrics-prod -httpListenAddr=unix:/tmp/vm2.sock ...
$ stat -c '%a %n' /tmp/vm2.sock
755 /tmp/vm2.sock # <- also follows umask (mind your umask!)
$ curl --unix-socket /tmp/vm.sock http://localhost/health
OK
The flip side: a careless umask now produces a world-connectable metrics socket. The footgun changed shape; it didn't disappear.
What v1.153.0 Actually Changed (and What It Signals)
Reading the release notes as an operator rather than a changelog consumer, three items stand out:
Two security fixes, both worth understanding before you upgrade-or-ignore.GHSA-8g4f-32hw-vqf8: vminsert, vmagent, and vmsingle now correctly apply memory limits to zstd-encoded blocks arriving via the native import path — previously a maliciously crafted request could balloon memory allocation during ingestion. If your import endpoints are network-reachable (see the delete-series finding below), this is your patch-or-else item. GHSA-xxqh-2hcc-9fp6: vmauth's OIDC Discovery HTTP client no longer follows redirects outside the original request host. Both were also backported to v1.148.5 and v1.136.19 — the project maintains LTS branches, which is more release hygiene than most OSS vendors manage. (The advisory detail pages were not yet published at review time; the descriptions here come from the release notes and fix PR #11616.)
vmauth learned OIDC single sign-on (docs, #10278): unauthenticated browser requests redirect to your IdP, vmauth verifies the response and sets a session cookie. This is a real feature for the "self-hosted Grafana behind SSO" pattern — it removes the last common excuse for putting dashboards behind basic auth — but note it landed in vmauth, the router, not in the storage components themselves.
Direct-to-vmstorage remote write (#11252, opt-in via -enableIngestionAPI): a documented escape hatch that skips vminsert in the cluster. Read it as an acknowledgement that at extreme ingest rates the router can be the bottleneck — and as a trap, because bypassing vminsert also bypasses its replication and retry machinery.
Smaller but operationally pleasant: vmalert can now restore an alert directly to firing state after restart instead of sitting through a redundant pending period (#11401) — closing a genuine "we restarted the alerter and swallowed the incident" gap; vmalert actually reloads alert_relabel_configs on config reload instead of only at startup (#11635); empty extra_label/extra_filters[] query args are ignored instead of erroring (#11618) — which matters because it makes vmauth-based query-arg scrubbing work as documented; and stream aggregation reverted a parallelization that cost CPU for negligible latency gain (#9878). The pattern across the list is a project maturing: fewer new frontiers, more "we found the thing that was quietly wrong."
The Failure Modes You Will Actually Hit
1. The unauthenticated delete API (the finding that should change your deployment). With default flags, POST /api/v1/admin/tsdb/delete_series?match[]=...just works. We tested it: no auth header, no auth key, HTTP 204, series gone — then confirmed via export that the data was really deleted, and found the operation in the logs (Deleted 1 series). The same applies to /tags/delSeries. The fix is one flag — -deleteAuthKey=... forces an auth key for exactly these endpoints — plus, per the docs' own security section, fronting the whole HTTP surface with vmauth or an equivalent gateway, because admin endpoints, /debug/pprof, and /metrics are all exposed on the same listener otherwise. If VictoriaMetrics is reachable by anything other than a trusted service network, this is a data-destruction primitive sitting on port 8428. In 2026, "the network is the perimeter" is not a design, it's a regression.
2. High-churn label sets are the real capacity ceiling. Our synthetic pod-churn payload produced 259,317 live series from a conceptual 5,000 — because every distinct label set is a new series, and the index bills you for all of them (totalLabelValuePairs: 1,555,879 from one 300k-sample burst). VictoriaMetrics handles churn better than most (that is the core compression story), but the failure mode is silent: ingest stays fast, queries stay correct, and RAM use creeps with index size. The boot-time adaptive limit (-search.maxUniqueTimeseries was auto-derived to 925,535 on our 30GB host, explicitly tied to -search.maxConcurrentRequests=8 and remaining memory) will eventually refuse queries rather than OOM — which is the right failure mode, but you want to catch the trend on /api/v1/status/tsdb (excellent cardinality breakdown by metric name, label name, and label pair — a Grafana-billable feature shipped free) before the limit starts refusing queries during an incident. The Enterprise "per-tenant resource limits" page exists precisely because shared multitenant clusters without caps are where VictoriaMetrics deployments go to die.
3. Enterprise gating on exactly the retention-economics features. Here is the commercial sharp edge: community edition gives you one global -retentionPeriod. Downsampling (-downsampling.period=30d:5m,180d:1h) and multiple retentions per dataset are Enterprise features. The docs do offer the honest community workaround — run N instances with different -retentionPeriod + -storageDataPath behind vmauth routing — but that is now N databases to back up and monitor. The economics argument that sold VictoriaMetrics ("cheaper than Thanos at 13x compression") quietly assumes you don't need tiered retention, and most teams that keep metrics a year discover they do. Factor the Enterprise license or the managed VictoriaMetrics Cloud into the TCO from day one, because that is where the retention features live.
4. The single-node ceiling is real and arrives without warning. The docs' own line: single-node is recommended below a million samples per second and "can be set up in High Availability mode" — but HA here means two independent nodes and deduplicated remote write to both (via -dedup.minScrapeInterval), which is a read story, not a storage-consistency story. There is no replication, no shared-nothing cluster failover, and a dead node means its unflushed 5-second window (plus whatever the last vmagent queue hasn't replayed) is gone until you restore from vmbackup. The correct sizing question for single-node is not "how fast can it ingest" but "what is my RPO if this box dies" — and with default flags the honest answer is five seconds of writes plus your backup interval.
Migration Path, Honestly Assessed
The drop-in claim is mostly true: point Prometheus' remote_write at <vm>:8428/api/v1/write and Grafana stops being able to tell the difference, because the query API is Prometheus-compatible with MetricsQL as a strict superset. Two differences will surface in your dashboards, both verified from the docs and our run: increase()/rate() don't extrapolate (slow integer counters return integers, not Prometheus' fractional estimates — your SLO charts will move by a few percent and that is VictoriaMetrics being more correct, but a dashboard-diff will flag it); and metric names are dropped after functions/binary operators unless you opt in with keep_metric_names. For the data itself, vmctl block-migrates Prometheus TSDB snapshots into VictoriaMetrics natively. The vmagent swap is where teams save the most operational weight: it speaks Prometheus scrape_configs natively (-promscrape.config=/etc/prometheus/prometheus.yml, with -promscrape.config.strictParse=false to tolerate unsupported sections), and every -remoteWrite.url gets its own persistent on-disk queue that survives remote-storage outages and replays in FIFO order — with -remoteWrite.streamParse for the documented OOM protection on very high-cardinality scrapes. That is a genuinely better collector than raw Prometheus at fleet scale, and it's the piece of this stack we'd adopt first.
Verdict
VictoriaMetrics is the best default answer in open-source metrics storage, and v1.153.0 is a good version of it. The storage engine earns the reputation: quarter-million-series ingest at 277k samples/s on four cores, 629ms full-index aggregations, 0.2s clean shutdown and reopen with every row intact, monthly partitions that make retention boring, and a security posture that is improving in the right direction (memory-limit advisory fixed and LTS-backported, OIDC redirect contained, vmauth SSO added, umask sockets fixed). The Prometheus-compatibility surface is faithful enough that migration is an ops project, not a re-platform, and vmagent's persistent queues are strictly better than what raw Prometheus gives you at fleet scale.
But deploy it like the quickstart shows and you will get exactly three nasty surprises: panels that look empty for the first 30 seconds of every series' life (flush window + -search.latencyOffset), a metrics port where anyone on the network can permanently delete data (-deleteAuthKey unset), and the late discovery that the retention-economics features that justified the whole migration — downsampling, tiered retention — are behind the Enterprise license. None of these are hidden; all of them are documented. The gap between "documented" and "configured correctly in your environment" is where this review spent most of its time, and it is the actual adoption cost.
Who should skip VictoriaMetrics entirely
Teams that need tiered retention or downsampling and won't buy Enterprise — the community workaround (N single-retention instances behind vmauth) multiplies your backup and monitoring surface. If 13x compression without downsampling already fits your budget, fine; if you're counting on downsampling for the long-term math, price the license first.
Anyone unwilling to run a fronting gateway — until you put vmauth (or your own ingress) in front with auth and route filtering, you are one unauthenticated POST away from a very bad day. This is a hard prerequisite, not an option.
Hard-multitenant SaaS on the write path — single-node has no multitenancy at all (everything is tenant 0:0); multitenancy means the cluster version, and per-tenant resource caps (the thing that keeps one noisy tenant from degrading everyone) are Enterprise.
Small estates already happy with vanilla Prometheus — if a single Prometheus with 15–30 days of local retention covers you and you have no long-term-storage requirement, you'd be adopting a second database to solve a problem you don't have. Revisit when retention pressure or federation actually arrives.
Teams that need traces and logs in the same query plane — VictoriaLogs exists as a separate product with its own engine; the "one observability backend" story is looser than what Grafana Mimir-plus-Loki-plus-Tempo markets as an integrated stack. If vendor-stack cohesion matters more than per-engine excellence, weigh accordingly.
For everyone else — Prometheus estates drowning in disk, platform teams that want long retention without the Thanos/Cortex/Mimir sidecar sprawl, and anyone who has already accepted "metrics storage is a database and I will operate it like one" — VictoriaMetrics is the correct default, and the migration is cheaper than the quarter you'll spend avoiding it. Just front it with vmauth, set -deleteAuthKey before the first day, and decide your retention-tiering answer before the disk fills, not after.
Update — November 7, 2026: the two-week cadence bit back, and migrations grew a silent-skip footgun
Three releases have shipped since this review's v1.153.0 run — v1.154.0 (October 12), v1.154.1 (October 20), and v1.155.0 (October 26) — and together they tell the two-sided story every fast-cadence project eventually tells. v1.154.0 shipped a query-path memory regression that can OOM your storage node; the project acknowledged it with unusually honest retroactive update notes and a "roll back to v1.153.0" advisory; and the v1.155.0 fix lands as a breaking change to a documented admin endpoint that quietly changes what vmctl migrates between deployments. We re-downloaded both the v1.155.0 and v1.153.0 cluster tarballs, verified each SHA-256 against the published checksums files (both matched), ran real single-vmstorage clusters, and reproduced the behavior change end to end — output below, verbatim. Scope note: the original benchmarks in this review ran on a 4-core / 30 GB host; this update's cluster ran on a smaller 4-core / ~5 GB sandbox, which is the right environment for behavior verification, not for throughput claims.
The v1.154.0 regression: what removing the global index actually did
The mechanics, verified against issue #11791 and the triggering commit (lib/storage: do not search global index when per-day index is enabled (#11697)): before v1.154.0, searches with time ranges longer than ~40 days switched to a global index. v1.154.0 removed that switch, so a long-range search now scans per-day indexes concurrently — N_partitions × M_days_in_partition goroutines at once. On a cluster with years of retention that is thousands of concurrent index scans triggered by a single request, and vmstorage gets OOM-killed. The victim list is wider than it looks: the report that surfaced it — vmstorage kept restarting ungracefully on VictoriaMetrics' own play deployment after the v1.154.0 upgrade, fixed by rolling back to v1.153.0 — notes that just opening vmui through vmselect sends the killing request, because vmui calls /admin/tenants with no time range, and no time range means the whole retention. The same index-scan explosion applies to long-range /api/v1/query_range, /api/v1/series, and /api/v1/label/.../values requests on both vmsingle and cluster vmstorage.
GET /admin/tenants (no start/end) == time range [0, now] == whole retention
BEFORE v1.154.0 v1.154.0 .. v1.154.1
------------------ -----------------------
range > ~40 days? range > ~40 days? (or no range at all)
| |
v v
switch to GLOBAL index NO global index switch anymore
| |
v v
one index scan scan EVERY per-day index
| | (one goroutine per day,
v | all started at once)
result v
N_partitions x M_days_in_partition
goroutines -- e.g. 36 months of
data = thousands of concurrent
index scans from ONE request
|
v
vmstorage RSS spike -> OOM kill
(restart loop: every vmui open
via vmselect re-triggers it)
Two companion regressions shipped in the same release, and both are worth understanding because they show how this class of bug arrives. First, #11774: chained-vmauth setups — which is exactly what the VictoriaMetrics operator deploys for VMCluster/VMDistributed, so the managed path was the broken path — gained 2–3 extra seconds on every request. The cause is almost comically ironic: v1.154.0 added a deliberate slowdown for requests bearing invalid auth tokens (a brute-force mitigation published as GHSA-rpfx-hrcp-f73x; the advisory detail page is still unpublished as of November 7, so the release note is the source), and the second vmauth in a chain receives the first one's Authorization header, classifies it as an invalid token, and sleeps. It was fixed in v1.154.1 eight days later, with a config workaround — strip the header at the first hop with headers: ["Authorization:"] — for anyone stuck between versions. Second, #11771: -search.skipSlowReplicas causes connection churn between vmselect and vmstorage, inflated vmstorage CPU, and cannot process vmselect conn ... write: broken pipe errors — and it is still open as of November 7. The fix sits in the v1.156.0 release-candidate changelog, which means if you run that flag, v1.154.0+ remains broken for you, and the vendor's advice is still to disable the flag or stay on v1.153.0.
Hands-on: the v1.155.0 fix is a breaking change, reproduced before and after
VictoriaMetrics fixed the OOM two ways in v1.155.0. PR #11787 caps how many per-day index scans a single request can start — the general fix for long-range queries. And PR #11802 changes the default time range of /admin/tenants from "entire retention" to "the current day" — a breaking change to a documented admin endpoint, made so the endpoint stops being a storage-killer by default. We ran the checksum-verified v1.155.0 and v1.153.0 cluster binaries — single vmstorage + vminsert + vmselect on localhost, three tenants (a sample now, a sample 27 hours old, a sample ~5 days old) — and probed the endpoint on both versions:
$ sha256sum vm-cluster.tar.gz # v1.155.0 cluster tarball
bfca5a259d6c10754684ff49b155d9235fbfa380f6732fb4710ea8796b464827
# matches victoria-metrics-linux-amd64-v1.155.0-cluster_checksums.txt
$ # three tenants ingested via /insert/<tenant>/prometheus/api/v1/import/prometheus
$ # tenant 5: sample now; tenant 7: 27h ago; tenant 9: ~5 days ago -> all HTTP 204
$ # ---- v1.155.0 vmselect ----
$ curl -s http://vmselect:18481/admin/tenants
{"status":"success","data":["5:0"]}
# ^ current day ONLY: tenants 7 and 9 are invisible
$ curl -s "http://vmselect:18481/admin/tenants?start=0&end=1794083043"
{"status":"success","data":["5:0","7:0","9:0"]}
# ^ explicit time range: all three tenants
$ # ---- v1.153.0 vmselect, same dataset, same probe ----
$ curl -s http://vmselect:28481/admin/tenants
{"status":"success","data":["5:0","7:0","9:0"]}
# ^ the old default: ALL tenants, whole retention
$ # persistence sanity on both versions: instant query at each
$ # sample's own timestamp -> tenant 7 = "7", tenant 9 = "9".
$ # tenant 5 (sample "now") returns EMPTY on both: the 30s
$ # -search.latencyOffset blind spot from the review above,
$ # confirmed here on the cluster query path.
Tenants 7 and 9 vanish from the un-argued endpoint on v1.155.0. That is the point of the change — and the whole risk. The operational sharp edge: vmctl in native VM-to-VM migration mode calls /admin/tenants under the hood to enumerate tenants, which turns the v1.155.0 update note into a migration-data-loss advisory. Run vmctl against a v1.155.0+ cluster without time-range flags, and every tenant with no data today is invisible to the migration — no error, no warning, the tenant simply is not in the list. The cluster docs now state the new contract plainly: "The start and end query args must be specified together. If both are missing, then only the tenants for the current day are returned. In order to get all tenants, specify both start and end so that the resulting time range includes the entire -retentionPeriod." Every vmctl invocation therefore needs both flags pinned so the enumeration covers retention:
./vmctl vm-native \
--vm-native-src-addr=http://vmselect-old:8481 \
--vm-native-dst-addr=http://vmselect-new:8481 \
--vm-native-filter-time-start=2024-01-01T00:00:00Z \
--vm-native-filter-time-end=2026-11-07T23:59:59Z \
--vm-native-step-interval=month
# start/end MUST span the full retention, or tenants with no data
# today are silently skipped by the /admin/tenants enumeration on
# v1.155.0+ (issue #11791). There is no warning; verify tenant
# counts on both sides after every migration.
Two v1.155.0 affordances ride the same fix and are worth wiring into dashboards now: vmselect exports vm_request_duration_seconds{path="/admin/tenants"} — a direct alerting hook on the endpoint that used to OOM your storage — and the bundled alerting rules gained TooManySnapshots (fires when more than two snapshots live for two hours; failed vmbackup/vmbackupmanager runs leave snapshots that pin merged data parts and quietly eat disk).
What else the three releases changed that a platform team should act on
vmui XSS patched (v1.154.0).GHSA-m2x2-9xpq-mmq5 (medium, published October 28): DOM XSS on vmui's Metric Relabeling Debug, Query Analyzer, and Dashboards pages, via a URL-controlled Server URL — the same UI whose tenant discovery triggered the OOM path above. vmui is increasingly exposed to real users in cluster deployments; treat it as attack surface, not a toy.
A cluster hardening detail we hit in our own run:vmstorage binds its internal data-plane listeners — -vminsertAddr (default 0.0.0.0:8400) and -vmselectAddr (default 0.0.0.0:8401) — on all interfaces by default, even when you pin -httpListenAddr to 127.0.0.1. Those internal protocols are unauthenticated. From this update's startup log, verbatim: started TCP vminsert server at "0.0.0.0:8400" and accepting vmselect conns at 0.0.0.0:8401. The review's "front everything with vmauth" advice covers the HTTP surface; on clusters, bind the two internal addrs to a private interface too, or firewall them — they are the write and read paths of your entire metrics database.
HTTP 204 from vminsert is not durability — verified by accident. While wiring the test cluster we initially pointed -storageNode at vmstorage's HTTP port instead of its internal listener — exactly the kind of mistake a 2 a.m. config change makes. The behavior that followed is worth knowing cold: /insert returned HTTP 204 immediately, the rows sat in vminsert's retry loop for seventeen seconds, and on graceful shutdown vminsert logged dropping 3 rows on graceful shutdown, since all the vmstorage nodes are unavailable. During a vmstorage outage your writers stay green while data queues inside the router — and the queue is not guaranteed to survive the router's own shutdown. Monitor vminsert's health of the storage nodes, not just its HTTP status.
The -healthcheck flag (v1.155.0), verified live. Every component now ships a built-in probe: ./vmstorage-prod -healthcheck -httpListenAddr=127.0.0.1:18482 exited 0 against the live listener and 1 against a dead port on our run — which finally makes a Docker HEALTHCHECK possible without curl or wget in the image.
The process_* metrics (v1.155.0), verified live. From our vmselect's /metrics: process_cpu_cores_host 4, process_memory_host_bytes 4984156160, and process_cpu_cores_cgroup_quota -1 / process_memory_cgroup_bytes -1 on an uncapped host. The -1 sentinel means "no cgroup limit found" — the value you will see on hosts without container quotas; these four metrics make RAM- and CPU-sizing dashboards honest on shared nodes.
vmalert state restores, again. The review praised v1.153.0's restore-an-alert-to-firing fix (#11401); v1.155.0 closed the gap it missed (#11574): with Prometheus as -remoteRead.url, the restore query used a MetricsQL-only function Prometheus rejects, so alerting-rule state reset on every vmalert restart. And v1.154.0 fixed the nastier one (#11645): a failed notifier-config reload used to stop the old notifiers before starting new ones, leaving vmalert running with no notifiers — alerts silently undelivered until the file changed or the process restarted.
Two silent-wrong-answer bugs fixed in v1.154.0. Relabeling regexes that start with an alternation (e.g. (a|b).*foo.*) matched more series than intended (#11712) — re-check what your relabeling rules actually matched while the bug was live. And the bundled TooHighChurnRate/TooHighSlowInsertsRate alerting rules had been inactive since v1.108.0 and false-positive-prone since v1.153.0 (#11670) — if you believed those alerts were protecting you, they were not.
Backfill memory leak fixed (v1.154.0, #11672). Storage previously held 8MiB × CPU per month of backfilled data until restart. Anyone doing historical imports on v1.153.0 was paying this without knowing — one more reason the "backfill then watch RSS" step belongs in every import runbook.
Enterprise licensing is changing shape (v1.156.0 changelog, unreleased as of November 7). The release-candidate notes replace verified license keys with a -eula ESA-acceptance flag — "license keys are no longer verified," the -license* flags deprecated and treated as -eula. If the Enterprise tier is in your TCO model — the review's cost section says it is, for downsampling and tiered retention — watch this land: enforcement moves from technical to contractual.
Does the verdict change?
Not structurally. The storage engine, the ingest performance, and the Prometheus-compatibility surface this review measured are untouched, and the incident response itself is a point in VictoriaMetrics' favor: retroactive update notes on the old release, honest "roll back to v1.153.0" advisories, root-cause mechanics published in the issue, and mitigations in fourteen days. But the review's "release hygiene is better than most vendors'" line now carries its asterisk: the two-week cadence has a demonstrated regression tail. v1.154.0 shipped three at once (the index-scan OOM, the chained-vmauth delay, the skipSlowReplicas churn), and one of them (#11771) is still open twenty-six days later. The operating rules that fall out: pin upgrade windows after minor releases and read the update notes before upgrading, not after; keep -search.skipSlowReplicas off until v1.156.0 ships; re-audit every vmctl invocation today for the --vm-native-filter-time-* pair; and treat the internal listener addresses as part of a correct cluster deploy, not an optional bind.
Update — November 12, 2026: the -eula era arrives, Prometheus learns to estimate what VictoriaMetrics refuses, and cardinality gets a price tag
Two things happened since the November 7 update that deserve a commercial treatment rather than a release-notes rewrite: VictoriaMetrics v1.156.0 shipped on November 9, four days after Prometheus v3.16.0 (November 5), and this cycle both projects moved on cost — VictoriaMetrics patched the stream-aggregation machinery that keeps cardinality bills down, while Prometheus shipped a query-cost feature that, by its own documentation, "is never used to reject a query." That sentence is the entire product-category difference between the two stacks, so this update prices it: every claim below was re-verified against the two release notes, both projects' published docs, and VictoriaMetrics' live pricing pages. This pass is desk research plus scripted Decimal math over the vendors' own published numbers — no new benchmark runs were executed for this update, and the hands-on results cited are the ones from this review's September run and the November 7 update.
v1.156.0: Enterprise is now a command-line flag away — legally, anyway
The licensing change the November 7 update told you to watch landed on schedule. From the release notes verbatim: Enterprise components now run with the -eula command-line flag to accept the ESA, "license keys are no longer verified," and the -license, -licenseFile, -licenseFile.reloadInterval and -license.forceOffline flags are deprecated and treated as -eula, so existing setups keep working after the upgrade. The Enterprise docs define what you are actually accepting: -eula permits evaluation use in non-production setups; production use requires a valid enterprise contract with the company; and VictoriaMetrics Cloud is built on top of Enterprise. The LTS lines accept -eula starting from v1.136.23 and v1.148.9.
Read that as an operator: the technical gate is gone. Nothing in the binary stops an Enterprise image from running in production — enforcement moved from cryptographic license-key checks to your contract. Two honest consequences follow. First, evaluation genuinely got cheaper: you can run downsampling, per-tenant rate limiting, and retention filters in staging today without a license-key exchange with sales, which removes the worst procurement surprise — paying for a feature you never validated at load. Second, nothing got cheaper: the features this review told you to price into the TCO from day one still require the contract, and the four commercial tiers (Launchpad, Silver, Gold, Platinum) remain ask-for-pricing only. If compliance is a checkbox in your procurement flow, this change removes friction; if your organization relied on license enforcement to keep uncontracted software out of production, this change quietly made that your problem.
The rest of v1.156.0 is maintenance, but two clusters matter to a cardinality-sensitive deployment. First, the release closes this review's largest open item: issue #11771 — the -search.skipSlowReplicas connection churn that the November 7 update told you to keep disabled — is fixed ("correctly cancel query request if -search.skipSlowReplicas flag is set"). Second, stream aggregation — the front-line tool for killing churn before it reaches storage — got three correctness fixes: aggregation and deduplication flushes no longer get skipped when incoming samples lag by a full interval (the flush delay is now capped at 90% of the interval, #11811); flushes properly delay after sudden ingestion-lag spikes instead of dropping more samples than necessary (#11866); and out-of-order samples arriving around a window boundary no longer produce spikes in rate_sum/rate_avg outputs (#11810). If your cost-control layer aggregates before storage, these were correctness bugs in the layer that protects your bill — they are worth the upgrade on their own.
Smaller items, one line each: vm_cache_mode now exposes the working-set cache mode (0 = split, 1 = switching, 2 = whole) for the storage/tsid, storage/metricIDs, storage/metricName, indexdb/tagFiltersLoops and promql/rollupResult caches (#11860) — the index caches that swell with churn are finally a first-class metric; vmagent gained -remoteWrite.queuePath for per-URL persistent-queue paths (#11662) and Kafka partition-rebalance metrics (#11370); CPU and memory cgroup limits configured via systemd slices are now detected correctly on cgroup v1 hosts — previously only the global host limit was used, which could cause OOM errors and CPU throttling on slice-pinned deployments (#11827); and process_cgroup_cpu_throttled_seconds_total exposes CPU throttling across all components (#10430).
Prometheus v3.16.0: cost estimation, explicitly without enforcement
On the other side of the comparison, Prometheus shipped its answer to the "one dashboard query can ruin your weekend" problem — and the answer is a gauge, not a guard. The feature-flag docs describe --enable-feature=query-cost: it exposes /api/v1/query_cost and /api/v1/query_range_cost, which return a pre-execution estimate of a query's resource cost (series touched and samples read) without running it, and the regular instant/range query endpoints accept a cost=true parameter that adds an estimated-versus-actual comparison to the response. Then the docs say the quiet part loudly: "The estimate is informational only and is never used to reject a query." Queries using info() get a warning because its runtime-dependent additional reads are omitted from the estimate.
Contrast what VictoriaMetrics does at the same decision point — this needs no benchmark, only its documented defaults: -search.maxSamplesPerQuery (default 1B samples), -search.maxSamplesPerSeries (30M), -search.maxPointsPerTimeseries (30k), a 30-second max search timeout, and the adaptive -search.maxUniqueTimeseries this review measured boot-deriving itself to 925,535 on a 30 GB host, explicitly tied to -search.maxConcurrentRequests and remaining RAM. Exceed them and the query is refused — by design, before it can OOM the node. Prometheus 3.16 estimates and tells you; VictoriaMetrics refuses and tells you why. Which behavior you want is a genuine philosophical choice — the Prometheus project has consistently optimized for "never silently drop data or queries," and the same instinct shows on the ingest side: sample_limit, label_limit, label_name_length_limit, label_value_length_limit and target_limit all default to 0 (disabled), and when set, a breach means "the entire scrape will be treated as failed" — you lose every series from that target, not just the offending ones. The default posture is unlimited; the enforcement, once you opt in, is total.
The rest of v3.16.0 stays relevant to this comparison: XOR2 compression improvements (the exact axis VictoriaMetrics differentiates on — Prometheus is iterating on it), fixes for out-of-order sample loss around WAL checkpoints and replay merges (#19737, #19805), smoothed and anchored vector selectors going stable, and an experimental integral() behind --enable-feature=promql-experimental-functions. The standing facts this review compared against are unchanged and were re-verified against the current docs: the local TSDB "is not clustered or replicated" and "should be managed like any other single node database"; the current block of incoming samples is kept in memory with a WAL of 128 MB segments and a minimum of three files; --storage.tsdb.retention.time defaults to 15 days; out_of_order_time_window defaults to 0s (out-of-order samples are rejected unless you opt in); and the LTS policy covers "selected releases" with "selected bug fixes for an extended period of one year." Both projects now publish an LTS policy, note — VictoriaMetrics' runs 12 months per line with a new line every 6 months and is Enterprise-only (currently v1.148.x and v1.136.x); Prometheus' is public but selective. Neither replaces "run current" as the default operating posture.
The cardinality ledger, side by side
PROMETHEUS 3.16 OSS VICTORIAMETRICS 1.156 OSS
=================== =========================
scrape target | vmagent / vmagent stream
| sample_limit, label_limit, | relabeling: drop the label
| target_limit -- all default 0; | BEFORE the series exists;
| breach => ENTIRE SCRAPE FAILS | stream aggregation: churn
v | dies HERE, pre-storage
head block: ALL live series in RAM v
| WAL: 128MB segments x >= 3 vminsert / vmsingle
| head evicts on ~2h compaction | -maxLabelsPerTimeseries,
v | -maxLabelValueLen drop series
blocks on disk | that breach -- silently
| v
PromQL: --enable-feature=query-cost vmstorage: index grows with every
| ESTIMATE ONLY. "never used to | distinct label set; RAM creeps
| reject a query" | with index caches (now
v | visible via vm_cache_mode)
one unbounded query = YOUR problem MetricsQL: ENFORCED -- maxSamples-
PerQuery, adaptive maxUniqueTimeseries,
30s timeout -> the query is REFUSED
Dimension
Prometheus 3.16.0 (OSS)
VictoriaMetrics 1.156.0 (OSS)
VictoriaMetrics Cloud
Cardinality visibility
TSDB status API; no live alerting hook
/api/v1/status/tsdb breakdowns, vmui Cardinality Explorer, bundled TooHighChurnRate alert (actually active since the #11670 fix)
From $200/mo + $0.511/GB/mo storage + $0.09/GB egress
The one piece that belongs to neither vendor's product and works in front of both: vmestimator, VictoriaMetrics' standalone cardinality meter. It taps the same Prometheus remote-write stream your storage sees, estimates cardinality across arbitrary label dimensions in real time, and exposes the results as its own metrics — its docs pitch exactly the failure mode this review reproduced in September ("a team deploys a new version with a trace_id or user_id label. Overnight, that job's cardinality explodes from 500 to 500,000 time series") and promise alerting "within minutes," with "the offending label" named in the alert. The wire has no vendor lock-in — Prometheus RW 1.x in, Prometheus exposition format out — so the honest note is that you can run it in front of a Prometheus you have no intention of migrating.
# vmagent / single-node: aggregate before storage, then drop the raw series
- match: 'http_request_duration_seconds_bucket'
interval: 1m
outputs: [rate_sum]
- match: '{__name__=~".+_total"}'
interval: 5m
outputs: [total]
What 500,000 active series actually cost on VictoriaMetrics Cloud
The managed tier is where this stops being philosophy, because VictoriaMetrics publishes its capacity model. The Cloud product page prices the initial tier at $200/month — 500k active time series, 16.6k datapoints/second, 170 GB daily read volume, business-hours support included — billed monthly on a pay-as-you-go basis with usage metered per hour, a $200-credit trial valid for 30 days, and AWS Marketplace consolidation available (the platform runs entirely on AWS: Frankfurt, Ireland, N. Virginia, Ohio, Singapore; PrivateLink is supported on request, with the AWS-side costs not included). Storage is a separate line item at $0.511 per GB per month, external traffic at $0.09 per GB (the vendor's own estimate of typical traffic cost: $1–$30/month), and API calls cost nothing. There is no public price ladder above the entry tier — higher tiers show "the exact cost of your compute and storage resources" only in-console when you create or resize a deployment.
Run the storage math on the vendor's own planning figure — 0.8 bytes per datapoint, the number the capacity-tier docs use for storage recommendations — and the entry tier at its ingestion cap (16.6k samples/s = 1,434,240,000 samples/day) writes about 1.15 GB/day. Filled to steady state:
Retention
Disk fill
Storage $/month
Compute + storage $/month
30 days
34.42 GB
$17.59
$217.59
90 days
103.27 GB
$52.77
$252.77
180 days
206.53 GB
$105.54
$305.54
365 days
418.80 GB
$214.01
$414.01
Egress adds the vendor-estimated $1–$30 on top. The point of the table: at one year of retention, storage is 51.7% of the bill — the compute tier you picked becomes the smaller line item, and cardinality attacks the bill through both doors at once, because the same docs warn that "high cardinality data increases the data point size, which may incur in needing more storage than recommended." The 0.8-byte figure is an average; churny data averages worse. (VictoriaMetrics' own tier math checks out, for what it is worth: the docs' planning figure of ~1,000 series per node-exporter host at a 30-second scrape gives 500 hosts × 1,000 ÷ 30 = 16,667 samples/s — the published 16.6k datapoints/s to the decimal, and the entry tier works out to $0.40 per host per month of compute for a 500-host estate.)
Now the churn dimension, which is the interesting one. Cloud capacity tiers are defined by four parameters — active time series, churn rate ("number of new active time series in the last 24 hours"), ingestion rate, and data read rate — and the entry tier's churn budget is 1.25M new series per day. Take the vmestimator docs' own failure example — one service adds a trace_id label and goes from 500 to 500,000 series overnight: that single deploy introduces 499,500 new series, or ~40% of the entire daily churn budget of a $200/month deployment; two and a half such deploys exhaust it. Exceed a hard limit and the limits docs are explicit: alerts fire, and then "new metrics eventually being rejected" — the documented hard cap is 200% of the tier ingestion rate, while churn above capacity degrades performance and voids the SLA rather than rejecting outright. And this review's own September run measured the mechanism behind the budget: a 5,000-conceptual-series pod-churn payload produced 259,317 live series — a 51.9× multiplier — with ingest still fast and queries still correct.
Two honest reads of that pricing. In its favor: we are not aware of another major managed metrics platform that prices churn as an explicit, first-class tier dimension — which is unusually honest, because churn (not series count) is what actually kills metrics deployments. Against it: teams whose cardinality is churn-driven — CI-heavy ephemeral environments, per-PR namespaces, pod-per-build pipelines — will discover that the binding constraint on their bill is the churn budget, not the active-series count they sized the tier on. If your estate looks like that, run the churn math against your own cardinality metrics before picking a tier, or stay self-hosted where the constraint is only RAM and disk.
Does the verdict change? (again)
Not structurally, again. v1.156.0 is a maintenance release that closes this review's largest outstanding defect (#11771) and completes a licensing transition that was telegraphed two weeks in advance — that is release hygiene, not news. The scorecard stands: the storage engine is still the best OSS default in its category; still front it with vmauth, still set -deleteAuthKey before day one, still price the Enterprise features before you need them. What the last five days changed is the terms of the comparison, in two specific places:
On cost governance, the two projects now have formally opposite philosophies. Prometheus 3.16 estimates query cost and explicitly refuses to enforce it; VictoriaMetrics enforces and explicitly refuses the query. Neither is wrong — Prometheus protects data and query completeness, VictoriaMetrics protects the node — but if your actual pain is "an unbounded Grafana query can eat a storage box," only one of these stacks treats that as its own problem.
On the Enterprise decision, the friction point moved. Evaluation is now free of license-key mechanics — run the Enterprise image with -eula in staging and validate downsampling and per-tenant rate limiting at load, no sales contact required. Production is now purely a contract question with no published price list (Launchpad/Silver/Gold/Platinum, all ask-for-pricing, SLA ladder from best-effort through 8×5 to 24×7 P1/P2 with 2-hour response, plus LTS releases with backports). If you were waiting for a reason to validate the Enterprise features before the quote call, v1.156.0 removed your last excuse; if you wanted price transparency, it did nothing for you.
Updated skip-list nuance, same standard as the original: teams with static service catalogs and low deploy churn get the fairest deal on Cloud — the churn budget is a formality for them, and $200/month against a 500-host estate is defensible. Teams running ephemeral, CI-heavy Kubernetes estates should size on churn first, or self-host and aggregate aggressively upstream — stream aggregation plus relabeling plus vmestimator in front is the complete OSS cardinality-control stack, and all three pieces are free and work unchanged in front of vanilla Prometheus. And teams under ~500 hosts with 15-day retention needs still have no economic case against vanilla Prometheus: $0 is hard to beat, and Prometheus's new cost-visibility tooling, estimate-only as it is, is a real signal that the project takes the same problem seriously.
References & Further Reading
VictoriaMetrics v1.153.0 release notes — the source for every v1.153.0 change cited here (GHSA-8g4f-32hw-vqf8, GHSA-xxqh-2hcc-9fp6, OIDC SSO, #11252, #11401, #11615, #11618, #11635)