Thanos: The Sidecar That Became the De Facto Way to Scale Prometheus

The problem, stated by the people who had it

The introduction post on Improbable's engineering blog — the one the v0.1.0 README linked as the explanation of the project — opens with the context most vendors leave out: "As you might guess from our flagship product SpatialOS, Improbable has a need for highly dynamic cloud infrastructure at a global scale, running dozens of Kubernetes clusters." Dozens of clusters means dozens of local Prometheus servers, each with a local disk, each blind to the others — and each capped by the retention you can afford to give it. Improbable's answer to that cap was to cut retention to nine days. The post continues: "Additionally, at Improbable, we care about reliability, simplicity and cost. If there are any dependencies, they should be easy to satisfy for small- and large-scale users alike, which also implies a negligible baseline cost." That sentence is the entire design brief. It is why Thanos's architecture has exactly one infrastructure dependency — object storage — and why that dependency was chosen explicitly because "it is extremely cost efficient" and "virtually every object storage solution can be accessed through the well known S3 API."

November 1, 2017: the shipper exists before the name does

The GitHub repo that is now thanos-io/thanos was created on November 1, 2017. Its commit history that week is unusually honest about what the project was, because nothing had been branded yet. The repository-creation commit — "Initial commit" from Improbable's Mat Davies at 17:38 UTC — sits on top of an implementation branch Fabian Reinartz had been pushing since 08:24 that same morning. By 11:21, the central idea of the whole project already existed in code: commit 3a437ef8, "shipper/promlts: add basic block shipper implementation." The working name was promlts — Prometheus Long-Term Storage — and the merge commit that betrays it survives verbatim in git: "Merge branch 'fabxc-init' of github.com:improbable-eng/promlts." By November 3, the other two pillars had landed: the Store gRPC API (PR #17, Reinartz) and the Query API (PR #16, Bartłomiej Płotka), whose commit message promises it will be "compatible with frontends like stock Prom UI or Grafana" — a promise the project has still not broken.

The timing is the detail worth slowing down on. Prometheus 2.0.0 — with its "completely rewritten storage layer, with WAL" — shipped on November 8, 2017, a week after the promlts repo was created and six days after the first block-shipper commit. Thanos was not built on a storage format; it was built on the storage format, during the very week its authors (Reinartz included, a Prometheus maintainer at the center of the 2.0 storage work) were finishing it. The PromCon 2018 abstract is the closest thing to a founding thesis the project ever published: "By building directly on top of the storage format introduced with Prometheus 2.0, Thanos achieves near real-time responsiveness even for cold queries against historical data. All while having virtually no cost overhead beyond that of the underlying object storage."

Three aims, one dependency, no changes to Prometheus

By the time InfoQ covered the open-sourcing on June 9, 2018, the project — now renamed Thanos — had converged on the three aims that still head its README: a global query view across all connected Prometheus servers, unlimited retention, and high availability of the components. InfoQ's interview with Bartłomiej Płotka recorded the mechanics that still define the project today: the sidecar "detects whenever Prometheus writes data to disk, and uploads them to object storage," HA pairs avoid double-uploads because "unique external labels for all Prometheus + sidecar instances" make every label set distinct, and the querier "deduplicates by 'replica' label... on the fly." The first tagged release, v0.1.0, arrived on September 18, 2018 with a component census that reads like a museum piece: "TSDB block upload logic for Sidecar. StoreAPI logic for Sidecar. On-the fly result merge and deduplication logic for Querier. Custom Thanos UI... Optimized object storage fetch logic for Store. Stable support for Google Cloud Storage... Support for S3 minio-based AWS object storage (experimental)." One item on that list was already condemned: "Gossip layer for all components" — shipped, per the release notes, only to have "a stable reference before gossip protocol removal."

The architecture of record: copy blocks, translate nothing

Everything Thanos ever did wrong grew out of a decision it got right. Prometheus 2.0's storage engine writes self-contained blocks — a directory with an index, compressed chunks, and a meta.json manifest — and compacts them into two-hour windows. Thanos's founders looked at that and made the cheapest possible bet: don't parse the blocks, don't transform them, don't even read them. Watch the directory, and when a complete two-hour block appears, copy it to object storage. Then, at query time, serve those same blocks back through a gRPC API and let a plain PromQL engine merge results from every source at once. "Blocks should be stored in native prometheus format" is a stated philosophy in the README; the Improbable post put the operational consequence bluntly — the sidecar "can simply watch Prometheus's data directory and upload new blocks into an object storage bucket as they appear. This simplifies maintenance, cost and system design."

Crisis I: the gossip layer that shipped pre-condemned

The first architecture decision Thanos reversed was its cluster layer, and the reversal is documented with unusual candor. v0.1.0 — the first tagged release, September 2018 — shipped a "Gossip layer for all components," and the release notes say exactly why: it was the "Initial version to have a stable reference before gossip protocol removal." The proposal to delete it (docs/proposals-done/201809, "Deprecated gossip clustering in favor of File SD") had been filed the same month the release went out. The reasoning survives in the design docs: service discovery for store endpoints did not need a peer-to-peer membership protocol — a static file (later, any service-discovery mechanism) did the job with no daemon to maintain. v0.4.0 (May 2019) disabled gossip by default and carried the warning "IMPORTANT: This is the last release that supports gossip." v0.5.0 (June 2019) removed it outright via PR #1008 — and the component started erroring out on any --cluster.* flag rather than silently ignoring them. The outside world had already noticed: Red Hat's December 2018 OpenShift deployment guide lists "the gossip protocol that Thanos uses to find new nodes" as its "first problem" and shrugs it off with "gossip is probably being removed from Thanos anyway." It was, five months later. A cluster protocol, in the trash, eight months after its first release. It is the cleanest example of the project's operating philosophy — the README's own words: "Each subcommand should do one thing and do it well."

Pivot I: Receive — the push path the founders swore they didn't need

The sidecar model has a structural flaw, and the project admitted it in December 2018, three months after v0.1.0. A sidecar can only upload complete blocks, so "the system would always lag the block length behind (typically 2 hours), which would prevent the most common query pattern of Prometheus: real time queries of very recent data." The remote-receive proposal (owner: Frederic Branczyk) argued that some users — "analytics use cases or cases where the data ingestion must be client initiated, such as software as a service type environments" — had no viable pull path at all. So Thanos grew a Receive component: an endpoint that speaks Prometheus's remote-write protocol, buffers samples locally, writes its own two-hour blocks, and ships them to the same bucket as the sidecars would.

What makes the proposal worth reading nine years later is its self-restraint. Receive is "only recommended for uses for whom pushing is the only viable solution." It is "not recommended in order to achieve a global view of data of a single tenant, for those cases the sidecar based approach with layered Thanos queriers is recommended." And the multi-tenancy section carries a warning that belongs in every platform engineer's notebook: enforcement via the prom-label-proxy is offered with the caveat to "thoroughly understand the mechanism if intending to employ this method, as wrong configuration could leak data." By v0.6.0 (July 2019), Receive had "basic hashring support" (PR #1217) and inter-node forwarding with per-tenant headers (PR #1243). By v0.42.0 (July 2026), the release's headline features were Receive's tenant lifecycle handling and per-endpoint TLS — the push path, promoted from concession to flagship. The 2026 reality is that Thanos grew into both shapes: an edge-style sidecar mesh for the pull world, and a Cortex-shaped receiver fleet for the push world, sharing one bucket and one query API.

Crisis II: two households — Cortex, then Mimir, then a fork of a fork

Thanos was born into a rivalry that still defines the long-term-storage market. Cortex (2016, Tom Wilkie and Julius Volz) solved the same three problems — global view, long-term storage, HA — with the opposite data flow: Prometheus's built-in remote-write pushes samples to a central, horizontally-scaled cluster, which stores them in chunks plus a NoSQL index. Grafana's own PromCon recap of the two projects is the fairest statement of the trade-offs, written before the marketing war: with Thanos, "if there is an interruption in this wide-area network, your query performance and availability will suffer" and "in the worst case, the latest 2 hours of data may not be available for querying"; with Cortex, "when an edge location becomes unavailable, all data up to that point is still available centrally" — but the NoSQL index "does tend to lead to a higher TCO for a Cortex-style system than a Thanos-style system."

The mutual respect was not just rhetorical. When Grafana forked Cortex into Mimir in March 2022, Wilkie opened with a concession: "Thanos is an awesome piece of software... I'm a big fan - so much so we used Thanos' storage in Cortex." Cortex had adopted Thanos's bucket-oriented storage years earlier — and in the other direction, Thanos v0.15.0 (September 2020) shipped a Query Frontend described in its own changelog as "based on Cortex Query Frontend." The two projects spent half a decade swapping organs. Even after Mimir's launch, Grafana's CTO-adjacent answers kept the receipts: in the mimir#3380 discussion, Marco Pracucci wrote that Mimir decoupled from Cortex and Thanos — "we're not using Thanos code anymore, except for the object storage clients." The objstore client library — the code that talks to S3, GCS, and Azure — remains, in 2026, the shared bloodstream of both ecosystems.

What Mimir genuinely changed was the scalability ceiling, and it is worth quoting precisely because it is the one argument Thanos does not answer. Prometheus TSDB blocks carry index limits — 64GB per index, 4GB per index section — that cap how much a single tenant's block can hold. Mimir's split-and-merge compactor shards a tenant's data across many blocks to get past that, and Grafana tested it at 1 billion active series for a single tenant. Thanos's answer to the same ceiling has been sharding (proposal 201909) and, in 2026, consistent-hashing shuffle sharding in Receive — plus Bartłomiej Płotka's launch-day framing, which remains the best one-sentence summary of the difference: "eventually it comes to trade-offs we try to make in Thanos between simplicity, reliability and cost vs ultra max performance (Mimir/Cortex)."

Everything in this section was executed for this tale, not quoted from docs: Thanos v0.42.5 and Prometheus 3.16.0 binaries from the official GitHub releases, run on a four-core Linux 6.8 host, with a FILESYSTEM object-store provider (the docs' own escape hatch for testing — the same provider family the docs label "mainly useful for testing and demos"). Block durations were shortened to 15 seconds to make the shipper observable in minutes instead of two hours — which the sidecar immediately, and correctly, complained about:

The upload produced the canonical bucket layout — one directory per block ULID — and thanos tools bucket inspect read back the block metadata model that the whole ecosystem shares. Note the LABELS column: the external labels are stamped into the block metadata by the sidecar, and SOURCE records which component wrote it:

With a Store Gateway and a Querier pointed at both the sidecar's gRPC endpoint and the store's, a PromQL query through the Thanos stack returns what Grafana sees — same API shape as Prometheus, with the external labels merged into every series:

Two honest caveats on this run. First, fifteen-second blocks and a filesystem bucket validate mechanics, not performance — the numbers above say nothing about throughput or latency at scale. Second, the e2e surfaced a live example of how the CLI surface has hardened: the --store flag from every 2019-era tutorial is now a hard error, and the sidecar's block-duration warning arrives with a file:line citation in the log — a project that has been debugging itself in public for nine years.

The scoreboard as of November 9, 2026: 14,270 stars, 5,143 commits, roughly 770 contributors (including anonymous), Apache-2.0, current stable v0.42.5 (October 27, 2026), maintained by nine core maintainers whose MAINTAINERS.md describes their own arrangement with unusual honesty: "We've chosen to maintain (mostly in our free time) Thanos, the de facto way to scale awesome Prometheus." The maintainer roster spans Google, Red Hat (three), Amazon Web Services, Shopify, Cloudflare, Dash0, and Vinted — the community-engine answer to "what happened after Improbable" is that the project outlived its origin company's interest (Improbable exited games in 2022) and now runs on distributed free time plus corporate sponsors' employees.

The patch train deserves its own paragraph because it is the most truthful artifact the project publishes. v0.42.1 (July 16): "fixes a small issue regarding timeouts in the Shipper component in the Receiver - we've accidentally set them too small. Sorry for that!" v0.42.2 (July 17): "Had to do another version release due to broken base image SHAs. No changes." v0.42.3 (July 29): fixes Receive losing its final block on shutdown. v0.42.4 (July 30): "broken base image SHAs" again. v0.42.5 (October 27): "v0.42.4 but with eMudhra CA fix." Five releases in four months, two of them pure container-plumbing re-releases, and the security posture that matters — CVE-2026-33186, a CVSS 9.1 authorization bypass in grpc-go — was fixed upstream of the patch train in v0.42.0 itself, by bumping the project's own grpc-go fork. This is what a mature, free-time-maintained CNCF project looks like from the outside: not a crisis, but not marketing either.

The technical direction is consolidation, not expansion. Native histograms — the Prometheus 3.x data model that replaces fixed buckets with real ones — finished their crawl through the stack: store support in v0.39, and v0.41's upgrade to Prometheus v3.8 turned the old tsdb.enable-native-histograms flag into a no-op because ingestion is simply always on. Batched Series and Query RPCs in v0.41 attacked the querier's network overhead, the same axis Marco Pracucci flagged when explaining Mimir's divergence. Receive — the 2018 push concession — received the v0.42 headline work: tenant lifecycle handling, per-endpoint TLS, consistent-hashing shuffle sharding in v0.41. And the breaking-change list is now mostly deletions of compatibility: v0.42.0 removed the pre-v0.8-era --debug.advertise-compatibility-label, finally breaking queries with Thanos binaries older than 2020. Eight years of forward compatibility, then a clean cut.

Thanos's institutional history is a case study in successful neutralization. Born inside a venture-funded games company, it went to the CNCF Sandbox on July 14, 2019, moved to the thanos-io org between the v0.6.0 and v0.8.0 changelogs, and reached incubating on August 19, 2020 — thirteen months, with the CNCF citing production users Alibaba Cloud, Banzai Cloud, HelloFresh, Monzo, and Red Hat. By the 2024 ThanosCon retrospective the numbers were roughly 600 unique contributors with ~80 active per month and hundreds of adopters. The corporate spread of the nine maintainers — Google (Bartłomiej Płotka, the last founder still leading), Red Hat, AWS, Shopify, Cloudflare, Dash0, Vinted — means no single employer can steer the project, which is precisely the property Grafana Labs chose to abandon by forking Cortex into an AGPLv3 project it controls. Thanos's bet is the opposite: Apache-2.0, CNCF-hosted, boring governance, features at the speed of free time.

The cross-licensing story is the quiet irony of the whole ecosystem. Grafana forked Cortex to build Mimir and stated openly that part of the motive was ensuring "other competitors are not taking advantage of our work for free" — yet Mimir's own maintainers confirm they still run Thanos's object-storage client code, and Cortex before them had adopted Thanos's block-based storage wholesale ("so much so we used Thanos' storage in Cortex," in Wilkie's words). The S3/GCS/Azure plumbing that two competing CNCF ecosystems rely on was written by the Thanos community, under Apache-2.0, for free.

Nine years in, the founding bet has held up almost embarrassingly well. The three aims of the 2018 README — global view, unlimited retention, HA — are still the three aims; the one dependency is still object storage, now with more providers than any competitor (S3, GCS, Azure, Swift, Tencent COS, Baidu BOS, Aliyun OSS, Tencent, OBS, plus FILESYSTEM and NFS "mainly useful for testing and demos"); and the design document's promise of "virtually no cost overhead beyond that of the underlying object storage" is the reason the architecture survived while the projects around it restructured. The two betrayals — killing gossip, adding Receive — were both admissions of error that made the system smaller or more complete respectively. The project's trajectory is consolidation: native histograms done, query protocol hardened to gRPC, Receive productionized, compatibility debt deleted. The risk is not technical: it is that two minors a year, maintained mostly in free time, leaves room for the Mimir side of the market to define what "scale" means while Thanos defines what "sane" means.

Who should skip this

Single-cluster teams with retention needs under a month. Thanos solves the multi-cluster, global-view, multi-year problem. A lone Prometheus with a 30-day local disk — or one backed up by anything that copies the data directory — gets none of the benefit and pays the full tax of sidecars, store gateways, and a compactor. The nine-day retention crisis was a dozens-of-clusters problem.

Single tenants pushing toward a billion active series. The TSDB index limits Mimir's split-and-merge compactor exists to break are real and shared — both projects run on Prometheus blocks. If one tenant's series count dominates your platform, Mimir's tested 1-billion-series path is the honest answer; Thanos's sharding will work harder to deliver less.

Anyone allergic to object storage. The single dependency is still a dependency. If your platform has no S3-compatible story, you will end up on the FILESYSTEM or NFS provider — which the project itself documents as "mainly useful for testing and demos," and operating that in production is a choice no one should make quietly.

Edge fleets that cannot persist two hours of blocks. If your Prometheus pods die before their first block lands, the sidecar model has nothing to ship and you are forced into Receive — which is a push architecture with a cluster to operate. At that point you are choosing push under duress; evaluate the push-first systems on their own merits instead of inheriting one through the back door.

Teams that need a vendor behind the storage layer. Thanos is nine free-time maintainers across seven companies plus contributors. That is governance strength and commercial weakness: no Thanos vendor is obliged to answer your 3 a.m. page, and the 2025–2026 cadence is what free time looks like. If you want a commercial entity on the hook, that product line exists — it is called Grafana Mimir, and it even speaks the same block format.