Cloudflare's 100TB Hash-Ring Fix: The Cost of 'Just Add More Hashes'

September 19, 2026. Cloudflare published a deep-dive on Friday that should be mandatory reading for anyone running a proxy fleet: Saving another 100TB of RAM with math (and Rust). An engineer filed a ticket that its internal load balancer — Pingora Backend Router (PBR), the service that routes cacheable requests to origin servers by URL — was using far more memory than expected. The root cause was the most boring possible suspect: the consistent-hash ring was too big. After a struct-layout fix and a 90% point-count reduction, PBR's memory footprint dropped by more than 100TB across the fleet, on top of the 100TB the DNS team freed in August by restructuring 1.1.1.1's cache entries.

Two aspects make this story worth a Platform Monkey teardown rather than a link drop. First, the failure mode is universal: the 160-points-per-weight-unit default that Cloudflare, nginx, and half the ecosystem inherited from the original ketama implementation is not a load-balancing number — it is a memory number, and almost nobody measures it. Second, the fix shipped as an upstreamable open-source change in the pingora-ketama crate (a v2 ring behind the unadvertised v2 cargo feature, available in 0.9.0), which means the remedy for "my consistent-hash ring is secretly a 6GB allocation" is now a dependency bump away for anyone running Pingora-based proxies.

What follows is the operator's teardown: the math that justifies how few hashes you actually need, the Rust struct-layout traps that made the first fix work, the dual-ring migration that avoided a global cache invalidation, and the numbers you should plug into your own Envoy, HAProxy, or Pingora configs before your next capacity review. If your takeaway is anything other than "I should go count the points on my rings," the article has failed.

The Setup: Where the Memory Actually Went

PBR routes cacheable requests to cache servers using consistent hashing on the URL. The property that makes consistent hashing useful for caches — add or remove a server and only ~1/N of keys remap — is bought by hashing each server onto the ring many times. In the ketama lineage that nginx inherited and pingora-ketama ports, the number of points per server is weight × 160, where weight is proportional to the server's disk capacity. The constant is literally named in the source:

/// This constant is copied from nginx. It will create 160 points per weight
/// unit. For example, a weight of 2 will create 320 points on the ring.
pub const DEFAULT_POINT_MULTIPLE: u32 = 160;

Then the multiplier you didn't choose: PBR runs not one ring but dozens, because cache features and compliance constraints mean different request classes can only land on server subsets. Feature combinations explode into 2^handful rings, each paying full point tax. The post's numbers: weight factor ~625 per server (disk-proportional), so k = 160 × 625 = 100,000 points per server, across ~2,048 servers per data center, across dozens of rings, across every data center on Earth. The ticket that started it all: 6GB of pure hash points in a single PBR replica.

  servers per DC ............ ~2,048     ─┐
  weight per server (disk) ... 625         │ ──▶ k = 160 × 625
  DEFAULT_POINT_MULTIPLE ..... 160       ─┘      = 100,000 points/server
                                                    × 2,048 servers
  ┌─────────────────────────────────────┐         = 204.8M points/ring
  │  ONE ring = ~204,800,000 points      │
  │  at 8 bytes/point (PointV1)          │         = ~1.6 GB / ring
  └─────────────────────────────────────┘
  feature-subset rings ........ dozens   ──▶  ~6 GB of points / replica
  data centers ............... global   ──▶  100+ TB of fleet RAM

Scale hides waste better than anything else in systems engineering. A 6GB allocation is invisible per node and completely ordinary in a fleet profile — until the day someone multiplies it by the fleet. The Performance team's ticket-to-100TB arc is the reminder that "ordinary" and "justified" are different words: the 160× multiplier survived years of review because nobody had derived what it was buying.

The Math: How Many Hashes Does a Ring Actually Need?

Here is the actual decision-driving content of the post, and it's better than most vendor deep-dives because the authors derived the exact standard deviation for k hashes per server rather than quoting an approximation:

Exp(k) = 1/N                                    — expected share per server
SD(k)  = (1/N) · sqrt( (N−1) / (k·N+1) )       — exact for k hashes/server
CV(k)  = SD(k)/Exp(k) = sqrt( (N−1) / (N·k+1) ) — imbalance as fraction of fair share

(At k=1 this reduces to the classic result SD = (1/N)·sqrt((N−1)/(N+1)).)

The coefficient of variation (CV) is the number to internalize: it's the expected load imbalance across your servers as a fraction of fair share. We recomputed the post's table from the published formula so you can see the shape of the curve without opening a spreadsheet:

Read the last two rows carefully, because this is the post's core insight: CV falls off as 1/√k, so every order-of-magnitude in point count buys only a ~3× reduction in imbalance. Going from 10,000 points to 100,000 points — a 10× memory cost — improved the error margin by 0.7 percentage points. The last 90% of the memory was buying nothing.

And then the collision tax: the post's math assumes a continuous ring, but real rings hash to 32-bit integers. At Cloudflare's scale, collisions arrive via the birthday paradox: with N·k total points, expected colliding pairs ≈ (N·k)²/2³³. We computed what that means at Cloudflare's scale:

This is the part of the story that should make a platform engineer sit up. The conventional wisdom — "more points, more accurate" — isn't just diminishing returns, it's negative returns past a threshold: at 204.8M points in a 2³² hash space you expect ~4.9M colliding pairs, and each collision silently drops a point's contribution to its server's range. You pay 10× the memory to make accuracy worse than the k=10,000 ring would deliver. The remedy Cloudflare chose: cut the point count by 90% (k: 100,000 → 10,000), keeping CV ≈ 1% while shrinking every ring by an order of magnitude.

The Fix, Part 1: Rust Struct Layout Is a Capacity Decision

The first fix predates the point-count reduction and doesn't require any math. The old point representation in pingora-ketama is:

struct PointV1 {
    node: u32,
    hash: u32,
}

Four bytes of hash, four bytes of node index. The hash is irreducible — it's a 32-bit CRC-based value, that's the algorithm — but the node index is grotesque overkill: PBR will never coordinate 2³² cache servers. A u16 indexes 65,536 servers; the post says 2¹⁶ ≈ 65k is a comfortable ceiling. The obvious change —

struct PointV2 {
    node: u16,
    hash: u32,
}

— is a no-op in Rust, and the post deserves credit for explaining why instead of skipping it: Rust aligns struct fields, and the size of a struct must be a multiple of its most-aligned field. With a u32 present, PointV2 still occupies 8 bytes — two bytes are simply wasted on padding. You cannot shrink a struct below its alignment by reordering fields. The working trick is to store the point as a raw byte array and decode fields through getters, so the compiler can't insert padding:

#[cfg(feature = "v2")]
struct PointV2([u8; 6]);

#[cfg(feature = "v2")]
impl PointV2 {
    fn new(node: u16, hash: u32) -> Self { ... }

    fn hash(&self) -> u32 {
        u32::from_ne_bytes(self.0[0..4].try_into().expect("There are exactly 4 bytes"))
    }

    fn node(&self) -> u16 {
        u16::from_ne_bytes(self.0[4..6].try_into().expect("There are exactly 2 bytes"))
    }
}

Six bytes per point instead of eight: an immediate 25% reduction in point memory with zero algorithmic change. Two engineering notes for anyone porting the idea:

• #[repr(packed)] would also work but is controversial for good reasons — packed structs create unaligned references, which are undefined behavior in Rust when taken. Cloudflare's byte-array pattern compiles to the same loads and stores without the UB footgun, at the cost of getter noise.
• Endianness is load-bearing. The code uses from_ne_bytes (native-endian) because the ring is process-local: it is never serialized. If you ever persist a ring or share it between processes on mixed architectures, switch to from_le_bytes/from_be_bytes and version the format, or you'll have a cross-platform data corruption bug that only manifests on your ARM fleet.

The Fix, Part 2: A v2 Ring That Can Run Beside the v1 Ring

The deeper fix required a new ring type, because point count is no longer a constant — it's a parameter. The crate models this as a versioned ring, with the v2 point count passed explicitly:

#[derive(Copy, Clone, Debug, Eq, PartialEq, Default)]
pub enum Version {
    #[default]
    V1,
    #[cfg(feature = "v2")]
    V2 { point_multiple: u32 },
}

Version::V2 { point_multiple } takes the point multiplier as a field, so a caller can build a 160× ring, a 16× ring, or whatever its math says — the library stops hardcoding the decision. The construction path multiplies each bucket's weight by the version's point_multiple, and the sort changes too: v2 rings use i_key_sort's OneKeyAndCmpSort (the dependency the v2 cargo feature pulls in) for a faster single-key sort with a secondary comparison for collision dedup, replacing v1's generic sort_unstable.

The public API is otherwise unchanged, which is the entire point of the design. Callers build the same Bucket list and get the same Continuum; choosing v2 is one enum variant:

use pingora_ketama::{Bucket, Continuum, Version};

// v1 (default): 160 points per weight unit — identical to what the crate
// has always shipped, byte-for-byte compatible with nginx/memcached ketama.
let ring_v1 = Continuum::new(&buckets);

// v2: compact 6-byte points + your own point multiplier.
// point_multiple: 16 = the ~90% reduction Cloudflare used (160 → 16).
let ring_v2 = Continuum::new_with_version(
    &buckets,
    Version::V2 { point_multiple: 16 },
);

Two details in the public API are worth flagging before you adopt:

• Hash compatibility is preserved on purpose. The ring hash input is still the legacy nginx/memcached-compatible <ip>\0<port> CRC32 chain (the source carries a TODO to remove it once backward compatibility is no longer needed). Changing point counts moves which servers own which requests; changing the hash input would move everything. Cloudflare changed only the former — the safe subset of the design space.
• The v2 sort dedups by hash only. v1's dedup_by compares full points; v2 compares a.0[0..4] == b.0[0..4] — the hash bytes only. Collisions in the node field would be a correctness bug; collisions in the hash field are the birthday-paradox tax discussed above, now bounded because the ring is 10× smaller.

The Migration: How Not to Invalidate Every Cache on Earth

Here is the operationally hardest part of the story, and the one most teams would get wrong. Changing the hash ring changes which backend server receives each cacheable URL. Flip the ring globally and you effectively invalidate almost every cached object on Earth at once — a memory optimization becomes an apocalyptic origin-traffic incident. The post describes a migration design that is textbook for any ring/hash/topology change:

                    ┌──────────────────────────────────────────┐
                    │              PBR replica                │
  request ────────▶ │                                          │
                    │  migration framework (per-request):      │
                    │    request hash ─▶ stable ring choice    │
                    │                                          │
                    │   ┌──────────────────┐  ┌─────────────┐ │      ┌────────────┐
                    │   │ OLD ketama ring  │  │ NEW v2 ring │ ──────▶ │ cache      │
                    │   │ (v1, 100k pts)   │  │ (16k pts)   │  …────▶ │ servers    │
                    │   └──────────────────┘  └─────────────┘ │      └────────────┘
                    └──────────────────────────────────────────┘
  Rollout: small validation DCs ─▶ progressively larger DC groups ─▶ global
  Controls: (1) % of traffic on new ring  (2) WHICH DCs move — independent axes
  Rollback: send requests back through the old ring, no redeploy needed
  Signals: backend-selection traces, ring-version counters, PBR connection
           errors, process memory, startup time, cache behavior, origin traffic

Four properties make this migration the pattern to copy, and none of them are about consistent hashing:

1. Both rings live in memory during transition. Cost: during the migration, PBR temporarily carried both the old 100k-point rings and the new compact ones — the memory win is realized only at the end, not during the rollout. This is the correct trade: paying peak memory temporarily to avoid any hard cutover.
2. Stable per-request ring selection. The migration framework decides ring choice by request hash, so a given URL consistently uses one ring during transition — no flip-flopping cache entries, no thundering re-hash.
3. Two independent rollout axes. Traffic percentage and data-center scope are controlled separately. A plain "10% → 50% → 100% of requests globally" would have spread cache churn everywhere simultaneously; DC-scoped rollout kept the blast radius small enough to attribute cause. If you learn one operational lesson from this post, make it this one — percentage rollouts are the wrong tool for anything that changes data placement.
4. Rollback without redeploy. Because both rings coexisted behind the framework, reversal was a routing decision, not a build-and-ship cycle. The post notes the old-ring path was removed only after the migration hit 100%.

The payoff chart in the post shows PBR's memory the week of the change minus a few weeks prior: a sharp drop on the day the old-ring code path was decommissioned. The difference: 100TB. Not by buying fewer servers — by deleting a constant nobody had re-derived in a decade.

What This Looks Like at Your Scale

Cloudflare numbers can be alienating — petabytes of RAM make 6GB seem like rounding error. So here is the same arithmetic at scales platform teams actually operate. The formula: ring bytes ≈ servers × weight × point_multiple × bytes_per_point (then multiply by however many feature-subset rings you run — the multiplier almost everyone forgets):

The top rows make the point that this is not a Cloudflare-only problem: at small scale the waste is kilobytes and irrelevant. The pattern turns into real money at the middle row — hundreds of MB per proxy process, multiplied by replica count — and fleet-defining at the bottom. And because CV falls as 1/√k, accuracy is a real trade: at N=200, k=160 gives CV 7.9% while k=16 gives CV 24.9%. If ~25% worst-server imbalance is unacceptable for your tier — and for a cache tier it usually is — then 16× is the wrong multiple and the honest answer is somewhere between: pick k from your imbalance SLO, not from a 2010-vintage constant. That is the entire argument of the v2 design: point count is now a parameter, not a legacy constant.

One more trap the story quietly illustrates: weights multiply the problem. Cloudflare's 625× weight factor pushed per-server point counts to 100k. If your load balancer takes per-server weights from capacity data (disk GB, CPU count), your ring size is coupled to your hardware refresh cycle — every storage-dense generation silently grows the ring. A 4TB-node fleet migrating to 16TB nodes quadruples point memory without anyone touching a config file. Ring size belongs in capacity reviews, not in "it came with the defaults."

The Ecosystem Context: You Are Probably Shipping 160× Too

Cloudflare is not an outlier; the 160 default is everywhere, and few operators can state what it costs them:

HAProxy numbers for the table: the default hash-map is 50,079 slots and the docs note it stays small enough to fit in cache lines. Envoy ring_hash defaults are from current Envoy docs (min 1024 / max 8M). The takeaway across all rows: every implementation ships a hash-density default tuned for the 2010s, and none of them documents the memory cost in capacity terms. If you run a large Envoy fleet with weighted hosts, do the arithmetic on minimum_ring_size before your next capacity review — same exercise, different constant.

The Skeptic's Checklist: What to Verify Before Crediting the Story

We verified the load-bearing claims against the shipped source; here's the receipt trail so you can re-run it:

• "160 is copied from nginx" — verbatim in pingora-ketama/src/lib.rs: pub const DEFAULT_POINT_MULTIPLE: u32 = 160; with the comment "This constant is copied from nginx."
• "The v2 ring shipped in the crate" — Cargo.toml declares [features] v2 = ["i_key_sort"]; the crate's max version is 0.9.0 (released Sep 9, 2026), and the v2 structs, Version enum, and Continuum::new_with_version are all present in 0.9.0's src/lib.rs. The blog calls it "a (for now) unadvertised cargo feature" — accurate; there's no changelog entry yet, so pin your dependency and read the source when adopting.
• "6-byte points, 25% saving" — PointV2([u8; 6]) with from_ne_bytes getters, exactly as described.
• "90% point reduction without appreciable error" — consistent with CV = √((N−1)/(N·k+1)): at N=2,048, k=100,000 gives CV 0.32%, k=10,000 gives CV 1.0%. Both are far below any cache-tier imbalance SLO we've seen written down. The 0.7-point claim in the post refers to the last-90k-hashes asymptote at their N.
• "100TB reclaimed" — the post's own difference-in-memory chart, measured fleet-wide; treat it as Cloudflare's reported number, not something reproducible outside their telemetry.
• "PBR = Pingora Backend Router" — the post's name for its internal cache-routing service built on the open-source Pingora framework, which Cloudflare has described in detail in prior posts (saving compute 1% at a time).

What we could NOT independently verify: per-DC ring counts, exact fleet RAM totals, and the migration timeline — internal details only the post attests. That's normal for vendor engineering blogs; the difference is this one shipped its fix in a public crate, which moves it from "trust me" to "cargo add and inspect."

Who Should Care, and Who Should Skip

Care if: you operate a proxy or load-balancing tier where consistent hashing meets weighted servers — CDN/edge caches, object-cache routers, distributed rate-limit sharders, session-affinity tiers. Care doubly if weights derive from hardware capacity (disk/CPU), because your ring grows with every hardware generation. Care triply if you run multiple feature- or compliance-subset rings; the combinatorial multiplier is where single-digit-MB-per-ring becomes GB-per-process.

Skip if: your load-balancing tier is small (dozens of servers, uniform weights) — your ring is measured in kilobytes and this entire conversation is a rounding error. Skip if your "consistent hashing" is actually Envoy Maglev or HAProxy map-based, whose flat table sizes already bound memory. Skip if you have no per-request class constraints forcing multiple rings — one modest ring is fine at 160×.

The platform-engineering takeaway isn't "copy Cloudflare." It's that the most expensive line of code in your infrastructure is probably a default constant that predates your employment, was tuned for someone else's fleet, and has never been re-derived against your SLOs. Cloudflare's Performance team turned a ticket into 100TB by doing the arithmetic nobody had done since the constant was inherited. Your fleet has the same kind of constants. Go find yours — and when you do, the pingora-ketama v2 ring is a working reference implementation of what "fixed" looks like: smaller points, fewer of them, and the count finally, deliberately chosen.