Every observability vendor sells the same screenshot: a beautiful distributed trace, complete with every span, captured mid-incident. What the screenshot never shows is the pipeline that made it affordable. At realistic microservice volumes — thousands of traces per second, each fanning across a dozen services — keeping every trace means paying your backend for a firehose you will query once per incident. Tail sampling is the industry's answer: hold the full trace briefly, decide after seeing how it ended, keep the 5–10% you will actually debug and drop the rest before it costs you anything.

The tail sampling processor is the mechanism, and on paper it is six lines of YAML. In production it is the most stateful, most misconfigured component in the entire OpenTelemetry Collector ecosystem: it silently splits traces across replicas if you scale it naively, it drops traces under load without an error page to tell you, and its memory footprint is a function of your traffic shape, not your config file. This guide is the operating manual for making it survive contact with a real platform — the two-tier topology the official scaling documentation prescribes, with complete runnable manifests against the freshly shipped v0.161.0 (published September 14, 2026), the sizing math that keeps you out of the circular-buffer cliff, and the Day-2 signals that tell you the sampler is quietly losing data.

TL;DR

  • Adopt if: you run your own collector tier (or an ADOT-style distribution) in front of a trace backend, you pay per GB or per span ingested, and your engineers investigate incidents by trace ID. Tail sampling is the single highest-leverage cost control in the tracing stack.
  • Skip if: your backend already does its own retention-based sampling (Grafana Cloud's trace backend, some APM vendors), your traffic is under ~100 traces/sec (head sampling at the SDK costs nothing and cannot lose data), or you cannot afford a second collector tier — a single-tier tail sampler is a footgun, not a compromise.
  • The one architectural rule: the tail sampling processor requires all spans of a trace on the same collector instance. You satisfy that with a stateless load-balancing gateway tier in front (routing by trace ID), never by putting the sampler behind a plain Kubernetes Service and hoping.
  • The one sizing rule: num_traces must exceed your per-instance in-flight trace count with headroom. At 5,000 traces/sec peak split over three sampler replicas with a 30s decision window, you are already at 50,000 in-flight traces per instance — exactly the default. Default config at that load is zero headroom, which means silent drops the moment traffic skews.
  • What changed recently: the processor gained a second evaluation mode (span-ingest, merged March 2026 via #46762), parallel event loops (num_shards, August 2026 via #48699), and smarter late-span handling (#50623, merged September 17 — two days after contrib v0.161.0). If your runbook predates 2026, it is stale.

The Structural Shift: Sampling Moved Into the Pipeline

Sampling used to be an SDK concern. The OpenTelemetry SDKs ship head samplers — AlwaysOn, TraceIdRatioBased, ParentBased — that decide at span creation whether a trace exists at all. That model is cheap (a rejected span is never serialized), consistent (the decision travels in the W3C tracestate header), and loss-free from the pipeline's perspective: nothing was ever sent.

Its fatal weakness is that the decision is made at the least-informed moment possible: before the request finishes. A head sampler that keeps 5% of traces keeps 5% of your errors too, because at span creation a failing request looks identical to a succeeding one. The only ways to fix that are heroic context propagation tricks or… making the decision later, where "later" means a component that has the whole trace. That component is the collector. The structural shift is that sampling stopped being an application-layer policy and became an infrastructure-layer policy — which is why every platform team that operates a collector fleet now owns a sampling program whether they planned to or not.

Tail sampling's own trade is different and worth stating plainly: the processor buffers every span it receives for the decision window (default 30 seconds) in memory. You are trading backend spend for collector RAM and operational complexity. The rest of this guide is about paying the lowest possible price for that trade.

Head vs. Tail: The Decision Matrix

The collector ships a separate probabilistic sampling processor for the head-style case, and the tailsampling README is unusually direct about when to prefer it: if you are not running tail sampling already, the probabilistic processor is more efficient — it hashes the trace ID and never buffers anything. If you are running tail sampling, add probabilistic keep-rate as another policy inside it instead of stacking both processors, so traces kept by other policies are not double-eligible for dropping.

DimensionHead (SDK / probabilistic processor)Tail (tailsampling processor)
Decision pointSpan creation / first collector hopAfter decision window (default 30s), full trace in view
Can keep 100% of errorsNo — errors are invisible at decision timeYes — status_code policy
Can keep 100% of latency outliersNoYes — latency policy on end-to-end duration
Memory costNone (hash on trace ID)Every span buffered for the window
Failure modeUndersampling (irritating, visible)Silent trace fragmentation / drops under load (dangerous, invisible without metrics)
Scaling modelStateless — any replica, any trafficStateful — all spans per trace must land on one instance
Adjusted counts (statistical correction)Native (probability in tracestate)Native only with the usetracestate gate (alpha)
Sweet spotBaseline keep-rate at high volumeKeep-the-interesting-stuff at moderate volume, with errors always kept

The production pattern that most teams converge on is a hybrid: a small head-sampled baseline (so you always have some representative traffic for capacity math) plus tail policies that keep 100% of errors, 100% of slow traces, and force-keep anything an engineer flags mid-incident. That hybrid is exactly what we configure below.

Architecture: Two Tiers, Not One

The official scaling docs classify the tail sampling processor as a stateful component — it cannot simply be scaled by adding replicas behind a load balancer, because different replicas would see different fragments of the same trace, each making its own (possibly conflicting) decision. The result is traces with missing spans that misrepresent what actually happened — worse than no trace at all, because it lies to you.

The prescribed topology is two layers with different jobs:

  apps (OTLP exporters, SDKs)          platform: observability namespace
 ┌─────────────────────┐
 │  service-a  svc-b … │  OTLP/gRPC 4317
 └─────────┬───────────┘
           │ (any k8s Service — these hops are stateless)
           ▼
 ┌─────────────────────────────────────────────────────────────┐
 │  TIER 1: gateway collectors (Deployment, 2+ replicas)       │
 │  stateless: receive → load_balancing exporter               │
 │  consistent-hash on traceID → one otlp exporter per backend │
 └───────┬──────────────────┬──────────────────┬───────────────┘
         │ OTLP             │ OTLP             │ OTLP
         ▼                  ▼                  ▼
 ┌──────────────┐  ┌──────────────┐   ┌──────────────┐
 │ sampler-0    │  │ sampler-1    │   │ sampler-2    │  TIER 2: sampler
 │ stateful:    │  │              │   │              │  collectors
 │ tail_sampling│  │ tail_sampling│   │ tail_sampling│  (StatefulSet,
 │ memory_limit │  │              │   │              │   stable pods)
 │ batch        │  │              │   │              │
 └──────┬───────┘  └──────┬───────┘   └──────┬───────┘
        │  kept spans (errors, slow, baseline)
        ▼ (otlp exporter → backend, e.g. Grafana Cloud OTLP endpoint)
   ┌──────────────────┐
   │  trace backend   │   ~5-10% of raw volume
   └──────────────────┘