OpenTelemetry Tail Sampling in Production: The Two-Tier Architecture That Actually Survives Load

Every observability vendor sells the same screenshot: a beautiful distributed trace, complete with every span, captured mid-incident. What the screenshot never shows is the pipeline that made it affordable. At realistic microservice volumes — thousands of traces per second, each fanning across a dozen services — keeping every trace means paying your backend for a firehose you will query once per incident. Tail sampling is the industry's answer: hold the full trace briefly, decide after seeing how it ended, keep the 5–10% you will actually debug and drop the rest before it costs you anything.

The tail sampling processor is the mechanism, and on paper it is six lines of YAML. In production it is the most stateful, most misconfigured component in the entire OpenTelemetry Collector ecosystem: it silently splits traces across replicas if you scale it naively, it drops traces under load without an error page to tell you, and its memory footprint is a function of your traffic shape, not your config file. This guide is the operating manual for making it survive contact with a real platform — the two-tier topology the official scaling documentation prescribes, with complete runnable manifests against the freshly shipped v0.161.0 (published September 14, 2026), the sizing math that keeps you out of the circular-buffer cliff, and the Day-2 signals that tell you the sampler is quietly losing data.

TL;DR

• Adopt if: you run your own collector tier (or an ADOT-style distribution) in front of a trace backend, you pay per GB or per span ingested, and your engineers investigate incidents by trace ID. Tail sampling is the single highest-leverage cost control in the tracing stack.
• Skip if: your backend already does its own retention-based sampling (Grafana Cloud's trace backend, some APM vendors), your traffic is under ~100 traces/sec (head sampling at the SDK costs nothing and cannot lose data), or you cannot afford a second collector tier — a single-tier tail sampler is a footgun, not a compromise.
• The one architectural rule: the tail sampling processor requires all spans of a trace on the same collector instance. You satisfy that with a stateless load-balancing gateway tier in front (routing by trace ID), never by putting the sampler behind a plain Kubernetes Service and hoping.
• The one sizing rule: num_traces must exceed your per-instance in-flight trace count with headroom. At 5,000 traces/sec peak split over three sampler replicas with a 30s decision window, you are already at 50,000 in-flight traces per instance — exactly the default. Default config at that load is zero headroom, which means silent drops the moment traffic skews.
• What changed recently: the processor gained a second evaluation mode (span-ingest, merged March 2026 via #46762), parallel event loops (num_shards, August 2026 via #48699), and smarter late-span handling (#50623, merged September 17 — two days after contrib v0.161.0). If your runbook predates 2026, it is stale.

The Structural Shift: Sampling Moved Into the Pipeline

Sampling used to be an SDK concern. The OpenTelemetry SDKs ship head samplers — AlwaysOn, TraceIdRatioBased, ParentBased — that decide at span creation whether a trace exists at all. That model is cheap (a rejected span is never serialized), consistent (the decision travels in the W3C tracestate header), and loss-free from the pipeline's perspective: nothing was ever sent.

Its fatal weakness is that the decision is made at the least-informed moment possible: before the request finishes. A head sampler that keeps 5% of traces keeps 5% of your errors too, because at span creation a failing request looks identical to a succeeding one. The only ways to fix that are heroic context propagation tricks or… making the decision later, where "later" means a component that has the whole trace. That component is the collector. The structural shift is that sampling stopped being an application-layer policy and became an infrastructure-layer policy — which is why every platform team that operates a collector fleet now owns a sampling program whether they planned to or not.

Tail sampling's own trade is different and worth stating plainly: the processor buffers every span it receives for the decision window (default 30 seconds) in memory. You are trading backend spend for collector RAM and operational complexity. The rest of this guide is about paying the lowest possible price for that trade.

Head vs. Tail: The Decision Matrix

The collector ships a separate probabilistic sampling processor for the head-style case, and the tailsampling README is unusually direct about when to prefer it: if you are not running tail sampling already, the probabilistic processor is more efficient — it hashes the trace ID and never buffers anything. If you are running tail sampling, add probabilistic keep-rate as another policy inside it instead of stacking both processors, so traces kept by other policies are not double-eligible for dropping.

The production pattern that most teams converge on is a hybrid: a small head-sampled baseline (so you always have some representative traffic for capacity math) plus tail policies that keep 100% of errors, 100% of slow traces, and force-keep anything an engineer flags mid-incident. That hybrid is exactly what we configure below.

Architecture: Two Tiers, Not One

The official scaling docs classify the tail sampling processor as a stateful component — it cannot simply be scaled by adding replicas behind a load balancer, because different replicas would see different fragments of the same trace, each making its own (possibly conflicting) decision. The result is traces with missing spans that misrepresent what actually happened — worse than no trace at all, because it lies to you.

The prescribed topology is two layers with different jobs:

  apps (OTLP exporters, SDKs)          platform: observability namespace
 ┌─────────────────────┐
 │  service-a  svc-b … │  OTLP/gRPC 4317
 └─────────┬───────────┘
           │ (any k8s Service — these hops are stateless)
           ▼
 ┌─────────────────────────────────────────────────────────────┐
 │  TIER 1: gateway collectors (Deployment, 2+ replicas)       │
 │  stateless: receive → load_balancing exporter               │
 │  consistent-hash on traceID → one otlp exporter per backend │
 └───────┬──────────────────┬──────────────────┬───────────────┘
         │ OTLP             │ OTLP             │ OTLP
         ▼                  ▼                  ▼
 ┌──────────────┐  ┌──────────────┐   ┌──────────────┐
 │ sampler-0    │  │ sampler-1    │   │ sampler-2    │  TIER 2: sampler
 │ stateful:    │  │              │   │              │  collectors
 │ tail_sampling│  │ tail_sampling│   │ tail_sampling│  (StatefulSet,
 │ memory_limit │  │              │   │              │   stable pods)
 │ batch        │  │              │   │              │
 └──────┬───────┘  └──────┬───────┘   └──────┬───────┘
        │  kept spans (errors, slow, baseline)
        ▼ (otlp exporter → backend, e.g. Grafana Cloud OTLP endpoint)
   ┌──────────────────┐
   │  trace backend   │   ~5-10% of raw volume
   └──────────────────┘

Tier 1 exists only to make trace-ID affinity someone else's problem: the load balancing exporter consistently hashes the trace ID and maintains one OTLP exporter per backend endpoint, so every span of a trace lands on the same tier-2 instance no matter which gateway received it. Tier 2 does the buffering, the deciding, and the exporting of survivors. The scaling doc's stated reason for separating the tiers (rather than running both pipelines in one collector) is failure isolation: a sampler restart under memory pressure must not take down your ingestion edge, and an ingestion edge rolling deployment must not reset every in-flight sampling decision at once.

The routing contract and its failure windows, straight from the exporter's own documentation:

• Default routing: traceID for traces (the trace ID hash picks the backend; spans of one trace always co-locate), service for logs and metrics. Other routing keys exist (resource, streamID, metric, attributes) but for the sampler tier you want trace-ID affinity, full stop.
• Topology changes rehash routes: when the backend list changes (sampler scaled 3→4), roughly R/N of existing routes move to a different backend — ~25% of in-flight routes on a 3→4 scale event, ~14% on 6→7. During that window, spans of an affected trace can split across two samplers. Larger backend counts dilute the blast; groupbytrace in front of the exporter makes traces dispatch atomically if you need that guarantee.
• Two levels of queueing, asymmetric defaults: each per-backend sub-exporter has its own queue/retry/timeout (resiliency "options 2", enabled by default), while the load-balancing exporter's own top-level queue/retry (resiliency "options 1") is disabled by default. Translation: by default, a dead backend's data exhausts the sub-exporter retry budget and is dropped rather than re-routed to a healthy peer. In an elastic Kubernetes tier you almost always want options 1 enabled too.

The Sizing Math (Do This Before Writing YAML)

Every number below is derived from the same worked example we deploy later: 5,000 new traces/sec peak, decision window 30s, average 15 spans/trace, ~1.2 KiB marshaled per span, three sampler replicas. The arithmetic is mechanical — and it is the difference between a sampler that survives Black Friday and one that silently sheds errors.

in-flight traces (cluster)  = 5,000/s × 30s        = 150,000
in-flight traces (per pod)  = 150,000 / 3          =  50,000
default num_traces          =                      =  50,000   ← zero headroom
recommended num_traces      = 2× in-flight         = 100,000   ← per pod

span payload in flight/pod = 50,000 × 15 × 1.2 KiB ≈ 858 MiB  ← payload only,
                                                            before batch buffers,
                                                            decision caches, queues
decision cache entries     = 1,667/s × 60s retention ≈ 100,000 per pod

route rehash on scale 3→4  = 1/4                  ≈  25% of live routes
route rehash on scale 6→7  = 1/7                  ≈  14% of live routes

post-sampling export volume (8% keep, kept traces ~1.4× avg size)
                            = 5,000 × 0.08 × 15 × 1.4 ≈ 8,400 spans/s to backend

Three consequences fall out of this arithmetic, and each one is a production incident waiting somewhere:

• The default num_traces is a trap at exactly this scale. 50,000 is the documented default. At 5k traces/sec over three replicas you sit exactly at it. The processor uses a circular buffer capped at num_traces: when a new trace arrives and the buffer is full, the oldest trace is evicted — and an evicted trace is dropped before its sampling decision, silently, even if it was a 5-second error trace one second from being kept. The README's own FAQ attributes high sampling_trace_dropped_too_early values to exactly this condition. Size for 2× headroom and alert on the metric (below).
• Memory is driven by traffic shape, not config. The ~858 MiB figure is payload for the decision window alone. Add the decision caches (configurable, sized ~100k entries at this rate), the batch processor's buffers, and exporter queues, and a "1 GiB collector" is already underwater. We give the sampler tier 2 GiB and set the memory_limiter hard limit at 1,792 MiB with a ~358 MiB spike allowance (the documented ~20% guidance) — soft limit ~1,434 MiB.
• Export volume needs its own sizing. ~8,400 spans/s surviving to the backend sounds modest until the backend hiccups: with the default queue of 1,000 requests and ~500 spans per batch, a full queue drains in ~250s at a 20k spans/s backend — but if the backend is down, the retry budget (5s→30s backoff, 300s elapsed cap by default) expires first and data is dropped. The exporter queue metrics below are your early warning.

Tier 1: The Load-Balancing Gateway

Everything in this section is deployed with the OpenTelemetry Operator (CRD opentelemetry.io/v1beta1), which is also how you get the headless service and stable pod addressing the k8s resolver wants. The gateway tier is a plain deployment — it is stateless and can scale behind any Service.

First the RBAC. The exporter's k8s resolver watches discovery.k8s.io/v1 EndpointSlices in the namespace of the backend service; without get, list, and watch the resolver cache stays empty and every export fails with couldn't find the exporter for the endpoint "" — the documented failure signature:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-gateway
  namespace: observability
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: otel-gateway-endpointslices
  namespace: observability
rules:
- apiGroups: ["discovery.k8s.io"]
  resources: ["endpointslices"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: otel-gateway-endpointslices
  namespace: observability
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: otel-gateway-endpointslices
subjects:
- kind: ServiceAccount
  name: otel-gateway
  namespace: observability

Then the gateway itself. Two details matter more than everything else in this manifest: the k8s resolver pointing at the sampler tier's headless service (the operator names it <cr-name>-collector-headless), and resiliency options 1 explicitly enabled at the exporter level so that a dead sampler pod's data is re-routed after retry exhaustion instead of dropped:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel-gateway
  namespace: observability
spec:
  mode: deployment
  replicas: 2
  serviceAccount: otel-gateway
  image: docker.io/otel/opentelemetry-collector-contrib:0.161.0
  resources:
    limits:
      memory: 512Mi
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318

    processors:
      # minimal hygiene at the edge; heavy work belongs on tier 2
      memory_limiter:
        check_interval: 1s
        limit_mib: 384

    exporters:
      load_balancing:
        routing_key: traceID
        # resiliency options 1: exporter-level queue+retry, OFF by default.
        # enabled here so a dead sampler endpoint re-routes instead of dropping
        timeout: 10s
        retry_on_failure:
          enabled: true
          initial_interval: 5s
          max_interval: 30s
          max_elapsed_time: 300s
        sending_queue:
          enabled: true
          num_consumers: 4
          queue_size: 1000
        protocol:
          otlp:
            # resiliency options 2: per-endpoint sub-exporter queue+retry,
            # ON by default; tuned here for fast failure detection
            timeout: 5s
            sending_queue:
              enabled: true
              num_consumers: 2
              queue_size: 500
        resolver:
          k8s:
            # operator-created headless service of the sampler tier
            service: otel-sampler-collector-headless.observability

    extensions:
      health_check:
        endpoint: 0.0.0.0:13133

    service:
      extensions: [health_check]
      telemetry:
        metrics:
          readers:
            - pull:
                exporter:
                  prometheus:
                    host: 0.0.0.0
                    port: 8888
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter]
          exporters: [load_balancing]

Three operational notes the exporter documentation insists on and the manifest encodes:

• Not-ready endpoints are excluded from the routing ring. The k8s resolver drops EndpointSlices whose conditions.ready is explicitly false — which is what you want during a sampler rollout (unready pods leave the ring before traffic hits them). Edge case worth knowing: a Service with publishNotReadyAddresses: true forces ready always-true, keeping not-ready pods in the ring.
• The k8s resolver converges faster than DNS. Both eventually reflect topology changes, but the k8s resolver watches EndpointSlices directly instead of polling A records — the docs recommend it for elastic environments, and the residual window (seconds) is the rehash window discussed above.
• DNS alternative if you skip the operator: resolver: dns: hostname: otel-sampler-collector-headless.observability.svc.cluster.local against a headless service — same consistent hashing, slower convergence (default resolver interval 5s), no RBAC needed.

Tier 2: The Sampling Tier

The sampler tier runs as a StatefulSet-mode collector (stable pod ordinals, stable network identity) and carries the full policy program. This is the manifest to read line by line:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel-sampler
  namespace: observability
spec:
  mode: statefulset
  replicas: 3
  image: docker.io/otel/opentelemetry-collector-contrib:0.161.0
  resources:
    limits:
      memory: 2Gi
    requests:
      memory: 2Gi
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317

    processors:
      # ORDER MATTERS: memory_limiter first, tail_sampling second, batch last.
      # memory_limiter hard 1792 MiB = 87.5% of the 2GiB container;
      # spike 358 MiB (documented ~20% guidance) -> soft limit ~1434 MiB
      memory_limiter:
        check_interval: 1s
        limit_mib: 1792
        spike_limit_mib: 358

      tail_sampling:
        # sizing from the worked example: 5k traces/s peak / 3 pods = 1,667/s
        # per pod; 30s window -> 50k in-flight per pod; 2x headroom -> 100k
        decision_wait: 30s
        num_traces: 100000
        expected_new_traces_per_sec: 2000
        # if the buffer ever fills anyway: block upstream (backpressure into
        # the gateway's queue) instead of silently evicting undecided traces
        block_on_overflow: true
        # kill pathological traces before they burn the window's memory
        maximum_trace_size_bytes: 5242880   # 5 MiB per trace
        decision_cache:
          sampled_cache_size: 50000
          non_sampled_cache_size: 200000
        policies:
          # --- the production program -----------------------------------
          # 1. always keep errors (the entire reason tail sampling exists)
          - name: keep-errors
            type: status_code
            status_code:
              status_codes: [ERROR]
          # 2. always keep slow traces: >2s end-to-end
          - name: keep-slow-traces
            type: latency
            latency:
              threshold_ms: 2000
          # 3. keep 100% of spans carrying an explicit force-sample flag
          #    (mid-incident escape hatch set by on-call via baggage/header)
          - name: force-sample
            type: boolean_attribute
            boolean_attribute:
              key: app.force_sample
              value: true
          # 4. never keep explicitly-muted traces (health checks, synthetic
          #    probes) even if slow or erroring
          - name: drop-muted
            type: drop
            drop:
              drop_sub_policy:
                - name: muted-traces
                  type: boolean_attribute
                  boolean_attribute:
                    key: app.do_not_sample
                    value: true
          # 5. drop the noise BEFORE the baseline policy sees it: k8s probes
          - name: drop-healthchecks
            type: drop
            drop:
              drop_sub_policy:
                - name: probe-paths
                  type: string_attribute
                  string_attribute:
                    key: url.path
                    values: [/healthz, /readyz, /livez, /metrics]
                    enabled_regex_matching: false
          # 6. baseline: 5% of everything else, for representative volume
          - name: baseline
            type: probabilistic
            probabilistic:
              sampling_percentage: 5

      # batch AFTER tail_sampling: survivors only, bigger batches downstream
      batch:
        timeout: 10s
        send_batch_size: 8192
        send_batch_max_size: 10000

    exporters:
      otlp/backend:
        endpoint: traces.example.com:4317
        tls:
          insecure: false
        retry_on_failure:
          enabled: true
          initial_interval: 5s
          max_interval: 30s
          max_elapsed_time: 300s
        sending_queue:
          enabled: true
          num_consumers: 4
          queue_size: 2000

    extensions:
      health_check:
        endpoint: 0.0.0.0:13133

    service:
      extensions: [health_check]
      telemetry:
        metrics:
          readers:
            - pull:
                exporter:
                  prometheus:
                    host: 0.0.0.0
                    port: 8888
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, tail_sampling, batch]
          exporters: [otlp/backend]

Reading notes for the six choices that are doing real work:

• block_on_overflow: true is a philosophical choice. Default behavior evicts the oldest undecided trace to admit new ones (data loss, invisible except via the dropped-early metric). Blocking instead applies backpressure up through the gateway queue — your ingestion edge slows down visibly rather than your trace data thinning out invisibly. For a debugging-focused pipeline, visible degradation beats silent loss. If tier-1 availability matters more than trace completeness, flip it back and alert harder on sampling_trace_dropped_too_early.
• maximum_trace_size_bytes is cheap insurance against pathological traces (a fan-out loop emitting 100k spans): the trace is dropped before the decision period so memory stays predictable. Introduced for exactly this purpose; 0 disables.
• Policy order is evaluation order, and drop policies run with full voting rights. A trace matching any drop sub-policy is not sampled even when keep-errors also matched — that is why drop-muted sits above keep-errors in intent (order does not gate it, but keep your mental model consistent). The deprecated invert_match style is gone from this config on purpose: use drop and not, the documented replacements.
• The decision caches outlive the buffer. sampled_cache_size remembers kept trace IDs after span data is released, so late spans of an already-kept trace inherit the keep decision instead of starting a fresh (differing) evaluation. non_sampled_cache_size does the same for dropped ones, and can be larger because at a 5% keep rate most decisions are drops. Without caches, late spans re-enter the full decision path — the classic cause of "half a trace in the backend."
• expected_new_traces_per_sec is a hint, not a limit — it pre-sizes internal data structures closer to real usage. Set it to per-pod reality (≈2,000 with headroom over the 1,667 computed), not the cluster total.
• num_shards is deliberately absent. The parallel-event-loop option (merged August 2026, capped at 256) reduces contention between ingestion and decision evaluation at very high per-pod load, and divides num_traces/cache/rate limits evenly across shards. At our per-pod rates the single loop is fine; reach for shards when sampling_decision_timer_latency p99 exceeds 1s on a right-sized instance. It cannot be combined with tail_storage.

The Policy Vocabulary (What You Can Express)

Everything above used six policies. The processor supports a wider vocabulary, and knowing it changes what you can express — all verified against the current README:

Two composition rules the docs state and configs in the wild get wrong: any drop vote wins over sample votes from other policies, and with the default sample_on_first_match: false every policy is evaluated (needed for correct adjusted counts under the tracestate gate — the docs explicitly warn against combining first-match with it).

trace-complete vs. span-ingest: The New Mode

For years the processor had exactly one evaluation model: accumulate spans per trace, fire a timer at decision_wait, evaluate policies against whatever arrived. March 2026 added a second model via #46762:

• trace-complete (default): decide on the timer path, with accumulated data. Most policy flexibility (stateful policies like latency/rate buckets work), later decisions, higher memory pressure.
• span-ingest: evaluate each incoming batch at ingest; terminal outcomes (a policy already guarantees keep/drop) finalize immediately — an error span can release its whole trace the moment it arrives instead of waiting out the window. Non-terminal traces finalize at cleanup. Stateful policies are rejected in this mode.

Why it matters operationally: span-ingest trades policy expressiveness (no latency/rate policies) for lower memory residency and faster keep decisions — error traces leave the buffer at error-arrival time. For an error-first sampling program on a memory-constrained tier it is worth piloting; the follow-up fixes through September (#47476, #48874, #51063) are all about making it production-grade. Our reference config stays on the default trace-complete because the latency policy is load-bearing in the program above. decision_wait_after_root_received (default 0s) accelerates either mode when a root span is available — worth setting once you have measured your real trace-completion distribution.

Late Spans and Adjusted Counts (The Deep Cuts)

Two topics almost every tail-sampling writeup skips, both of which will bite you in quarter three:

Late spans. A span is "late" if it arrives after the decision was made. Three scenarios, per the docs: (1) the trace still lives in the circular buffer → late spans inherit the decision; (2) no decision cache → the component has amnesia, late spans start a fresh decision window and can produce a different answer (half-traces in the backend); (3) decision cache configured → keep decisions are remembered even after span data is released. The September 17 fix (#50623) additionally reuses the decision threshold for late-arriving spans under the tracestate gate. Monitor lateness with the sampling_late_span_age histogram; if late spans are common in your platform (async jobs, queues), the caches are not optional.

Adjusted counts. If anyone downstream computes "how many requests did we actually serve?" from sampled traces, they need the sampling probability to correct the counts. The usetracestate feature gate (alpha, off by default) makes the processor read and write W3C tracestate probability fields (rv/th) so an upstream head sampler and the tail sampler interoperate without double-counting — the probabilistic policy then decides against tracestate randomness instead of its own FNV hash, and rate/bytes policies report the threshold they actually applied. The docs' warning is blunt: do not combine it with sample_on_first_match. If your analytics team has never asked about adjusted counts, they will, the week after you ship tail sampling.

Day-2: Metrics, Alerts, and the Runbook

The processor and exporter expose a full self-observability surface (the collector's own telemetry docs cover the readers syntax we used above). These are the queries that matter, in priority order — wire them before the first rollout weekend:

# 1. THE silent-loss signal: traces evicted before their decision.
# Any sustained rate > 0 means num_traces is too small for live traffic
# (or decision_wait too long). This is the metric the README's FAQ
# attributes to load-induced drops.
rate(otelcol_processor_tail_sampling_sampling_trace_dropped_too_early[5m]) > 0

# 2. Buffer residence time approaching the decision window.
# p95 of removal age close to decision_wait == drops are imminent
# under any volume increase.
histogram_quantile(0.95,
  rate(otelcol_processor_tail_sampling_sampling_trace_removal_age_bucket[10m]))
# alert when this exceeds ~0.8 * decision_wait (24s at a 30s window)

# 3. Decision evaluation itself is slow: >1s p99 delays decisions past
# the window and doubles as the signal to try num_shards > 1.
histogram_quantile(0.99,
  rate(otelcol_processor_tail_sampling_sampling_decision_timer_latency_bucket[10m])) > 1

# 4. Keep-rate drift: the actual sampled fraction.
# Expect ~(error_rate + slow_rate + baseline 5%); sudden change = policy
# or traffic-shape change, not "better monitoring".
sum(otelcol_processor_tail_sampling_global_count_traces_sampled{sampled="true"})
/
sum(otelcol_processor_tail_sampling_global_count_traces_sampled)

# 5. WHICH policy is doing the keeping (needs no gates; per-policy votes):
sum(rate(otelcol_processor_tail_sampling_count_traces_sampled{decision="sampled"}[10m])) by (policy)
/
sum(rate(otelcol_processor_tail_sampling_count_traces_sampled[10m])) by (policy)

# 6. Gateway tier: routing health. success=false means the k8s resolver
# cannot see backends (RBAC? namespace?) — every export is failing.
rate(otelcol_loadbalancer_num_resolutions{success="false"}[5m]) > 0

# 7. Backend egress saturation: queue near capacity (scale signal at
# 60-70% per the scaling docs; enqueue failures = already dropping).
otelcol_exporter_queue_size / otelcol_exporter_queue_capacity > 0.7
rate(otelcol_exporter_enqueue_failed_spans[5m]) > 0

# supporting signal: memory_limiter refusing = the 2GiB tier is undersized
rate(otelcol_processor_refused_spans[5m]) > 0

Three diagnostics that need a feature gate or config flag, worth knowing before an incident forces you to learn them:

• Which policy kept this trace? Enable --feature-gates=+processor.tailsamplingprocessor.recordpolicy and every kept span gains tailsampling.policy (plus tailsampling.composite_policy / tailsampling.cached_decision where applicable). Cheap, off by default, invaluable when someone asks why a specific trace exists.
• Span-level late ratio: sampling_late_span_age{le="+Inf"} / count_spans_sampled — the denominator needs +processor.tailsamplingprocessor.metricstatcountspanssampled. Rising ratio → grow caches or decision_wait.
• Policy evaluation errors (sampling_policy_evaluation_error): usually an OTTL typo or an attribute type mismatch; the policy silently votes "not sample" while erroring.

Deployment Order (and the Smoke Test)

Order matters because the gateway's resolver will happily route into a namespace where nothing listens yet:

# 0. operator itself (one-time)
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

# 1. namespace + tier-2 RBAC-independent pieces
kubectl create namespace observability || true

# 2. SAMPLER TIER FIRST — it must exist before the gateway resolves it
kubectl apply -f sampler.yaml
kubectl -n observability rollout status statefulset/otel-sampler-collector --timeout=300s

# 3. verify the headless service the gateway will resolve
kubectl -n observability get svc otel-sampler-collector-headless
kubectl -n observability get endpointslices -l operator.opentelemetry.io/collector-headless-service=Exists

# 4. gateway RBAC, then the gateway
kubectl apply -f gateway-rbac.yaml
kubectl apply -f gateway.yaml
kubectl -n observability rollout status deployment/otel-gateway-collector --timeout=300s

# 5. smoke test: emit a forcing error trace and grep the backend
#    (any OTLP-capable client; telemetrygen ships in contrib distributions)
kubectl run telemetrygen --rm -it --restart=Never --image=docker.io/otel/opentelemetry-collector-contrib:0.161.0 \
  -- telemetrygen traces --otlp-insecure --traces 100 \
  --otlp-endpoint otel-gateway-collector.observability:4317

The acceptance test is not "traces appear in the backend." It is the ratio: 100 generated traces minus the fraction your policies keep should appear, errors at 100%, and alert #1 flat at zero for the whole run. Then repeat the run while deleting a sampler pod (kubectl -n observability delete pod otel-sampler-collector-1) and watch alert #6 and the queue metrics through the failover — that is your real rollout rehearsal.

When to Skip All of This

The honest close: not every platform should run this topology. Skip the two-tier sampler if any of these hold:

• Your backend samples server-side anyway. If you are on a backend whose trace storage already applies retention-based sampling (Grafana Cloud's managed trace pipeline is the common case — see their OpenTelemetry integration docs), you are paying the collector's memory tax to solve a problem the backend already solves. Verify against your plan's actual retention semantics before building a tier.
• Volume is low. Under ~100 traces/sec cluster-wide, head sampling at 100% is free and unfalsifiable — there is nothing to save, and the operator + two tiers is more machinery than the entire rest of your observability stack.
• You cannot commit to the Day-2 surface. The seven alerts above are not optional hygiene; a tail sampler run without them is a device that silently destroys error traces under load. If nobody owns the metric, nobody owns the data loss.
• You wanted the managed path from day one. The distributions above are the OSS route; the AWS Distro for OpenTelemetry packages the same collector with the same processor behind AWS's operational surface, and the community Helm charts offer the middle ground if the operator is too much. The tail sampling semantics — affinity, sizing, caches — are identical everywhere; only the packaging changes.

And a final pointer for the wider pipeline: the same two-tier stateful pattern (stateless affinity router in front, stateful processor behind) recurs wherever full-stream context matters — spanmetrics aggregation via the service routing key is the documented sibling use case, and the log-reduction and tail-sampling examples in the exporter docs follow it too. Learn it once here, reuse it every time a processor needs "the whole story" to decide. For the storage side of the same pipeline decisions, see our Loki vs. Elasticsearch comparison.

References & Further Reading

• Scaling the Collector — official documentation — stateless/scraper/stateful taxonomy, two-tier topology, queue thresholds, when NOT to scale.
• Tail Sampling Processor (contrib) — policy reference, sampling strategies, decision caches, monitoring section. Beta stability; contrib and k8s distributions.
• Load Balancing Exporter (contrib) — routing keys, resolvers, resiliency options 1 vs 2, R/N reroute math.
• Kubernetes resolver example — the RBAC manifest this guide's gateway tier is built from.
• Collector v0.161.0 release — current stable referenced by every manifest here (core published 2026-09-14, contrib 2026-09-15).
• #46762 — sampling_strategy (span-ingest) plus #48699 — num_shards and #50623 — late-span threshold reuse: the 2026 changes that invalidated pre-2026 runbooks.
• OpenTelemetry Operator docs — v1beta1 CRD, deployment/statefulset modes, sidecar injection, Target Allocator.
• Collector configuration docs and management docs — pipeline model, component ordering, self-telemetry (readers).
• W3C Trace Context Level 1 — the tracestate header the probability-sampling interop builds on.
• Kubernetes EndpointSlices and headless Services — the API objects the k8s and DNS resolvers watch.
• golang.org/x/time/rate — the token-bucket implementation behind rate_limiting/bytes_limiting.
• Grafana OpenTelemetry docs — the managed-backend alternative referenced in "when to skip."