Platform Monkey ctx

OpenTelemetry Trace Sampling Strategies: Head, Tail, and the Budget

· By
Sources

In a distributed architecture producing 10k+ requests per second, capturing 100% of traces is not a technical challenge — it is a financial one. The cost of ingesting, indexing, and storing every span quickly exceeds the value of the data, as 99% of healthy traffic provides zero diagnostic utility.

The goal of a sampling strategy is representativeness: the ability to maintain a statistically accurate view of system performance while ensuring that "interesting" events — errors, latency spikes, and rare edge cases — are captured with 100% fidelity.

The Sampling Hierarchy

Sampling is not a single toggle but a tiered pipeline. Decisions are made at different stages of the trace lifecycle, each with different costs and information availability.

1. Head-Based Sampling (Deterministic)

Head-based sampling makes a decision at the root span. If the root span is dropped, the entire trace is typically dropped across all downstream services.

The industry standard is TraceID Ratio Sampling. This is a deterministic process: the TraceID is hashed, and if the resulting value falls below a certain threshold, the trace is kept. This ensures that if Service A decides to sample a trace, Service B (which sees the same TraceID) will make the same decision, preventing "broken" traces with missing spans.

The Analytics Hazard: A naive backend sees 10% of traces and reports 10% of the traffic. To fix this, the OTel spec uses tracestate to propagate the sampling probability. The backend then multiplies the observed rate by the inverse of the probability (e.g., observed 100 rps @ 10% sampling = 1,000 actual rps).

2. Tail-Based Sampling (Contextual)

Tail-based sampling defers the decision until the entire trace (or a significant portion of it) has been collected. This allows the system to sample based on the outcome of the request.

This requires a buffering layer (typically an OTel Collector tier) that holds all spans for a specific TraceID for a defined decision_wait period (e.g., 30 seconds) before applying policies.

Architectural Blueprint: The Hybrid Pipeline

For high-volume production environments, neither approach is sufficient alone. Head-sampling alone misses the "needle in the haystack" errors; tail-sampling alone can overload the collector memory if 100% of spans are sent to it.

The Hybrid Pattern uses head-sampling to protect the pipeline and tail-sampling to ensure diagnostic quality.

[ App / SDK ] 
      |
      | (Head Sampling: 5% Deterministic)
      v
[ Load Balancer ] -- (Stickiness: Hash by TraceID)
      |
      v
[ OTel Collector Tier (Tail Sampling) ] 
      |
      |-- Buffer Spans (decision_wait: 30s)
      |-- Evaluate Policies (Error? Slow? Random?)
      |-- Drop non-matching traces
      v
[ Observability Backend (Jaeger/Tempo/Honeycomb) ]

Implementation: Collector Configuration

The following configuration implements a production-grade tail-sampling tier that prioritizes errors and high-latency requests while maintaining a 5% baseline for healthy traffic.

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 2048
    spike_limit_mib: 256

  tail_sampling:
    decision_wait: 30s
    num_traces: 100000
    expected_new_traces_per_sec: 500
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow-requests
        type: latency
        latency:
          threshold_ms: 2000
      - name: baseline-healthy-traffic
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/backend]

Infrastructure: Ensuring Trace Stickiness

Tail sampling fails if spans of the same trace arrive at different collector instances. You must implement TraceID-based load balancing.

# Example Envoy Cluster configuration for OTel Collector load balancing
clusters:
- name: otel_collector_cluster
  lb_policy: RING_HASH
  # The load balancer hashes the 'traceid' header to ensure all spans
  # for a single trace land on the same pod.
  hash_policy:
    - header:
        name: "x-trace-id" 
  load_assignment:
    cluster_name: otel_collector_cluster
    endpoints:
      - lb_endpoints:
          - endpoint:
              address:
                socket_address:
                  address: otel-collector-0.otel-collector-svc
                  port_value: 4317
          - endpoint:
              address:
                socket_address:
                  address: otel_collector-1.otel-collector-svc
                  port_value: 4317

Production Realities

Implementing tail sampling introduces specific operational failure modes that are often ignored in vendor documentation.

The Memory Wall

Tail sampling is memory-intensive. The collector must store every span of every trace for the duration of the decision_wait. If you have 1,000 spans per trace and 1,000 traces per second, a 30s wait means buffering 30 million spans in RAM.

Failure Mode: If the collector hits its memory limit, it will either crash (OOM) or begin dropping spans randomly, which corrupts the very traces you are trying to sample.

The "Late Arrival" Problem

Distributed systems suffer from clock drift and network jitter. If a span from a downstream service arrives after the decision_wait timer has expired, the collector has already made the sampling decision. That late span is either dropped or exported as an orphan, leaving the trace incomplete.

The Budgeting Trade-off

A common mistake is setting sampling_percentage: 100 for errors. In a "storm" scenario (e.g., a database outage), 100% of your traffic might be errors. This will saturate your network and crash your backend.

Best Practice: Use a composite policy with max_total_spans_per_second to cap the total ingestion rate regardless of how many "interesting" traces are found.

Who Should Skip This Pattern

This architecture is overkill for teams in the following scenarios:

References & Further Reading