A high-traffic service processes 100,000 requests per second. Recording a complete distributed trace for every request would generate hundreds of gigabytes of trace data per hour. Storing and querying it all is expensive and mostly useless: 99.9% of requests are successful and look identical. Trace sampling decides which requests to record in full. The challenge: you want to capture all failures, latency outliers, and interesting requests, while discarding the boring majority.

Head-Based Sampling#

The sampling decision is made at the start of the request, before any spans are created. The root service flips a coin: record this trace (e.g., 1% probability) or discard it. This decision propagates through all downstream services via the trace context header. All services either record or discard together.

Simple to implement. The problem: you decide to sample before you know whether this request is interesting. A request that takes 10 seconds and fails gets discarded 99% of the time by a 1% sampler. You miss the failures you most want to see.

Tail-Based Sampling#

The sampling decision is made after the request completes, once all spans are available. A trace collector aggregates all spans for a request, evaluates criteria (error occurred? latency > threshold?), then decides whether to store the trace.

Always sample: errors, requests over 2 seconds, requests to specific endpoints under investigation. Sample at 1%: successful requests under latency threshold.

graph TD A[Request completes: all spans collected] --> B[Tail sampler: evaluate criteria] B --> C{Any error spans?} C --> |Yes| D[Sample: store full trace] C --> |No| E{Latency over 2 seconds?} E --> |Yes| D E --> |No| F{Random 1% sample?} F --> |Yes| D F --> |No| G[Discard: drop all spans for this trace] style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style F fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style G fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

Tail Sampling Infrastructure#

Tail-based sampling requires collecting all spans for a trace before making the sampling decision. Spans from different services arrive at different times. A collector must buffer spans, group them by trace ID, wait for the root span (which indicates completion), then evaluate.

This requires: a collector cluster where all spans for a trace ID route to the same collector instance (consistent hashing by trace ID), a buffer with a timeout (e.g., 30 seconds — if root span hasn’t arrived, make a decision and flush), and memory proportional to in-flight request count.

At 100,000 requests/second with average 500ms duration, ~50,000 traces are in-flight at any moment. Each trace has 10 spans averaging 1KB: 500MB in-flight buffer per collector node. Manageable, but requires careful capacity planning.

Adaptive Sampling#

Fixed rates (1%) waste budget during low traffic and undersample during spikes. Adaptive sampling adjusts the rate to hit a target traces-per-second. During a traffic spike, lower the rate; during low traffic, raise it. The budget (e.g., 100 traces/second) stays constant regardless of traffic.

At Oracle#

The Oracle Cloud observability platform used head-based sampling at 10% by default with a flag to force-sample specific request types. When debugging a latency regression in a payment microservice, we temporarily set the sampling rate to 100% for that service’s traces. This identified that 2% of requests had a specific SQL query taking 8+ seconds that was invisible at 10% sampling. Forced full sampling for targeted debugging was the most useful operational lever.

What I’m Learning#

Head-based sampling is simple but blind; tail-based sampling requires more infrastructure but captures what matters. In practice, most teams use head-based sampling with forced full sampling for errors and targeted investigations. Pure tail-based sampling is worth the infrastructure cost only when systematic capture of all anomalies is required.

Have you used tail-based sampling in production, and was the infrastructure complexity worth the improved capture rate?