Posts for: #Observability

Trace Storage and Querying

sohilladhani
#distributed-systems  #system-design  #observability  #databases 
A distributed trace is a tree of spans. Each span records one operation: service name, operation name, start time, duration, tags (key-value pairs), and parent span ID. Storing traces so that you can find “all traces for user 12345 that had errors in the payment service last hour” requires a storage design that supports multiple query patterns simultaneously on append-heavy write load. The Write Pattern Spans arrive as a stream: millions per second in a large system (after sampling).
[Read more]

Trace Sampling

sohilladhani
#distributed-systems  #system-design  #observability  #architecture 
A high-traffic service processes 100,000 requests per second. Recording a complete distributed trace for every request would generate hundreds of gigabytes of trace data per hour. Storing and querying it all is expensive and mostly useless: 99.9% of requests are successful and look identical. Trace sampling decides which requests to record in full. The challenge: you want to capture all failures, latency outliers, and interesting requests, while discarding the boring majority.
[Read more]