A service needs a database password, an API key, a TLS certificate. The naive approach: put secrets in environment variables, config files, or hardcode them. The problem: secrets in config files end up in git history. Environment variables get logged. Hardcoded secrets live in every build artifact. A centralized secrets management service solves this: services request secrets at runtime from a vault, which enforces access control and logs every access.

Traditional data warehouses coupled compute and storage on the same nodes. Running a large query required buying more hardware: more CPU and more storage together, even if you only needed more CPU for a week. Compute-storage separation decouples them: data lives in cheap object storage (S3, GCS, Azure Blob), and compute clusters spin up on demand to query it. Pay for compute only while queries run. Scale each dimension independently.

A data warehouse ingests raw events from production systems and transforms them into queryable datasets for analytics. The transformation pipeline has stages: raw ingestion, cleaning and normalization, business-level aggregation. Medallion architecture names these stages: bronze, silver, gold. The naming is less important than the principle: each layer has a contract about data quality, and data only flows forward. Bronze: Raw and Immutable Bronze is append-only raw data exactly as it arrived from the source: JSON payloads from Kafka, CSV exports from operational databases, API responses.

In a webhook delivery system, each customer endpoint is an external dependency you don’t control. Some endpoints are fast and reliable. Others time out consistently, return 500s intermittently, or go down for hours during deployments. Without endpoint health tracking, your delivery workers waste time retrying dead endpoints while healthy endpoints get delayed because the retry queue fills up with failed attempts to unavailable ones. Per-Endpoint Health Scoring Track delivery outcomes per endpoint: successful deliveries, failures, consecutive failures, last success time.

Webhooks are outbound HTTP calls your service makes to customer endpoints when events occur. A payment completes: you POST to https://merchant.example.com/webhooks/payment. Unlike inbound requests where you control the client, webhook delivery fires into unknown infrastructure. Customer servers go down, time out, return 500s. Building reliable webhook delivery means treating outbound HTTP as a distributed messaging problem with at-least-once semantics. The Core Problem Naive webhook delivery: event fires, you make an HTTP call, move on.

A high-traffic service processes 100,000 requests per second. Recording a complete distributed trace for every request would generate hundreds of gigabytes of trace data per hour. Storing and querying it all is expensive and mostly useless: 99.9% of requests are successful and look identical. Trace sampling decides which requests to record in full. The challenge: you want to capture all failures, latency outliers, and interesting requests, while discarding the boring majority.

Getting code from a merged PR to production involves more than copying files to servers. A deployment must: roll out gradually, monitor for regressions, pause or roll back automatically on failure signals, and coordinate across dozens of services that depend on each other. Deployment orchestration is the system that makes this happen reliably without human intervention on the critical path. The Deployment State Machine Each deployment is a state machine: pending, running, paused, succeeded, rolled_back.

A test suite with 10,000 tests takes 2 hours to run sequentially. Distributed across 50 machines, it takes 2-3 minutes. Test parallelization is a solved problem in principle but creates real distributed systems challenges: test isolation, work distribution, and flaky tests that fail intermittently and poison the signal. Work Distribution Naively split 10,000 tests evenly: 200 per machine. The problem: tests have wildly varying durations. 200 fast unit tests complete in 10 seconds.

A large Java monorepo takes 45 minutes to build from scratch. 90% of the codebase hasn’t changed since the last build. Build artifact caching reuses outputs from previous builds when inputs haven’t changed, turning a 45-minute build into a 3-minute build. At scale, this is one of the highest-leverage infrastructure investments an engineering organization can make. Content-Addressed Build Cache The key insight: build outputs are deterministic given their inputs. If the input files and compiler version are identical to a previous build, the output is identical.

You deploy a new version of your JavaScript bundle. Users on cached CDN nodes still see the old version. CDN cache invalidation is how you force cached content to be replaced before its TTL expires. It’s more complex than database cache invalidation because the content lives in hundreds of PoPs around the world that must all be notified. Why CDN Invalidation Is Hard Cache invalidation in a single Redis instance: delete the key.

A CDN has 300 PoPs. Each PoP independently caches content. On a cache miss, each PoP fetches from origin. For a popular asset that isn’t yet cached: 300 PoPs, each with a cache miss, all simultaneously fetching from origin. That’s 300 concurrent origin requests for one piece of content. Origin shield solves this: a second caching tier that sits between all edge PoPs and origin, collapsing those 300 origin fetches into one.

A viral post on Reddit has 50,000 comments. You can’t load them all at once. You need pagination — but comment pagination is harder than standard cursor-based pagination because the data is a tree, not a list. Loading “the next 20 comments” in a threaded comment system requires deciding what “next” means for hierarchical data. Top-Level Pagination The simplest strategy: paginate only top-level comments. Load the first 25 top-level comments. Each top-level comment shows its top 3 replies inline.

An IoT device goes offline. A backend service wants to read its current temperature. The device isn’t there to respond. A device shadow (also called digital twin) solves this: a server-side representation of the device’s last-known state that can be read and written to even when the device is offline. When the device reconnects, it synchronizes with the shadow. The Problem IoT devices are intermittently connected. Cellular, wifi, and industrial networks all have gaps.

A factory floor has 10,000 sensors each sending readings every second. Sending all 10,000 readings per second to the cloud costs bandwidth money and creates a massive ingestion problem. Most readings are uninteresting: the temperature was 23.4C last second, it’s 23.4C this second. Edge processing filters, aggregates, and transforms data before it reaches the cloud, pushing computation closer to where data originates. What to Do at the Edge Filtering: only forward readings that cross a threshold or change by more than a delta.

Amazon has thousands of warehouses. When you order a product, the system checks which warehouses have it in stock, selects the optimal fulfillment location, and reserves that unit. Multi-location inventory is different from seat inventory consistency: seats are identical and interchangeable, but warehouse location matters for delivery time and shipping cost. The same product at a warehouse 2,000 miles away is not the same as one 50 miles away. The Data Model Inventory is per-SKU per-warehouse: (sku_id, warehouse_id, quantity_available, quantity_reserved).

Amazon’s left sidebar: “Brand: Nike, Adidas, New Balance. Price: Under $50, $50-$100. Size: 8, 9, 10, 11.” Each filter is a facet. Selecting one narrows results and updates the counts on all other facets. Faceted search is not just filtering — it’s computing counts for all possible filter values simultaneously, on the filtered result set. The Count Problem The hard part is not filtering: “show me Nikes under $50” is a simple index query.

A Tinder match happens when two users both swipe right on each other. The moment user B swipes right on user A, the system must detect that A has already swiped right on B and trigger the match notification. This is mutual match detection: an efficient, low-latency check for bidirectional intent. The Naive Approach When user B swipes right on user A: query the swipes table for swiper_id=A AND swiped_id=B AND direction=right.

Tinder processes 1.6 billion swipes per day. Each swipe is a binary decision: left (pass) or right (like). This data drives match detection, recommendation quality, and abuse prevention. Swipe storage is a write-heavy, eventually consistent problem with specific read patterns: “has user A already swiped on user B?” and “who has liked user A?” Write Path At 1.6 billion swipes per day, that’s 18,500 writes per second on average, with heavy peaks during evenings.

Tinder has 50 million users. When you open the app, you see a deck of profiles. Not 50 million profiles — maybe 100. Candidate generation is the step that narrows from the full user population to the small set of candidates worth scoring and presenting. It runs before the expensive ranking step, and its job is to be fast and to recall the right candidates, not to rank them perfectly.

Gmail filters billions of spam messages per day. The filters must run before message delivery, add minimal latency, and maintain low false-positive rates: legitimate email landing in spam is a worse outcome than spam landing in inbox. Spam filtering is a layered classification pipeline with each layer trading latency for accuracy. Layer 1: Connection and Sender Reputation Before reading a single byte of message content, the receiving mail server checks the sender:

Web search builds one massive index shared by all users. Email search is different: every query is private to one user, and results depend on that user’s labels, read status, and folder structure. Gmail’s search index is not one index — it’s one index per user. Per-user search indexes are the right architecture when search results are private, personalized, and must reflect user-specific metadata. Why Not a Shared Index A shared inverted index for email would store every user’s email together: word “invoice” points to all documents across all users containing “invoice.

Gmail groups related emails into conversations. Reply to a thread from three different email clients, forward it twice, and all seven messages collapse into one conversation. This is email threading: the problem of grouping messages that belong to the same conversation, despite arriving out of order, from different clients, with inconsistent subject lines. Threading Identifiers Email headers carry threading signals: Message-ID: a unique identifier for each message, set by the sending client.

An advertiser wants to reach “women aged 25-34 in San Francisco who have browsed running shoes in the last 7 days.” When a bid request arrives for a user, the DSP must evaluate whether this user matches the targeting criteria in under 5ms. The ad targeting pipeline is the system that makes this lookup fast. Audience Segments as Bit Arrays Each targeting criterion defines an audience segment: a set of user IDs.

An advertiser sets a $1,000 daily budget. Without pacing, the DSP would spend the entire budget in the first hour of the day, when bid prices happen to be low, and the ad would show to nobody for the remaining 23 hours. Budget pacing distributes ad spend evenly across the day (or according to a target schedule) so the advertiser gets coverage throughout the day rather than a burst. The Pacing Problem Pacing is a rate control problem with a stochastic input.

When you load a web page, an auction happens in under 100 milliseconds to decide which ad you see. A publisher sends a bid request to an ad exchange. The exchange fans it out to dozens of demand-side platforms (DSPs). Each DSP decides whether to bid and at what price. The exchange picks the winner. All of this completes before the page finishes loading. Real-time bidding (RTB) is one of the most latency-sensitive distributed systems in production.

A payment system that approves every transaction is useless. One that declines too many legitimate transactions loses customers. Fraud detection is a latency-sensitive classification problem: decide in under 100ms whether a transaction is fraudulent, using signals that fraudsters will actively try to evade. The Signal Stack Fraud signals come from multiple layers: Velocity checks: has this card been used more than 5 times in the last minute? Has this device initiated more than $500 in the last hour?

When your location changes, every friend who has you in their Nearby Friends list needs to know. At 10 million active users each with 200 friends, a single location update could trigger 200 downstream notifications. At 333,000 updates per second, that’s 66 million fan-out operations per second. This is location fan-out: propagating position changes to all interested subscribers efficiently. Why This Is Different From Chat Fan-Out Chat fan-out sends one message to N subscribers.

Nearby Friends shows you that a friend is “0.3 miles away.” It does not show you their exact coordinates. This is not just a UI choice: the system should never expose precise location data even if the client asks for it. Privacy-aware proximity means computing distance and revealing enough for the feature to be useful, while structurally preventing exact location exposure. Bucketed Distance The simplest approach: compute exact distance server-side, then bucket the result before returning it.

A Nearby Friends feature needs to know where each of your friends is right now. Not where they were an hour ago. The data has a very short useful life: a location update older than 30 seconds is essentially stale. This is ephemeral location storage: high-frequency writes, short TTL, and no need for historical persistence. The Write Pattern Each mobile client sends location updates every 30 seconds while the app is active.

You pick a seat, enter your credit card details, and for those 10 minutes the seat is yours. Nobody else can take it. If you don’t complete payment in time, the seat releases back to inventory. This is the hold-and-confirm pattern: temporarily reserve a resource, give the user time to complete a multi-step transaction, then either confirm the reservation or release it. Why You Need It Without holds, the user experience is broken.