A service needs a database password, an API key, a TLS certificate. The naive approach: put secrets in environment variables, config files, or hardcode them. The problem: secrets in config files end up in git history. Environment variables get logged. Hardcoded secrets live in every build artifact. A centralized secrets management service solves this: services request secrets at runtime from a vault, which enforces access control and logs every access.

Traditional data warehouses coupled compute and storage on the same nodes. Running a large query required buying more hardware: more CPU and more storage together, even if you only needed more CPU for a week. Compute-storage separation decouples them: data lives in cheap object storage (S3, GCS, Azure Blob), and compute clusters spin up on demand to query it. Pay for compute only while queries run. Scale each dimension independently.

A row-oriented database stores each row contiguously: all columns for row 1, then all columns for row 2. This is optimal for OLTP workloads that read or write one row at a time. Analytics queries read one or two columns across millions of rows: “sum of revenue where region = ‘us-east’.” Row storage forces you to read every column of every row to compute this, even though you only need two.

A data warehouse ingests raw events from production systems and transforms them into queryable datasets for analytics. The transformation pipeline has stages: raw ingestion, cleaning and normalization, business-level aggregation. Medallion architecture names these stages: bronze, silver, gold. The naming is less important than the principle: each layer has a contract about data quality, and data only flows forward. Bronze: Raw and Immutable Bronze is append-only raw data exactly as it arrived from the source: JSON payloads from Kafka, CSV exports from operational databases, API responses.

In a webhook delivery system, each customer endpoint is an external dependency you don’t control. Some endpoints are fast and reliable. Others time out consistently, return 500s intermittently, or go down for hours during deployments. Without endpoint health tracking, your delivery workers waste time retrying dead endpoints while healthy endpoints get delayed because the retry queue fills up with failed attempts to unavailable ones. Per-Endpoint Health Scoring Track delivery outcomes per endpoint: successful deliveries, failures, consecutive failures, last success time.

Your service delivers webhooks to customer endpoints. How does the customer know the POST came from you and not an attacker? Without authentication, any party that knows a customer’s webhook URL can forge events. Webhook signatures solve this: you sign each payload with a shared secret, the customer verifies the signature before processing. HMAC-SHA256 Signing Generate a secret per customer endpoint (not one global secret: if one leaks, only that customer is affected).

Webhooks are outbound HTTP calls your service makes to customer endpoints when events occur. A payment completes: you POST to https://merchant.example.com/webhooks/payment. Unlike inbound requests where you control the client, webhook delivery fires into unknown infrastructure. Customer servers go down, time out, return 500s. Building reliable webhook delivery means treating outbound HTTP as a distributed messaging problem with at-least-once semantics. The Core Problem Naive webhook delivery: event fires, you make an HTTP call, move on.

A distributed trace is a tree of spans. Each span records one operation: service name, operation name, start time, duration, tags (key-value pairs), and parent span ID. Storing traces so that you can find “all traces for user 12345 that had errors in the payment service last hour” requires a storage design that supports multiple query patterns simultaneously on append-heavy write load. The Write Pattern Spans arrive as a stream: millions per second in a large system (after sampling).

A high-traffic service processes 100,000 requests per second. Recording a complete distributed trace for every request would generate hundreds of gigabytes of trace data per hour. Storing and querying it all is expensive and mostly useless: 99.9% of requests are successful and look identical. Trace sampling decides which requests to record in full. The challenge: you want to capture all failures, latency outliers, and interesting requests, while discarding the boring majority.

Getting code from a merged PR to production involves more than copying files to servers. A deployment must: roll out gradually, monitor for regressions, pause or roll back automatically on failure signals, and coordinate across dozens of services that depend on each other. Deployment orchestration is the system that makes this happen reliably without human intervention on the critical path. The Deployment State Machine Each deployment is a state machine: pending, running, paused, succeeded, rolled_back.

A test suite with 10,000 tests takes 2 hours to run sequentially. Distributed across 50 machines, it takes 2-3 minutes. Test parallelization is a solved problem in principle but creates real distributed systems challenges: test isolation, work distribution, and flaky tests that fail intermittently and poison the signal. Work Distribution Naively split 10,000 tests evenly: 200 per machine. The problem: tests have wildly varying durations. 200 fast unit tests complete in 10 seconds.

A large Java monorepo takes 45 minutes to build from scratch. 90% of the codebase hasn’t changed since the last build. Build artifact caching reuses outputs from previous builds when inputs haven’t changed, turning a 45-minute build into a 3-minute build. At scale, this is one of the highest-leverage infrastructure investments an engineering organization can make. Content-Addressed Build Cache The key insight: build outputs are deterministic given their inputs. If the input files and compiler version are identical to a previous build, the output is identical.

You deploy a new version of your JavaScript bundle. Users on cached CDN nodes still see the old version. CDN cache invalidation is how you force cached content to be replaced before its TTL expires. It’s more complex than database cache invalidation because the content lives in hundreds of PoPs around the world that must all be notified. Why CDN Invalidation Is Hard Cache invalidation in a single Redis instance: delete the key.

A CDN has 300 PoPs. Each PoP independently caches content. On a cache miss, each PoP fetches from origin. For a popular asset that isn’t yet cached: 300 PoPs, each with a cache miss, all simultaneously fetching from origin. That’s 300 concurrent origin requests for one piece of content. Origin shield solves this: a second caching tier that sits between all edge PoPs and origin, collapsing those 300 origin fetches into one.

Cloudflare has 300+ Points of Presence (PoPs) globally. When you make a request to a Cloudflare-protected site, you don’t connect to a server in a specific city — you connect to the nearest Cloudflare PoP, wherever that is. This works via anycast: multiple servers worldwide share the same IP address, and BGP routing delivers your packets to the geographically closest one. PoP placement and anycast are the foundation of how CDNs achieve global low latency.

DNS was designed in 1983 with no authentication. A recursive resolver has no way to verify that the answer it receives actually came from the authoritative nameserver. This creates attack surface: an attacker who can inject a forged DNS response can redirect traffic for any domain to any IP. DNSSEC adds cryptographic signatures to DNS responses. Understanding the attacks first explains why DNSSEC exists. DNS Cache Poisoning The Kaminsky attack (2008): DNS queries use UDP, which is connectionless.

DNS returns an IP address. It doesn’t have to return the same IP address every time. The authoritative nameserver can return different IPs based on where the query originated, which backend is healthy, or how much traffic each backend should receive. DNS-based traffic routing uses this flexibility to implement geographic routing, load balancing, and failover without changing a line of application code. GeoDNS The recursive resolver’s IP address reveals the approximate location of the client (or at least the ISP).

Every system design interview starts with a client making a request. Before that request reaches your load balancer, DNS has already run. DNS translates api.example.com into an IP address. It’s a globally distributed read-heavy system that handles 3.5 trillion queries per day. Understanding how resolution works explains why DNS changes take time to propagate and how DNS-based traffic routing is possible. The Resolution Chain A DNS query for api.example.com follows a chain of four server types:

A viral post on Reddit has 50,000 comments. You can’t load them all at once. You need pagination — but comment pagination is harder than standard cursor-based pagination because the data is a tree, not a list. Loading “the next 20 comments” in a threaded comment system requires deciding what “next” means for hierarchical data. Top-Level Pagination The simplest strategy: paginate only top-level comments. Load the first 25 top-level comments. Each top-level comment shows its top 3 replies inline.

Reddit comments nest arbitrarily deep. A top-level comment has replies. Each reply has replies. The thread can be 15 levels deep. Storing and querying hierarchical data in a relational database has multiple approaches, and the right one depends on how the data is read: do you fetch the whole tree at once, or do you load levels lazily? Adjacency List The simplest model: each comment stores its parent’s ID. comments(id, post_id, parent_id, author, body, created_at) A top-level comment has parent_id = NULL.

An IoT device goes offline. A backend service wants to read its current temperature. The device isn’t there to respond. A device shadow (also called digital twin) solves this: a server-side representation of the device’s last-known state that can be read and written to even when the device is offline. When the device reconnects, it synchronizes with the shadow. The Problem IoT devices are intermittently connected. Cellular, wifi, and industrial networks all have gaps.

A factory floor has 10,000 sensors each sending readings every second. Sending all 10,000 readings per second to the cloud costs bandwidth money and creates a massive ingestion problem. Most readings are uninteresting: the temperature was 23.4C last second, it’s 23.4C this second. Edge processing filters, aggregates, and transforms data before it reaches the cloud, pushing computation closer to where data originates. What to Do at the Edge Filtering: only forward readings that cross a threshold or change by more than a delta.

A temperature sensor in a factory sends a reading every 30 seconds. It runs on a microcontroller with 256KB of RAM and a 2G cellular connection. It can’t maintain a persistent HTTP connection. It can’t handle TLS handshakes with 10KB certificates. IoT devices require a protocol built for constrained environments. MQTT is that protocol, and device registration is the first problem you solve before any data flows. MQTT Basics MQTT is a publish/subscribe protocol over TCP, designed for low-bandwidth, high-latency, unreliable networks.

Video conferencing runs over UDP, not TCP. TCP retransmits lost packets, which adds latency: waiting for a retransmit before playing the next frame makes real-time audio and video stutter. UDP drops lost packets. The application must handle loss itself — and must do so in under 20ms to stay imperceptible. Packet loss concealment is the set of techniques for making packet loss invisible or inaudible to users. Why UDP TCP’s retransmission is fine for file transfer: it doesn’t matter if a packet arrives 200ms late as long as it arrives.

A two-person WebRTC call is peer-to-peer: Alice sends video directly to Bob and vice versa. A ten-person call can’t work the same way: each participant would need to send 9 video streams and receive 9 streams, consuming 9x the upload bandwidth of a one-to-one call. Group video calls require a media server. There are two architectures: SFU (Selective Forwarding Unit) and MCU (Multipoint Control Unit). The choice determines server cost, client CPU usage, and call quality.

Two browsers want to send video directly to each other. They can’t just open a TCP connection: they’re behind NAT, firewalls, and don’t know each other’s public IP addresses. WebRTC solves peer-to-peer media transport. But before peers can connect, they need a signaling server to exchange connection metadata. WebRTC handles the media; signaling handles the handshake. The Signaling Problem WebRTC is transport-agnostic about signaling: it doesn’t specify how peers find each other or exchange connection parameters.

Amazon has thousands of warehouses. When you order a product, the system checks which warehouses have it in stock, selects the optimal fulfillment location, and reserves that unit. Multi-location inventory is different from seat inventory consistency: seats are identical and interchangeable, but warehouse location matters for delivery time and shipping cost. The same product at a warehouse 2,000 miles away is not the same as one 50 miles away. The Data Model Inventory is per-SKU per-warehouse: (sku_id, warehouse_id, quantity_available, quantity_reserved).

Amazon’s left sidebar: “Brand: Nike, Adidas, New Balance. Price: Under $50, $50-$100. Size: 8, 9, 10, 11.” Each filter is a facet. Selecting one narrows results and updates the counts on all other facets. Faceted search is not just filtering — it’s computing counts for all possible filter values simultaneously, on the filtered result set. The Count Problem The hard part is not filtering: “show me Nikes under $50” is a simple index query.

Amazon sells 350 million products. A shoe has size, color, and material. A TV has screen size, resolution, refresh rate, and HDR type. A book has ISBN, author, and page count. No two product categories share the same attributes. A relational table with a column per attribute would have thousands of columns, almost all null for any given product. Variable-attribute product catalog is the data modeling problem of storing structured but heterogeneous data efficiently.

A Tinder match happens when two users both swipe right on each other. The moment user B swipes right on user A, the system must detect that A has already swiped right on B and trigger the match notification. This is mutual match detection: an efficient, low-latency check for bidirectional intent. The Naive Approach When user B swipes right on user A: query the swipes table for swiper_id=A AND swiped_id=B AND direction=right.