When 500,000 people hit your site at 10:00:00 AM, your backend doesn’t scale to 500,000 concurrent sessions in milliseconds. A virtual waiting room absorbs the spike: users enter a queue, receive a position, and are admitted to the actual purchase flow in controlled batches. Without it, the site goes down for everyone. With it, users wait but the experience is orderly. The Queue Token Pattern When a user arrives during high load, issue them a signed queue token containing their position and timestamp.

Write amplification is when a single logical write triggers multiple physical writes to storage. It’s one of those problems that’s invisible at small scale and becomes a serious bottleneck as data grows. LSM trees were designed to minimize it. Replication inherently causes it. Understanding where it comes from helps you predict system behavior under load. Sources of Write Amplification Storage layer: writing 4KB of data to a filesystem that uses 512-byte sectors requires updating multiple sectors plus the inode, the directory entry, and potentially metadata blocks.

System design interviews always start with “estimate the scale.” Engineers without production experience guess numbers and move on. Engineers who’ve been paged at 3am because they guessed wrong take this seriously. Capacity planning is the discipline of estimating resource requirements before you run out of them. The Estimation Framework Start with requests per second. Then translate into resource requirements: CPU, memory, network bandwidth, storage. Example: URL shortener serving 100 million active URLs.

You add a new derived field to your user profiles: “days since last purchase.” You can compute it for new events going forward. But existing users already have purchase history. You need to compute this field for all existing users retroactively. That’s a backfill. Backfills happen constantly in production systems. They’re more disruptive than they look. The Basic Problem You have a dataset of 500 million user records. You need to reprocess all of them to compute the new field.

The saga pattern coordinates multi-step distributed transactions by breaking them into a sequence of local transactions with compensating actions for rollback. What the saga post didn’t cover: there are two fundamentally different ways to implement coordination, and choosing between them shapes your entire system architecture. Choreography: Services React to Events In choreography, there is no central coordinator. Each service listens for events and reacts by doing its part and publishing its own events.

A model trained offline achieves 94% accuracy. In production, accuracy is 78%. The model didn’t change. The data did, in subtle ways you didn’t notice. This is training-serving skew: the distribution of features the model sees during training doesn’t match what it sees during serving. Where Skew Comes From Feature preprocessing differences: the training pipeline normalized “amount” by dividing by the maximum amount in the training dataset. The serving pipeline didn’t.

A model update ships. Fraud scores change. Conversion rates drop. You need to roll back to the previous model in minutes, not hours. If your model deployment is treated differently from code deployment, rollback is painful. If models are versioned and deployed with the same tooling as code, rollback is a one-command operation. What Model Versioning Means A model is a file (or set of files) produced by a training run.

You have a new fraud detection model. It performs better in offline evaluation: higher precision, higher recall on historical data. But offline metrics don’t always translate to production. A model can look great on historical data and behave unexpectedly when it sees live traffic with its latency constraints, real-time features, and edge cases that didn’t appear in the training set. Shadow mode lets you run the new model in production without affecting users.

Training a machine learning model is a batch job that runs for hours or days. Serving predictions from that model to users requires sub-100ms latency. These two requirements produce completely different infrastructure. The model serving layer is where the offline ML world meets the online serving world. The Prediction Service A model serving service exposes an API: input features in, prediction out. POST /predict { "model": "fraud_detection_v3", "features": { "amount": 1250.

You have a statistically significant result: the new checkout flow increases conversion by 2%. Now what? You don’t immediately flip it on for everyone. You ramp it: gradually increase the percentage of traffic seeing the new experience, watching for problems that didn’t show up in the experiment. Why Ramp After a Successful Experiment The experiment ran at 10% traffic for two weeks. It didn’t show problems. But some issues only appear at scale:

You have experiment assignments: which user got which variant. You have metric events: user completed checkout, user added to cart, user bounced. To compute experiment results, you need to join assignments to events. At millions of events per day across dozens of experiments, this join is the core engineering challenge of an experimentation platform. The Data Model Three streams of data: Assignment events: user 1001 was assigned to treatment in experiment “checkout_v2” at 14:32:00.

Treatment group conversion rate: 4.3%. Control group: 4.1%. Difference: 0.2%. Is that real, or random noise? You need statistics to answer that question, and most engineers implement A/B testing without really understanding what their stats are telling them. The Core Problem If you flip a fair coin 10 times and get 6 heads, you don’t conclude the coin is biased. You know 6 out of 10 is within the range of normal random variation.

You want to test two versions of a checkout flow. Half of users should see version A, half should see version B. The same user should always see the same version across sessions. The assignment must happen in milliseconds. And you need to run 50 experiments simultaneously without them interfering with each other. Deterministic Hashing The simplest assignment mechanism: hash the user ID, take the result modulo 100, assign to treatment or control based on the bucket.

A search index with a billion documents returns thousands of candidates matching a query. Showing all of them ranked by TF-IDF doesn’t produce great results. The ranking signals needed for good results (machine-learned models, user personalization, freshness, domain authority) are expensive to compute. You can’t compute them for a thousand candidates in under 100ms. The solution: a funnel. Use cheap signals to eliminate most candidates fast, then apply expensive signals to the small remainder.

A user searches for “couch.” The documents say “sofa.” Without query expansion, no results. With it, the search system understands that “couch” and “sofa” are synonyms and expands the query to match either. This seemingly simple addition improves recall dramatically, but introduces its own set of problems. Synonym Expansion The most basic form: maintain a synonym dictionary. Before executing the query, expand each term with its synonyms. "couch" → ["couch", "sofa", "settee", "loveseat"] Query: "buy couch" becomes: (buy) AND (couch OR sofa OR settee OR loveseat) Synonym dictionaries are domain-specific.

An inverted index maps terms to the list of documents containing them. A web-scale inverted index has billions of documents and trillions of term-document pairs. It doesn’t fit on one machine. You have to partition it. Two Partitioning Strategies Document partitioning: split documents across machines. Machine 1 has documents 1-1M, machine 2 has documents 1M-2M, and so on. Each machine builds a complete local inverted index over its documents. A query for “distributed systems” goes to every machine in parallel, each returns its local top-K results, a merger combines and re-ranks.

Before PageRank, search engines ranked results by how many times the search term appeared on the page. That was gameable: stuff a page with keywords and you’d rank for them. PageRank’s insight was that a link from one page to another is an implicit endorsement. A page linked to by many high-quality pages is probably high quality itself. The Algorithm Each page starts with a rank of 1/N (where N is the total number of pages).

A web crawler visits URLs, extracts content, discovers new URLs from links, and repeats. The loop is simple. Doing it at Google’s scale, visiting hundreds of billions of pages without overwhelming any single website, requires real engineering. We covered priority queues, content fingerprinting, and checkpointing in the context of web crawling. Those are the mechanics of the frontier. This post is about the distributed architecture around it. The Crawl Frontier The frontier is the queue of URLs to visit.

Generating a thumbnail is a one-time cost. Serving it is a continuous cost. For a platform with a billion photos, every profile picture load, every post thumbnail, every image preview is a serve request. Getting the serving architecture right determines whether your image infrastructure is a background cost or a constant fire drill. URL Structure as Cache Key Every image variant needs a stable, unique URL. The URL structure defines the cache key for the CDN.

Two images of the same photo. One was uploaded as a JPEG, the other was screenshotted, scaled 5%, and re-uploaded as a PNG. They look identical to a human. Their SHA-256 hashes share zero bits in common. Cryptographic hashing detects exact duplicates. Perceptual hashing detects near-duplicates. Why Cryptographic Hashes Don’t Work A single pixel difference changes SHA-256 completely. Re-encoding, resizing, or adding a watermark produces a completely different cryptographic hash. Detecting “this is the same photo someone already uploaded” requires a similarity measure, not an equality check.

A user uploads a photo. The platform needs: a thumbnail at 150x150, a medium preview at 800x600, the original compressed to 80% quality, EXIF data extracted, content safety scanning, and a perceptual hash for deduplication. None of these should block the upload response. All of them need to happen reliably. The Fan-Out After Upload The upload endpoint does one thing: store the raw file, acknowledge the upload, publish an event. Everything else is async.

A user uploads a 500MB video. 90% through, their mobile connection drops. They reconnect and have to start over. That’s the worst case for user experience and a waste of everyone’s resources. Resumable uploads solve this by making partial progress durable. Why Standard HTTP Uploads Break A standard HTTP POST sends the entire file as one request body. The server processes it when the full body is received. If the connection drops at byte 490MB of a 500MB upload, the server discards everything received so far.

Changing a config value in etcd is fast. Ensuring every running instance of every service has picked up that change, and knowing with confidence that they all have, is slower and harder. The Propagation Problem A service runs on 200 instances. You update a config key. The etcd watch mechanism notifies all watchers within milliseconds. But watchers are long-lived connections: a service instance that restarted recently might not have its watch re-established yet.

A misconfigured timeout brought down a service. The operator changed one value in a config file. Services that picked up the change started timing out on all requests. To fix it, you need to know what changed, when, and be able to revert it in under a minute. That requires config versioning. Config as Versioned Records Every write to a config store should record: what key changed, what the new value is, what the old value was, who made the change, and when.

Kubernetes stores all its state in etcd. Every pod spec, every deployment, every service endpoint. When a pod is scheduled, etcd records it. When a service endpoint changes, etcd records it. Everything that needs to be consistent across all Kubernetes control plane components lives there. etcd is a strongly consistent key-value store built on Raft. Understanding why you’d want this for configuration, and specifically the watch mechanism that makes it useful, is the interesting part.

Surge pricing reacts to imbalance. Driver repositioning tries to prevent it. If the system knows which zones will be in high demand in the next 30 minutes, it can suggest that idle drivers reposition there before the surge starts. That prediction is the hard part. Demand Forecasting Per Zone Historical patterns are strong for predictable events: the airport zone is high demand every Friday evening. The downtown zone surges at bar close time on weekends.

New Year’s Eve at midnight. Thousands of people try to book rides simultaneously. Drivers are outnumbered 10 to 1. Without any intervention, every rider waits an hour. Surge pricing’s goal is to make more drivers available (higher fares attract drivers on the fence about working) and reduce demand (some riders opt for alternatives). The system design question is how you compute and apply it. The Core Computation Surge is a function of the demand-to-supply ratio in a geographic zone.

“Send the nearest driver” sounds like the right dispatch rule. It minimizes pickup time for this one rider. But it ignores the driver’s position after the drop-off, other pending rider requests, and overall system efficiency. Real dispatch is an optimization problem. Nearest Driver Is a Greedy Heuristic Greedy nearest-driver assignment: find the closest available driver, assign them, done. Fast to compute, easy to understand. The problem is it optimizes for one request in isolation.

Every Uber driver’s phone sends a GPS update every 4-5 seconds. Uber has around 5 million active drivers globally. That’s about 1 million location updates per second hitting the write path. Most systems are read-heavy. Driver tracking is unusually write-heavy, and the data model is unusual too: old locations are worthless, only the current one matters. The Data Model A naive approach: a driver_locations table with columns driver_id, latitude, longitude, updated_at.

A live stream ends. The viewer who missed it wants to watch it on demand. The platform already has all the segments: they were generated during the live stream and pushed to the CDN. Making them into a VOD (video on demand) is less a technical challenge and more a stitching-and-indexing problem. Segments Are Already There During a live stream, transcoding at ingest generates 2-6 second segments at each quality level.