System design interviews always start with “estimate the scale.” Engineers without production experience guess numbers and move on. Engineers who’ve been paged at 3am because they guessed wrong take this seriously. Capacity planning is the discipline of estimating resource requirements before you run out of them.
The Estimation Framework Start with requests per second. Then translate into resource requirements: CPU, memory, network bandwidth, storage.
Example: URL shortener serving 100 million active URLs.
Changing a config value in etcd is fast. Ensuring every running instance of every service has picked up that change, and knowing with confidence that they all have, is slower and harder.
The Propagation Problem A service runs on 200 instances. You update a config key. The etcd watch mechanism notifies all watchers within milliseconds. But watchers are long-lived connections: a service instance that restarted recently might not have its watch re-established yet.
A misconfigured timeout brought down a service. The operator changed one value in a config file. Services that picked up the change started timing out on all requests. To fix it, you need to know what changed, when, and be able to revert it in under a minute. That requires config versioning.
Config as Versioned Records Every write to a config store should record: what key changed, what the new value is, what the old value was, who made the change, and when.