A misconfigured timeout brought down a service. The operator changed one value in a config file. Services that picked up the change started timing out on all requests. To fix it, you need to know what changed, when, and be able to revert it in under a minute. That requires config versioning.

Config as Versioned Records#

Every write to a config store should record: what key changed, what the new value is, what the old value was, who made the change, and when. This is an audit log, and it’s the foundation for rollback.

etcd stores a revision number with every key. Every time any key in the cluster changes, the global revision increments. You can read a key at a specific revision: “show me the value of /config/timeout as it was at revision 1042.” This is point-in-time read, similar to MVCC in databases.

# Get current value
etcdctl get /config/timeout

# Get value at specific revision
etcdctl get /config/timeout --rev=1042

# Roll back: restore old value
etcdctl put /config/timeout "3000"  # was 3000 before the bad change

Atomic Multi-Key Changes#

A problem with applying multiple config changes one by one: the system briefly sees a partially-applied config. Service A reads timeout=5000 and max_connections=50 (old), but Service B reads timeout=5000 (new) and max_connections=50 (old). The combination they see never existed in a valid state.

etcd transactions let you atomically update multiple keys in one operation. Either all updates apply or none do. Services reading after the transaction always see the complete new config or the complete old one, never a mix.

graph TD A[Config Change: update 3 keys] --> B{etcd Transaction} B -->|Atomic commit| C[All 3 keys updated atomically] B -->|Failure| D[Zero keys updated, no partial state] C --> E[Watchers see complete new config] D --> F[Watchers see unchanged old config] style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style F fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

Change Approval Workflow#

For production config changes, a “config as code” approach adds a human review step. Config changes are proposed in a pull request, reviewed, merged to a config repository, and then applied to etcd by an automation pipeline. This adds latency but prevents unreviewed changes from reaching production.

Emergency rollback bypasses the review: an operator runs a rollback command that applies the last known good config state, with the change logged automatically.

At Salesforce#

We had 4000+ service configurations managed as files in a repository. Deployments pushed config updates to services. When a bad config shipped, rollback meant reverting the file in the repository and redeploying, which took 15-20 minutes. We treated this as acceptable until we had an incident where that 15-minute window caused a 63% drop in query throughput across a large service. After that, we built an emergency override system that could push a specific config value instantly, bypassing the normal deployment pipeline. The lesson: regular rollback paths can be slow; emergency rollback needs a fast path.

What I’m Learning#

Config versioning is table stakes. The fast emergency rollback path is what saves you during incidents. You need both, and the emergency path needs to be tested regularly so it works when you need it.

Have you had a production incident where config rollback took longer than it should have, and what did you change afterward?