Changing a config value in etcd is fast. Ensuring every running instance of every service has picked up that change, and knowing with confidence that they all have, is slower and harder.

The Propagation Problem#

A service runs on 200 instances. You update a config key. The etcd watch mechanism notifies all watchers within milliseconds. But watchers are long-lived connections: a service instance that restarted recently might not have its watch re-established yet. A network partition between some instances and etcd delayed the watch event for some of them. One instance had a code bug that silently ignored watch events for that key prefix.

After 30 seconds, some instances have the new config, some don’t. How do you know?

Confirmation via Config Epoch#

Each time the config changes, increment a monotonically increasing epoch number as part of the config. Services include their current config epoch in health check responses or in regular heartbeats to a coordination service.

The deployment system waits until it sees all service instances reporting the new epoch before marking the rollout complete. If an instance doesn’t report the new epoch within N minutes, it’s flagged for investigation.

Config structure:
{
  "epoch": 1043,
  "timeout_ms": 5000,
  "max_connections": 100
}

Health check response includes current epoch:
GET /health → {"status": "healthy", "config_epoch": 1043}
graph TD A[Config updated: epoch 1043] --> B[etcd notifies all watchers] B --> C[Instance 1: applies epoch 1043] B --> D[Instance 2: applies epoch 1043] B --> E[Instance 3: delayed, still on epoch 1042] C --> F[Health check: reports epoch 1043] D --> F E --> G[Health check: still reports 1042 - flagged] F --> H[Rollout: 2/3 confirmed, waiting for instance 3] style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style F fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style G fill:#000000,stroke:#ff0000,stroke-width:2px,color:#fff style H fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

Fallback: Periodic Re-Sync#

Watches can be missed. A reconnecting client should always re-read the current config value from etcd on reconnect, not assume its last watch event had the latest value. This reconciliation loop means any instance that was partitioned from etcd will pick up the current state immediately on reconnect, even if it missed watch events during the partition.

This is the same pattern as read repair and anti-entropy: use events for fast propagation, use periodic full reconciliation to catch anything events missed.

Canary Config Rollouts#

For risky config changes, apply them to a canary subset first: 5% of instances get the new config, 95% keep the old. Monitor error rates and latency on canary instances. If metrics look good after 10 minutes, proceed to full rollout. If canary instances degrade, roll back the config on just the canary group.

This requires the config propagation system to support per-instance or per-group config overrides, not just cluster-wide values.

At Oracle#

Our config distribution was file-based, pushed during deployments. When we added a new config value for a feature, we’d deploy it to production and find that some nodes in the cluster had an old version of the config file because the deployment had partially failed. Partial deployment of config files was our most common source of “works on most nodes but not all” bugs. An epoch-based approach where every node reports its current config hash would have caught these instantly instead of through debugging.

What I’m Learning#

Config propagation is not done when you update the store. It’s done when you’ve confirmed every instance has applied it. Building that confirmation loop from the start saves hours of debugging later.

How do you verify in your systems that a configuration change has actually reached all running instances?