Kubernetes stores all its state in etcd. Every pod spec, every deployment, every service endpoint. When a pod is scheduled, etcd records it. When a service endpoint changes, etcd records it. Everything that needs to be consistent across all Kubernetes control plane components lives there.

etcd is a strongly consistent key-value store built on Raft. Understanding why you’d want this for configuration, and specifically the watch mechanism that makes it useful, is the interesting part.

Why Not Just a Database?#

A relational database could store configuration. The problem: services need to know when configuration changes. They could poll every N seconds, but polling adds latency and wastes resources. What you want is push: when configuration changes, affected services find out immediately.

etcd’s watch mechanism does exactly this. A client opens a watch on a key or key prefix. The server holds that connection. When the key changes, the server sends the updated value to all watchers without any polling.

// Watch a key: etcd holds connection and streams changes
watchResponse := etcdClient.Watch(ctx, "/config/feature-flags")
for resp := range watchResponse {
    for _, event := range resp.Events {
        // event.Type is PUT or DELETE
        // event.Kv.Value is the new value
        applyConfigChange(event)
    }
}
graph TD A[Service A: watches /config/timeout] --> B[etcd cluster] C[Service B: watches /config/feature-flags] --> B D[Operator updates /config/timeout = 5000ms] --> B B --> E[Notifies Service A: new value 5000ms] B --> F[Service A applies config in ~100ms] style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style F fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

Strong Consistency for Configuration#

etcd uses Raft, which means all reads and writes go through a leader. Every read reflects the most recent committed write. This is linearizable consistency.

For configuration, this matters. If you set a feature flag to disabled, you need all services to see it as disabled. Eventually consistent config propagation means some services see “enabled” while others see “disabled” for some time window. For some flags that’s fine; for a kill switch, it’s not.

The cost: every read and write hits the Raft leader. Read throughput is limited by the single leader. etcd mitigates this by supporting serializable reads (reading from any replica, potentially stale), letting callers choose their consistency trade-off per operation.

Leases and TTL#

etcd supports leases: a key can be associated with a lease that has a TTL. If the client holding the lease doesn’t renew it, the key is automatically deleted. This is how ZooKeeper ephemeral nodes work, but as a first-class etcd concept.

Service registration uses leases: a service registers its endpoint in etcd with a 30-second lease and renews every 10 seconds. If the service crashes, the lease expires and the endpoint is removed. Service discovery consumers watching the endpoint key prefix see the removal immediately.

At Oracle#

We had manual config file distribution: operators edited a config file, ran a deployment script to push it to all nodes, nodes read on startup. When a node was restarted with an old config, it would pick up stale values. We spent years debugging “but I changed that config” bugs. A centralized config store with watch-based live reload would have eliminated an entire class of problems. We didn’t have etcd available in our stack at the time, so we used Spring Cloud Config Server, which is the same idea but with HTTP polling instead of watches.

What I’m Learning#

The watch mechanism is what separates a config store from a regular database. Poll-based config reloading is technically simpler but introduces latency, load, and “when did each service actually pick up the change?” uncertainty. Push-based watches give you near-instant propagation and a clear event stream for auditing.

Have you worked with configuration systems that used push-based notification, and how much faster was propagation compared to polling?