Storing 3 replicas of a chunk on 3 different machines sounds like good fault tolerance. It’s not, if all 3 machines are in the same network rack. A rack losing power or a top-of-rack switch failing takes down all 3 replicas simultaneously. You’ve built 3 copies but achieved the fault tolerance of 1.

How Racks Fail#

In a data center, servers are grouped into racks of 20-40 machines. Each rack shares a top-of-rack (TOR) switch and often shares power distribution. A rack-level failure, power outage or switch failure, takes out every machine in that rack at once. Data centers also have row-level and room-level failure domains, though those are rarer.

Rack-aware replication in HDFS uses the network topology explicitly. When placing 3 replicas of a chunk:

  • Replica 1: local machine (for write speed)
  • Replica 2: different machine in a different rack
  • Replica 3: different machine in the same rack as Replica 2 (saves cross-rack bandwidth)

This placement ensures that no single rack failure can wipe out all replicas. At least one replica always survives a rack-level failure.

graph TD A[Chunk to Write] --> B[Replica 1: Rack A, Server 1 - local] A --> C[Replica 2: Rack B, Server 5 - different rack] A --> D[Replica 3: Rack B, Server 7 - same rack as R2, saves bandwidth] B -->|intra-cluster| E[Write Pipeline Complete] C --> E D --> E style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

The Write Pipeline#

To avoid the client sending the same data 3 times across the network, HDFS uses a write pipeline. The client sends data to Replica 1. Replica 1 forwards it to Replica 2 while still receiving. Replica 2 forwards to Replica 3. Acknowledgments flow back in the opposite direction. The write is confirmed only when all 3 replicas acknowledge.

This cuts network usage: data crosses the wide switch once (client to Rack A) and the cross-rack link once (Rack A to Rack B), not three times.

The Rebalancer#

As data is added and nodes are added or removed, replicas can become unevenly distributed. Some datanodes fill up while others are empty. HDFS runs a background rebalancer that moves chunks around to even out utilization while respecting rack constraints.

At Oracle#

We didn’t run HDFS, but we thought about failure domains for our 5G core configuration data. Configurations for network slices were stored in a MySQL cluster. We realized our primary and replica were in the same physical rack. A rack power event would have taken both down. We explicitly requested rack diversity for database VMs as part of infrastructure provisioning. It sounds obvious in retrospect.

What I’m Learning#

Fault tolerance is only as strong as the failure domain analysis behind it. Three replicas in one rack is theater. Rack-aware placement makes the replication factor actually mean what you think it means.

Have you ever audited your replication setup against actual failure domains in your infrastructure?