Three replicas gives you fault tolerance. It also means every byte of storage you buy, you get one-third of it for actual data. For cold storage with petabytes at rest, that cost is hard to justify. Erasure coding is how you get fault tolerance without paying 3x.

The Idea#

Take a chunk of data. Split it into k data fragments. Generate m parity fragments using the data fragments. Store all k+m fragments on different nodes. As long as you have any k fragments available, you can reconstruct the original data.

A common configuration is 6+3: 6 data fragments, 3 parity fragments, 9 fragments total across 9 nodes. You can lose any 3 nodes and still reconstruct the data. Storage overhead is 9/6 = 1.5x, versus 3x for triple replication.

Triple replication: 1TB data → 3TB storage, tolerate 2 failures
Erasure coding 6+3: 1TB data → 1.5TB storage, tolerate 3 failures
Same fault tolerance, half the storage cost.

The Trade-Off: Rebuild Cost#

Reading data is cheap: read k fragments from k nodes (parallel reads). Reconstruction after a failure is expensive: read all remaining fragments from up to k nodes, run the mathematical reconstruction algorithm (Reed-Solomon), write the rebuilt fragment to a replacement node. This uses significantly more network bandwidth than restoring a replica in a 3x replication scheme.

graph TD A[Original Data Block] --> B[Split into 6 Data Fragments] B --> C[Generate 3 Parity Fragments via Reed-Solomon] B --> D[Fragment 1: Node A] B --> E[Fragment 2: Node B] B --> F[Fragment 3: Node C] C --> G[Parity 1: Node G] C --> H[Parity 2: Node H] C --> I[Parity 3: Node I] style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style F fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style G fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style H fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style I fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

When to Use Each#

Erasure coding fits cold or warm storage: video archives, backup data, analytical datasets. Infrequent reads, infrequent failures, but high data volume where the storage cost reduction is significant.

Replication fits hot data: active databases, frequently-read files, any workload where read latency matters and you want all replicas readable. Reading from the nearest replica is always faster than reconstruction.

HDFS supports both: 3x replication for hot data in default storage tier, erasure coding for cold storage tiers where the cost savings outweigh the reconstruction overhead.

At Salesforce#

We stored audit logs that had to be retained for 7 years per compliance requirements. Initially on full-replication object storage at significant cost. The logs were read maybe twice a year for legal holds. We moved audit log archives to erasure-coded storage after 30 days: 60% storage cost reduction with the same durability SLA. Read latency went from milliseconds to seconds for reconstruction, but for a 7-year-old audit log retrieved for a lawsuit, nobody cared.

What I’m Learning#

Erasure coding and replication aren’t competing approaches. They’re tools for different parts of the data lifecycle. The question is where the access pattern justifies the reconstruction overhead cost.

Have you worked with storage tiering strategies that mixed replication and erasure coding?