The namenode knows where data lives. It doesn’t move data. Once a client has chunk locations from the namenode, all data flows directly between the client and datanodes. That separation is what makes HDFS scale.

The Write Path#

Writing a file:

The client contacts the namenode, which allocates chunk IDs and assigns 3 datanodes for each chunk based on rack-aware placement.

The client sends data to the first datanode. The first datanode simultaneously writes to disk and forwards data to the second datanode. The second forwards to the third. This pipeline means the cross-rack bandwidth is used once, not three times.

Each datanode sends an acknowledgment upstream when it has committed its chunk to disk. The write is not confirmed to the client until all 3 datanodes acknowledge. This is synchronous replication: the write latency includes the slowest datanode in the pipeline.

Write pipeline for 3 replicas:
Client → DataNode A → DataNode B → DataNode C
              ack ←       ack ←       ack ←
Client receives write confirmation only after all 3 ack

The Read Path#

Reading a file:

The client asks the namenode for chunk locations. The namenode returns a list of datanodes for each chunk, sorted by proximity to the client (same rack preferred).

The client reads each chunk from the nearest available datanode. No coordination with the namenode during the read. If a datanode is slow or unresponsive, the client switches to another replica.

graph TD A[Client: read file.mp4] --> B[Namenode: returns chunk locations] B --> C[Client reads Chunk 1 from nearest DataNode] B --> D[Client reads Chunk 2 from nearest DataNode] B --> E[Client reads Chunk 3 from nearest DataNode] C --> F[Client reassembles file] D --> F E --> F style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style F fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

Append-Only Writes#

HDFS doesn’t support arbitrary in-place writes. You append to files or write new files. This simplifies consistency: there’s no scenario where two clients are writing to the same offset in a file. The write pipeline assumes sequential writes to a chunk.

This constraint matches the actual workload: MapReduce jobs produce output files that are written once and read many times. Log aggregation systems append continuously. Random writes aren’t the use case.

Speculative Execution for Reads#

When a read is slow (straggler datanode), HDFS can issue the same read to a second replica in parallel and use whichever response arrives first. This is hedged requests applied to distributed file storage: trade extra load for better tail latency.

At Oracle#

We had large export files generated by overnight batch jobs, read by downstream systems in the morning. Initially we read them sequentially from a single NAS mount. When the NAS was under load, reads were slow. We moved to a system where exports were split into segments, each segment read from a different server, then merged by the downstream consumer. Throughput tripled. Poor man’s HDFS, basically.

What I’m Learning#

HDFS’s performance comes from parallel data access: multiple chunks from multiple datanodes simultaneously, without any central server in the hot path. The namenode is only consulted for the “where is it?” question. Everything after that is direct client-to-datanode I/O.

How have you parallelized large reads or writes in systems where single-server throughput was the bottleneck?