A 10GB video file on a single disk is just a file. A 10GB video file in a distributed system is a problem. How do you store it? Which machine does it go on? What happens when that machine dies?

The answer used in GFS and HDFS: split the file into fixed-size chunks, store each chunk on a different machine, and keep the mapping from file to chunks in a separate metadata server.

Chunk Servers and the Metadata Split#

Files are divided into chunks, typically 64MB or 128MB each. A 10GB file becomes 160 chunks. Each chunk is stored on multiple machines for fault tolerance, usually 3 replicas.

The critical design decision: separate the data plane from the metadata plane. Chunk servers store raw bytes. The namenode (in HDFS) or master (in GFS) stores the metadata: which chunks make up which file, where each chunk is stored, and which chunk servers are alive.

Client wants to read file.mp4:
  1. Ask namenode: what chunks does file.mp4 have?
  2. Namenode returns: [chunk_1 on servers A,B,C], [chunk_2 on servers D,E,F]...
  3. Client talks directly to chunk servers to read data
  4. Namenode is never in the data path
graph TD A[Client] --> B[Namenode: metadata only] B --> C[Chunk locations returned to client] A --> D[Chunk Server A: chunk 1 data] A --> E[Chunk Server B: chunk 2 data] A --> F[Chunk Server C: chunk 3 data] style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style F fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

The namenode never sees the actual file bytes. This keeps metadata operations fast and scales data throughput independently of metadata throughput.

Why 64MB Chunks?#

Large chunks reduce metadata volume. 1 billion 1KB files would need 1 billion chunk records. With 64MB chunks, most files are a single chunk. The namenode can hold all metadata in memory because there isn’t much of it.

Large chunks also mean fewer client-server round trips for large sequential reads. Reading a 1GB log file means touching ~16 chunks, not thousands.

At Oracle#

We stored large binary configuration snapshots for network slice states. Initially we used a single-row blob column in MySQL. Blobs over 16MB caused query issues and replication lag spikes. We split large configs into 4MB segments stored in separate rows, with a manifest row pointing to them. Chunk-based thinking applied to a relational database. The manifest row was cheap to load; the segment rows were fetched only when needed.

What I’m Learning#

Separating metadata from data is the foundational decision in distributed file systems. It makes both planes independently scalable, keeps the metadata server’s working set small enough to fit in memory, and keeps data reads off the critical path through the metadata server.

Have you designed systems where separating the “index” from the “data” solved a scaling problem?