HDFS has a single namenode. Every read and write starts there. For a system designed to scale to petabytes, having one machine that every client must contact sounds like a terrible idea. It kind of is. Understanding why they did it anyway, and how they mitigated it, is the interesting part.

What the Namenode Stores#

The namenode holds all filesystem metadata in memory: the directory tree, file-to-chunk mappings, chunk-to-datanode mappings, and file permissions. For a large cluster with billions of small files, this memory requirement grows until the namenode can’t hold it all. This is the “small files problem” in HDFS: a cluster that handles a few large files well can choke on millions of small ones.

All metadata changes are written to an edit log before being applied. On startup, the namenode replays the edit log to reconstruct state. Periodically, it takes a snapshot of in-memory state (fsimage) so replays don’t have to start from scratch. The edit log plus fsimage pattern is similar to write-ahead logging.

The Single Point Problem#

Original HDFS had no standby namenode. If it went down, the entire cluster was unavailable. Warm standby was added later: a standby namenode continuously replays edit log entries and can take over when the primary fails, with a failover time of tens of seconds.

Later, HDFS federation allowed multiple namenodes, each managing a separate namespace volume. Different applications could use different namespace volumes, distributing the metadata load across multiple namenodes.

graph TD A[Namenode Primary] -->|edit log stream| B[Namenode Standby] A --> C[Shared Edit Log: JournalNodes] B --> C C -->|replayed to| B B -->|promoted on failure| D[Namenode Active] style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

The Trade-Off That Made It Work#

The namenode handles metadata requests only. It never moves file bytes. A cluster with a 1Gbps namenode and 1000 datanodes each with 10Gbps bandwidth has 10 terabits per second of data throughput while the namenode handles only thousands of metadata RPC calls per second. The namenode is not in the data path, so its throughput limits don’t limit data throughput.

The design works because most of the I/O is data I/O, and data I/O bypasses the namenode entirely.

At Salesforce#

Our metadata database became a bottleneck in a different way. A single MySQL table stored all configuration keys for all orgs. As org count grew past 100K, every deployment that needed to check “does this org have feature X enabled” hit the same table. We added a read replica and cached frequently-read keys in Redis. Same problem as namenode overload: the index server for a large dataset gets hot even when actual data is spread across many machines.

What I’m Learning#

Single master for metadata works when the metadata fits in memory and the data plane is entirely separate. The failure mode is the small files problem (too much metadata) and single point of failure (mitigated with standby). HDFS federation shows you can split the namespace when one namenode isn’t enough.

What single-master systems have you seen that hit scaling limits, and how did you work around them?