A user uploads a 500MB video. 90% through, their mobile connection drops. They reconnect and have to start over. That’s the worst case for user experience and a waste of everyone’s resources. Resumable uploads solve this by making partial progress durable. Why Standard HTTP Uploads Break A standard HTTP POST sends the entire file as one request body. The server processes it when the full body is received. If the connection drops at byte 490MB of a 500MB upload, the server discards everything received so far.

A live stream ends. The viewer who missed it wants to watch it on demand. The platform already has all the segments: they were generated during the live stream and pushed to the CDN. Making them into a VOD (video on demand) is less a technical challenge and more a stitching-and-indexing problem. Segments Are Already There During a live stream, transcoding at ingest generates 2-6 second segments at each quality level.

The namenode knows where data lives. It doesn’t move data. Once a client has chunk locations from the namenode, all data flows directly between the client and datanodes. That separation is what makes HDFS scale. The Write Path Writing a file: The client contacts the namenode, which allocates chunk IDs and assigns 3 datanodes for each chunk based on rack-aware placement. The client sends data to the first datanode. The first datanode simultaneously writes to disk and forwards data to the second datanode.

Three replicas gives you fault tolerance. It also means every byte of storage you buy, you get one-third of it for actual data. For cold storage with petabytes at rest, that cost is hard to justify. Erasure coding is how you get fault tolerance without paying 3x. The Idea Take a chunk of data. Split it into k data fragments. Generate m parity fragments using the data fragments. Store all k+m fragments on different nodes.

Storing 3 replicas of a chunk on 3 different machines sounds like good fault tolerance. It’s not, if all 3 machines are in the same network rack. A rack losing power or a top-of-rack switch failing takes down all 3 replicas simultaneously. You’ve built 3 copies but achieved the fault tolerance of 1. How Racks Fail In a data center, servers are grouped into racks of 20-40 machines. Each rack shares a top-of-rack (TOR) switch and often shares power distribution.

HDFS has a single namenode. Every read and write starts there. For a system designed to scale to petabytes, having one machine that every client must contact sounds like a terrible idea. It kind of is. Understanding why they did it anyway, and how they mitigated it, is the interesting part. What the Namenode Stores The namenode holds all filesystem metadata in memory: the directory tree, file-to-chunk mappings, chunk-to-datanode mappings, and file permissions.

A 10GB video file on a single disk is just a file. A 10GB video file in a distributed system is a problem. How do you store it? Which machine does it go on? What happens when that machine dies? The answer used in GFS and HDFS: split the file into fixed-size chunks, store each chunk on a different machine, and keep the mapping from file to chunks in a separate metadata server.

Most of your data is accessed once and then never again. Storing it on fast, expensive storage forever is just burning money. Hot, Warm, Cold The canonical model is three tiers based on access frequency. Hot storage (SSD-backed, high IOPS) handles recent data that’s accessed constantly. Warm storage (standard HDD or S3 Standard-IA) holds data accessed occasionally. Cold storage (archival, like Glacier) holds data that might never be touched again but legally must be retained.

You save a 200 MB file. One word changed. Re-uploading 200 MB to sync that change is absurd. Delta sync is how you avoid it. The Core Idea Split the file into blocks. On an update, compare the new version’s blocks against the stored version’s blocks. Transfer only the blocks that changed. Rsync pioneered this. It computes a fast rolling checksum for each block on the remote side, sends those checksums to the client, the client finds which local blocks match and which don’t, and transmits only the mismatches.

Two users upload the same 50 MB file. Naive storage keeps two copies. Content-addressable storage keeps one. What “Content-Addressable” Means Instead of locating data by where it lives (a path, a filename), you locate it by what it is. Hash the content, use the hash as the key. Same content, same hash, same storage location. SHA-256 a file and store the result as its address. The practical consequence: deduplication becomes automatic.

A user hits Ctrl+Z forty times and expects to land exactly where they were yesterday. That is not just undo. That is a complete audit trail of every edit, stored efficiently, queryable at any point in time. The naive approach: store a full copy of the document after every change. Works for ten users. Collapses at ten thousand. Deltas, Not Copies Instead of storing full document state after every edit, store only what changed: the operation (insert 3 chars at position 12, delete 5 chars at position 20).