A web crawler visits URLs, extracts content, discovers new URLs from links, and repeats. The loop is simple. Doing it at Google’s scale, visiting hundreds of billions of pages without overwhelming any single website, requires real engineering. We covered priority queues, content fingerprinting, and checkpointing in the context of web crawling. Those are the mechanics of the frontier. This post is about the distributed architecture around it. The Crawl Frontier The frontier is the queue of URLs to visit.