A web crawler visits URLs, extracts content, discovers new URLs from links, and repeats. The loop is simple. Doing it at Google’s scale, visiting hundreds of billions of pages without overwhelming any single website, requires real engineering.

We covered priority queues, content fingerprinting, and checkpointing in the context of web crawling. Those are the mechanics of the frontier. This post is about the distributed architecture around it.

The Crawl Frontier#

The frontier is the queue of URLs to visit. Naive implementation: one queue, workers pop a URL, fetch it, add new URLs back to the queue. This breaks in multiple ways.

Problem 1: you might visit the same URL multiple times. Seen-URL deduplication using a bloom filter keeps the frontier from re-adding already-visited URLs. Accept a small false positive rate (occasional missed URLs) to keep memory usage bounded.

Problem 2: you might crawl the same website too fast. Sending 1000 requests per second to a single server violates politeness and likely gets your crawler blocked. The frontier must enforce per-domain crawl rates.

Per-Domain Queues#

The solution to politeness: organize the frontier as per-domain sub-queues. URLs for example.com wait in the example.com queue. A worker assigned to example.com fetches one URL, then waits (politeness delay, typically 1-10 seconds) before fetching the next.

This requires mapping workers to domains. A consistent hashing ring assigns each domain to a specific worker or worker group. All URLs for example.com go to the same worker, ensuring the politeness delay is enforced without coordination.

graph TD A[New URL discovered] --> B{Already seen?} B -->|Yes - bloom filter hit| C[Discard] B -->|No| D[Mark as seen in bloom filter] D --> E[Add to per-domain queue] E --> F[Worker for that domain] F --> G[Fetch URL after politeness delay] G --> H[Extract content and links] H --> A style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#ff0000,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style F fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style G fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style H fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

Crawl Budget and Prioritization#

Not all URLs are equally important. A news homepage changes hourly; a corporate “About Us” page hasn’t changed in years. Crawl budget is finite: you need to allocate more crawl time to high-value, frequently-changing pages.

Signals for prioritization: PageRank of the source page, historical change frequency of the URL, link depth from known authoritative pages. URLs from the New York Times homepage get higher priority than URLs from a low-traffic personal blog.

DNS at Scale#

Resolving the DNS for every URL at fetch time is slow and hammers DNS servers. Crawlers cache DNS results, typically for minutes to hours. A large crawler with millions of distinct domains maintains a local DNS cache that handles the hot domains in memory, falling back to actual DNS resolution only on cache misses.

At Oracle#

We had to periodically re-download updated configuration templates from vendor documentation pages. The “web crawler for config updates” was effectively a tiny, domain-specific crawler. We kept a simple SQLite database of URL + last-fetched timestamp + content hash. If the hash changed since last fetch, process the update. This solved exactly the change-detection and politeness problems at the micro-scale we needed.

What I’m Learning#

Web crawling at scale is a frontier management problem more than a fetching problem. The fetch is trivial; managing which URL to fetch next, at what rate, with which priority, without revisiting is where the real distributed systems work lives.

Have you built systems that needed to poll or crawl external resources, and how did you handle rate limiting and change detection?