Design a distributed web crawler that indexes the web for a search engine. The system must crawl 1 billion pages per day (~12,000 pages/second), respect robots.txt directives, enforce politeness policies (no more than 1 request per second per domain), and prioritize important or frequently-updated pages. The crawler stores raw HTML for downstream indexing and must handle the full messiness of the real web (redirects, duplicates, spider traps, malformed HTML).
In your 90-second answer, cover: - Distributed URL frontier: how you manage and prioritize the queue of URLs to crawl - Politeness enforcement: ensuring you do not overwhelm any single website - Deduplication: avoiding crawling the same content twice - Fault tolerance: how the system recovers from worker failures without losing progress
Constraints: the crawler must re-crawl the top 10 million most important pages at least once per day. It must detect and avoid spider traps (infinite URL spaces). Politeness must be enforced globally, not just per-worker.
How to approach it
- Hint 1
The URL frontier is not a simple queue -- it has two dimensions: priority (which URLs are most important to crawl next) and politeness (which URLs can be crawled right now without violating rate limits). Separate these concerns into a priority queue that feeds into per-domain politeness queues.
- Hint 2
Deduplication happens at two levels: URL-level (have we seen this URL before?) and content-level (have we seen this page content before, even at a different URL?). Bloom filters handle URL dedup efficiently; content fingerprinting (simhash, MinHash) handles near-duplicate detection.
- Hint 3
For fault tolerance, do not acknowledge a URL as crawled until the fetched content is safely persisted. If a worker crashes mid-crawl, the unacknowledged URLs return to the frontier automatically. This is the same at-least-once pattern used in message queue systems.
Ready to answer it out loud?
Record your answer in 1:30 and Preptile scores it 1–10 with specifics — what landed, what you skipped, and what to say next time.
Practising needs an invite code. Join the waitlist and we’ll send you one.