You are building a web crawler that must index 1 billion web pages across the internet, refreshing them approximately once per week. The crawler feeds a search engine by generating a reverse index of words to pages. Explain your crawler architecture, how you avoid infinite loops and duplicate content, how you prioritize which pages to crawl, and how you would scale to handle 2 petabytes of content per month.
How to approach it
- Hint 1
Use a URL frontier (priority queue) to manage which URLs to crawl next -- not all pages are equally important.
- Hint 2
To detect duplicate content, generate page signatures (e.g., using Jaccard similarity or cosine similarity) rather than comparing raw HTML.
- Hint 3
Respect robots.txt and crawl rate limits -- being polite is part of the design.
Ready to answer it out loud?
Record your answer in 1:30 and Preptile scores it 1–10 with specifics — what landed, what you skipped, and what to say next time.
Practising needs an invite code. Join the waitlist and we’ll send you one.