Web Crawling at Scale
Difficulty: Hard | Topic #33
What to Learn
URL frontier (priority queue + politeness delays), distributed fetch workers, deduplication via bloom filters or URL hashes, robots.txt compliance, re-crawl scheduling, DNS caching.
Resources
- Hello Interview: Web Crawler Breakdown ↗
- System Design Interview: Web Crawler — URL Frontier, Robots.txt, Deduplication ↗
- ByteByteGo: How Search Really Works (Crawling, Indexing, Ranking) ↗
Covered by Problems
| Problem | Difficulty | Link |
|---|---|---|
| News Aggregator | Medium | → |
| Web Crawler | Hard | → |
Key Concepts to Master
- URL frontier with priority queues (freshness, PageRank) and back queues per domain for politeness
- Distributed fetch workers consuming from the frontier
- URL deduplication using a distributed bloom filter or hash set
- Robots.txt fetching and compliance per domain
- DNS cache to avoid repeated lookups per domain
- Content change detection with checksums