Web Crawler

Difficulty: Hard

What It Tests

Distributed crawling at scale, URL management, politeness constraints, deduplication.

Topics Covered

• Web Crawling at Scale
• Message Queues & Async Processing
• Database Design: SQL vs NoSQL
• Job Scheduling & Background Processing
• Database Sharding
• ZooKeeper / Distributed Coordination

Hello Interview Breakdown

Read the full Hello Interview breakdown →

Video Walkthroughs

• System Design Interview: Design a Web Crawler (Ex-Meta Staff Engineer) →
• System Design Interview: Design a Web Crawler →

Approach Hints

• URL frontier has two layers: a priority queue (recency, PageRank) and per-domain back queues with crawl-delay enforcement for politeness
• Distributed fetch workers pull from per-domain back queues; DNS caching avoids repeated lookups per domain
• URL deduplication via a distributed bloom filter (false positives acceptable — just means skipping a URL occasionally)
• Content deduplication via SimHash to detect near-duplicate pages; store crawled pages in S3 with metadata in a relational DB sharded by URL hash