Job Scheduling & Background Processing
Difficulty: Medium | Topic #23
What to Learn
Cron-based scheduling, distributed job queues (Celery, Sidekiq), idempotent job design, at-least-once execution guarantees, failure retries with backoff, job deduplication, priority queues.
Resources
- Hello Interview: Long Running Tasks ↗
- System Design Interview: Job Scheduler with FAANG Engineer ↗
- Distributed Job Scheduler — System Design on Whiteboard ↗
Covered by Problems
| Problem | Difficulty | Link |
|---|---|---|
| Online Judge (LeetCode) | Medium | → |
| Job Scheduler | Medium | → |
| Price Tracking Service | Medium | → |
| Web Crawler | Hard | → |
Key Concepts to Master
- At-least-once delivery and why idempotent jobs are mandatory
- Exponential backoff with jitter for retries
- Job deduplication via unique job IDs and a seen-set
- Priority queues with starvation prevention
- Distributed cron with leader election to avoid duplicate job triggers