Big Data Architectures
Difficulty: Hard | Topic #32
What to Learn
Lambda architecture (batch layer + speed layer + serving layer), Kappa architecture (stream-only), batch processing with Spark/Hadoop, data lake vs data warehouse, compaction and partitioning for query efficiency.
Resources
- Hello Interview: Flink Deep Dive ↗
- ByteByteGo: What is a Data Pipeline? ↗
- ByteByteGo: Why is Kafka so Popular? ↗
Covered by Problems
| Problem | Difficulty | Link |
|---|---|---|
| Ad Click Aggregator | Hard | → |
| Metrics Monitoring System | Hard | → |
| Search Engine / FB Post Search | Hard | → |
Key Concepts to Master
- Lambda: batch layer for accuracy, speed layer for low latency, serving layer merges both — operationally complex
- Kappa: stream-only, reprocess from Kafka for corrections — simpler but requires replayable source
- Data lake (raw, schema-on-read) vs data warehouse (curated, schema-on-write)
- Parquet/ORC columnar formats for scan efficiency
- Partitioning by time + entity for partition pruning