Design a distributed message queue system similar to Apache Kafka that provides durable, ordered, high-throughput message streaming between producers and consumers.
Scale requirements: - 1 million messages per second write throughput - 500 topics with varying partition counts - Message size: 1 KB average, up to 1 MB max - Consumers must be able to replay messages (retention-based, not deletion-on-read) - 99.99% durability (no message loss after acknowledgment)
Areas to cover: 1. Topic and partition model 2. Message storage and replication 3. Producer and consumer protocols 4. Ordering guarantees and consumer groups 5. Rebalancing and fault tolerance
Constraints: - Messages must be durable once acknowledged by the broker - Ordering guaranteed within a partition - Support consumer groups with automatic partition assignment - Handle broker failures without data loss - Efficient sequential reads for high consumer throughput
How to approach it
- Hint 1
Start with the data model: how are messages organized into topics and partitions? Why is partitioning essential for both write throughput and ordering? How does a producer decide which partition to write to?
- Hint 2
Consider the storage engine: how do you store messages for both high write throughput and efficient sequential reads? Think about append-only log segments, page cache utilization, and zero-copy transfers.
- Hint 3
Think about consumer groups and rebalancing: how do multiple consumers in a group coordinate which partitions each consumes? What happens when a consumer crashes or a new one joins?
Ready to answer it out loud?
Record your answer in 1:30 and Preptile scores it 1–10 with specifics — what landed, what you skipped, and what to say next time.
Practising needs an invite code. Join the waitlist and we’ll send you one.