Design a centralized log aggregation system that collects, processes, and makes searchable the logs from thousands of services across a large distributed infrastructure.
Scale requirements: - 5,000 microservices producing logs - 500 GB of logs per day (compressed) - Peak ingestion: 2 million log events per second - Logs must be searchable within 30 seconds of emission - Retain hot logs for 7 days, warm for 30 days, cold for 1 year
Areas to cover: 1. Log collection and shipping from services 2. Ingestion pipeline and buffering 3. Indexing and full-text search 4. Storage tiering and retention 5. Querying and dashboarding
Constraints: - Structured logging (JSON) with support for unstructured fallback - Must handle log spikes during incidents without data loss - Support multi-tenant log isolation (team-based access control) - Cost-efficient storage for 1-year retention
How to approach it
- Hint 1
Start with collection: how do logs get from application containers to your pipeline? Think about sidecar agents (Fluentd/Filebeat), stdout capture, and the buffering needed to handle bursts without back-pressuring applications.
- Hint 2
Consider the ingestion and indexing layer: how do you handle 2M events/second? Think about Kafka as a buffer, stream processors for enrichment, and Elasticsearch or a columnar store for indexing.
- Hint 3
Think about storage tiering: hot data in Elasticsearch for fast search, warm data with reduced replicas, cold data in object storage (S3) with a separate query engine. How do you make this transparent to the user?
Ready to answer it out loud?
Record your answer in 1:30 and Preptile scores it 1–10 with specifics — what landed, what you skipped, and what to say next time.
Practising needs an invite code. Join the waitlist and we’ll send you one.