Design a metrics collection and alerting system that monitors the health and performance of a large-scale distributed infrastructure and triggers alerts when anomalies are detected.
Scale requirements: - 10,000 services reporting metrics - 5 million unique time series - 1 million metric data points ingested per second - Query latency under 500ms for dashboard rendering - Alert evaluation every 15 seconds
Areas to cover: 1. Metric collection and ingestion 2. Time-series storage and retention 3. Query engine for dashboards and ad-hoc analysis 4. Alert rule evaluation and notification 5. Downsampling and long-term storage
Constraints: - Support multiple metric types: counters, gauges, histograms - High cardinality labels must be handled efficiently - Alerts must fire within 60 seconds of threshold breach - Support alert silencing, grouping, and escalation - Minimize false positives while catching real incidents
How to approach it
- Hint 1
Start with collection: how do metrics get from services to your system? Think about pull-based (Prometheus scraping) vs push-based (StatsD/OTLP) models, and their tradeoffs for service discovery and reliability.
- Hint 2
Consider the storage engine: time-series data has unique access patterns (recent data queried frequently, append-only writes). How do TSDB engines (Prometheus TSDB, InfluxDB, Mimir) optimize for this?
- Hint 3
Think about the alerting pipeline: how do you evaluate thousands of alert rules every 15 seconds efficiently? What about alert fatigue -- how do grouping, deduplication, silencing, and escalation help?
Ready to answer it out loud?
Record your answer in 1:30 and Preptile scores it 1–10 with specifics — what landed, what you skipped, and what to say next time.
Practising needs an invite code. Join the waitlist and we’ll send you one.