Walkthrough: Designing a Metrics & Monitoring System

The problem

You’re asked to design a metrics and monitoring system — the system underneath Prometheus, Datadog, or CloudWatch. The interviewer says something like: “Tens of thousands of servers emit measurements — CPU, request latency, queue depth. Engineers need dashboards over this data and alerts when things go wrong.”

Sounds like a write-heavy database. It isn’t — or rather, that framing misses what makes the workload tractable. The trap is reaching for a generic “Kafka + wide-column store” pipeline without noticing the time-series shape: writes are append-only per series, points are never updated, queries aggregate over series, not rows — and the thing that actually breaks these systems is not data volume but cardinality, the number of distinct series alive at once.

Below is how I’d walk through this, start to finish.

1. Requirements

First 3–5 minutes. Questions I’d ask to pin down what we’re building:

Functional requirements

• R1. Ingest measurements from all hosts/services — numeric samples attached to a named, labeled series.
• R2. Query with aggregation — rates, percentiles, group-bys over arbitrary time ranges (p99 latency by region, last 6h).
• R3. Alerting — rules evaluated continuously against incoming data, firing within a bounded delay.
• R4. Dashboards — sub-second reads over recent data for many concurrent viewers.
• R5. Retention with downsampling — raw data for weeks, aggregates for a year-plus.

Scope

Explicitly out of scope:

• Logs and traces (different data shapes, different systems — the three pillars share dashboards, not storage)
• Anomaly detection / ML on metrics (a consumer of this system)
• Incident management beyond alert delivery (paging schedules, escalation)
• Billing/metering pipelines (metrics-shaped, but with exactly-once requirements this design deliberately doesn’t pay for)

Non-functional requirements

• Fleet size and series count. Hosts × series-per-host is the cardinality number everything depends on.
• Resolution. Samples every 10 s? 1 s? Resolution multiplies ingest directly.
• Alert latency. How stale may a firing alert be? This bounds the whole ingest path.
• Query recency bias. What fraction of queries touch the last few hours? (In practice: nearly all — it decides the storage split.)
• Durability. Is losing a few seconds of metrics on a crash acceptable? (Usually yes — worth saying, because it buys a lot of design freedom.)

Say the interviewer confirms: 50K hosts, ~1K series each → ~50M active series; 10-second resolution → 5M samples/sec; alerts must fire within 60 s; raw retention 30 days, downsampled 13 months; dashboards dominated by last-6-hours queries; losing < 1 minute of data in a crash is tolerable, but alerting silence is not.

2. Capacity estimate

• Ingest: 5M samples/sec. A sample is ~16 bytes raw (timestamp + float64) → 80 MB/sec. With time-series compression (~1.5 bytes/sample, below) → ~8 MB/sec compressed. Trivial bandwidth.
• Storage: 5M/sec × 86,400 × 30 days × 1.5 B ≈ ~20 TB for raw retention — a handful of nodes.
• The index: 50M active series, each needing label-to-series lookup structures in memory across the ingest fleet. At ~1 KB of index overhead per series, that’s ~50 GB of pure index — and it grows with every new label combination ever seen in the retention window, not just the active set.

I’d say out loud: “The estimate says something unusual: the bytes are easy — 20 TB is nothing. The pressure is all in cardinality: 50M series of in-memory index, and queries that fan across tens of thousands of series each. So the design centers on the series index and per-series write batching, and the ‘high write throughput’ part is solved by an append-only layout plus compression, not by a giant fleet.”

3. High-level architecture

Three logical subsystems:

1. Ingest — collectors receive (or scrape) samples, shard them by series, batch, and write to the storage tier.
2. Time-series storage — recent data in memory (the head), sealed into immutable time-partitioned blocks; a label index over both.
3. Query & alerting — a query engine that resolves label selectors to series, fetches chunks, and aggregates; alert rules are just queries on a timer.

flowchart TD
  Agents[Agents on 50K hosts] --> Ingest[Ingest shards]
  Ingest --> Head[(Head · in-memory)]
  Query[Query engine] --> Head
  Dash[Dashboards] --> Query

  classDef store fill:#eef2f1,stroke:#2c5f5d,stroke-width:2px,color:#1f2937;
  class Head store

Storage decides the system’s character: the head/block split — mutable recent data in RAM, immutable history on cheap storage — is the move that makes both 5M writes/sec and 13-month retention affordable on the same system.

4. Architecture evolution

Each level keeps the prior one’s data layout — Level 3 is Level 2’s TSDB sharded and tiered, which is why the open-source ecosystem (Thanos, Cortex, Mimir) could build Level 3 around Prometheus rather than replacing it.

The design in this walkthrough is Level 3, with Level 4’s recording rules named where they solve cardinality.

I’d say out loud: “I’m designing the sharded TSDB — Level 3. The single-node version is the same design in miniature, which is a good sign: scaling here is sharding a sound layout, not changing the layout.”

Target architecture (Level 3)

Every section that follows explains one part of this diagram:

flowchart TD
  Agents[Agents on 50K hosts] --> LB[Ingest router · hash by series]
  LB --> Ing1[Ingester shard 1]
  LB --> Ing2[Ingester shard N]
  Ing1 --> WAL1[(WAL)]
  Ing1 --> Head1[(Head · in-memory)]
  Ing2 --> Head2[(Head · in-memory)]
  Head1 -->|seal 2h blocks| Blocks[(Block store · object storage)]
  Head2 -->|seal 2h blocks| Blocks

  Query[Query engine] --> Head1
  Query --> Head2
  Query --> Blocks
  Alert[Alert evaluator] --> Query
  Dash[Dashboards] --> Query

  classDef store fill:#eef2f1,stroke:#2c5f5d,stroke-width:2px,color:#1f2937;
  class WAL1,Head1,Head2,Blocks store

5. Core design choice: the storage layout

Three options:

(a) Generic store, row per sample. (metric, labels, ts, value) rows in Cassandra or a sharded SQL. Writes scale; but every query re-groups rows into series at read time, the label index is your problem anyway, and 16-byte samples carry tens of bytes of row overhead. You end up rebuilding a TSDB inside a database that fights you.

(b) Purpose-built TSDB layout. Samples append to per-series chunks in an in-memory head block, protected by a WAL; every 2 hours the head seals into an immutable block (chunks + label index for that window) that tiers to object storage. Reads merge head + blocks.

(c) Pre-aggregate only. StatsD-style: compute fixed rollups at ingest, store no raw series. Cheap and fast — and the moment someone asks a question you didn’t pre-aggregate (“p99 by customer for last Tuesday’s incident”), the data doesn’t exist.

Choice: (b), with (c)‘s technique adopted selectively as recording rules for known-expensive queries. The workload-specific justification: samples within one series arrive in timestamp order with near-constant deltas — which is exactly what Gorilla-style compression exploits (delta-of-delta timestamps, XOR’d floats) to hit ~1.5 bytes/sample, a 10× saving available only if storage is organized per-series. Layout (a) throws that locality away.

Tradeoff matrix

Architecture decisions

Core data structures

The label index is literally an inverted index — the same structure as a search engine’s, with labels as terms and series as documents. Queries are set intersections over posting lists.

6. Write path [R1]

flowchart TD
  Agents[Agents] -->|"batch: (series, ts, value)"| LB[Ingest router · hash by series]
  LB --> Ing1[Ingester shard 1]
  Ing1 -->|1 · append| WAL1[(WAL)]
  Ing1 -->|2 · apply| Head1[(Head · in-memory)]
  Head1 -->|3 · every 2h, seal| Blocks[(Block store)]

  classDef store fill:#eef2f1,stroke:#2c5f5d,stroke-width:2px,color:#1f2937;
  class WAL1,Head1,Blocks store

Step by step:

1. Agents batch samples and send every scrape interval. The router hashes the series identity (name + sorted labels) so every sample of a series reaches the same ingester shard — series locality starts here.
2. The ingester appends the batch to its WAL (sequential disk write, the only durability cost on the hot path) and applies it to the head: look up the series (or create it — this is where cardinality is born), append to its active chunk.
3. Compression happens in the append: timestamps store delta-of-delta (for 10-second scrapes, almost always a few bits), values XOR against the previous float (slowly-changing gauges compress to almost nothing).
4. Every 2 hours, the head seals: chunks and that window’s label index write out as an immutable block to object storage; WAL segments for the window are dropped; head memory frees.
5. Replication: each series writes to 2–3 ingesters (the router fans out), so one ingester loss costs nothing and a restart replays only the WAL.

7. Read path [R2, R4]

flowchart TD
  Dash[Dashboard query] --> Query[Query engine]
  Query -->|1 · label selectors| Head1[(Head index)]
  Query -->|1 · label selectors| Blocks[(Block indexes)]
  Query -->|2 · fetch chunks| Head1
  Query -->|2 · fetch chunks| Blocks
  Query -->|3 · aggregate| Dash

  classDef store fill:#eef2f1,stroke:#2c5f5d,stroke-width:2px,color:#1f2937;
  class Head1,Blocks store

Step by step, for p99 of request_latency{service="api"} by region, last 6h:

1. Resolve series. The label index returns posting lists: service="api" ∩ __name__="request_latency" → say 12K series IDs. This intersection is the query’s real cost driver — before a single sample is read.
2. Fetch chunks for those series across the time range: recent windows from the heads (RAM), older from block storage, with block-level caching because immutability makes caches trivially correct.
3. Aggregate: decompress chunks, compute per-series values, then the group-by and percentile. Aggregation pushes down to the shard level — each ingester pre-aggregates its share of series, the query node merges partials, so the fan-in is shards × groups, not series × samples.
4. Dashboards (R4) get two boosts: nearly all their queries hit the head (RAM-speed), and repeated panel queries are served from a short-TTL results cache keyed by (query, time-bucket).

Downsampling (R5) runs as a compaction job over sealed blocks: 10-second raw → 5-minute and 1-hour aggregate blocks (storing min/max/sum/count per window, so rates and averages stay computable). A 13-month query reads 1-hour blocks — ~300× fewer samples — and the query engine picks resolution by range automatically. Retention is then literal: delete raw blocks past 30 days, keep downsampled ones.

8. Deep dives

Cardinality: the failure dimension [R1, R2]

The system’s real capacity limit is series count, and it fails abruptly: one deploy adds a user_id label to one popular metric, and 1K series become 50M overnight — head memory, index size, and query fan-out all multiply at once. This is the metrics equivalent of the news feed’s celebrity problem: the distribution of series-per-metric is power-law, and one bad actor is the whole tail.

Defenses, in the order I’d deploy them: per-tenant/per-metric series limits enforced at ingest (new series beyond the cap are rejected with a visible error — loud beats silent); label allow-lists for high-volume metrics; recording rules that pre-aggregate the expensive shapes teams actually query (Level 4’s technique, applied surgically); and cardinality observability — a dashboard of top series-creating metrics, because the system must be able to monitor itself.

Alerting: queries on a timer [R3]

An alert rule is a query (rate(errors[5m]) / rate(requests[5m]) > 0.01) evaluated every interval, with state: inactive → pending (condition true, waiting out the for-duration) → firing. Running rules through the same query engine as dashboards means an alert is debuggable by pasting it into a graph — operationally worth more than any cleverness in a separate stream-evaluation path. Rules shard across evaluators by rule group; firing alerts deduplicate and group in a notification layer before paging (one rack failure should be one page, not 400).

The 60-second alert latency budget from §1 traces the whole pipeline: scrape interval (10 s) + ingest lag (sub-second) + evaluation interval (15–30 s) + for-duration. Worth saying: the floor on alert latency is the scrape interval, which is why “we’d sample every 60 s to save money” quietly breaks the alerting requirement.

Pull vs push [R1]

Pull (the server scrapes agents’ /metrics endpoints) gives central control of resolution, automatic staleness marking, and — the underrated one — target liveness for free: a failed scrape is the “host down” signal, with no heartbeat machinery. Push is necessary for short-lived jobs (a cron that ends before any scrape) via a gateway, and for sources behind NAT. The senior answer is the boring one: pull by default, push gateway for the exceptions, and the same ingest path behind both.

9. API design [R1, R2, R3]

POST /v1/ingest                                        [R1]
  body: [{ series: {name, labels}, samples: [[ts, value], ...] }, ...]
  returns: { accepted, rejected: [{series, reason: "series_limit"}] }

GET  /v1/query_range?query=...&start=...&end=...&step=...   [R2, R4]
  returns: { series: [{labels, points: [[ts, value], ...]}, ...] }

POST /v1/rules                                         [R3]
  body: { group, interval, rules: [{ alert, expr, for, labels }] }

Two non-obvious decisions:

Ingest returns per-series rejections, not a blanket 200. Cardinality limits (§8) only work if the producer finds out — a silently-dropping metrics system is the most dangerous kind, because the gap looks like calm.

query_range takes an explicit step. The server computes exactly the points a chart will draw (one per pixel-ish), instead of shipping raw samples for the client to decimate — which at 12K series × 6h × 10s resolution is the difference between a 50 KB and a 2 GB response.

The API hides the head/block split, shard count, and downsampling tiers — a query over 13 months and a query over 5 minutes are the same call. That’s the boundary that lets storage re-tier without touching a single dashboard.

10. Data model and storage

Requirements → components

Access frequency

Storage technology choices

Access pattern matrix

Schemas

Series and chunks [R1, R2]

series:
  series_id:   uint64                  # hash of name + sorted labels
  labels:      {name: "request_latency", service: "api", region: "us-east"}

chunk (head, then block):
  series_id, start_ts, end_ts
  encoding:    gorilla (delta-of-delta ts, XOR values)
  samples:     ~120 per chunk

Block [R2, R5]

block (object storage):
  window:      [t, t+2h)               # raw; 5m/1h for downsampled tiers
  chunks/      compressed series chunks
  index        label → posting lists for this window
  meta.json    window, resolution, source blocks
  tombstones   pending deletions (applied at compaction)

11. Failure modes

Grouped by what they threaten. The pattern: name what fails, name what degrades, name what doesn’t.

Ingest failures

An ingester crashes

Fails: one shard’s head. Degrades: nothing visible — the router’s replication (each series on 2–3 ingesters) keeps serving; the restarted node replays its WAL and rejoins. Doesn’t degrade: queries (other replicas answer) or durability beyond the seconds-of-WAL the requirement already tolerated.

Cardinality bomb

Fails: head memory and index growth rate, fleet-wide, within minutes of one bad deploy. Degrades — with limits: the offending metric stops accepting new series, its owner gets rejections with a reason, everyone else is untouched. Without limits: ingesters OOM in sequence and take every tenant down with them. This is the failure that turns §8’s limits from nice-to-have into the system’s actual load-bearing wall. Doesn’t degrade: sealed blocks — history is immutable and safe regardless.

Query failures

Query of death

Fails: a query node’s memory — someone requests 50M series × 30 days at raw resolution. Degrades: that query, by policy — series-count and sample-count budgets estimated before execution from posting-list sizes, hard caps, and per-tenant query queues. Doesn’t degrade: ingest and alerting, which run on separate resources precisely so a dashboard can’t starve a page.

Alerting failures

The alerting path itself fails

Fails: rule evaluation — and this is the system’s worst case, because its output is silence, indistinguishable from health. Degrades: nothing user-visible, which is the problem. The defenses are structural: meta-monitoring (a tiny, independent watchdog that alerts when the alert pipeline stops reporting — a dead-man’s switch, e.g. an always-firing heartbeat alert whose absence pages), evaluator redundancy with leader election per rule group, and a deliberate dependency rule: the monitoring system may not depend on anything it monitors (its own storage, the company’s main Kafka, the service mesh under test). Doesn’t degrade — if the watchdog works: time-to-detection of the monitoring outage itself.

Scrape gaps during network partitions

Fails: samples from the unreachable segment. Degrades: gauges show gaps; rate() over missing windows under-reports — and “absent data” must be a first-class alert condition (absent(up{job="api"})), because a partition that silences a service’s metrics looks exactly like a healthy quiet service otherwise. Doesn’t degrade: data from reachable targets; history.

12. What I’d skip, and say I’m skipping

Time check — five minutes left. Things I’d explicitly defer:

• Logs and traces. Same dashboards, different physics (high-cardinality text vs numeric series). Correlation between pillars is a UI concern here.
• The query language design. I’ve assumed PromQL-shaped selectors and functions; designing the language is its own conversation.
• Notification plumbing. Dedup, grouping, silences, paging integrations — a real subsystem (Alertmanager exists for a reason), but it consumes firing alerts; it doesn’t shape storage.
• Exact consistency of counters across ingester replicas. Replicated ingest can double-count at the margins; queries deduplicate by series at read time, and metrics tolerate approximate — saying why that’s fine here (and wouldn’t be for billing) is the point.
• Multi-region federation. Per-region stacks with a global query layer; the Level 5 row. Name it, move on.

Saying “I’d skip this, and here’s why” is a strong signal. It shows you know the full surface and are making deliberate scoping choices.

Interview flow summary

Architecture at a glance

WRITE PATH                              READ PATH

Agents (scrape/push)                    Dashboard / alert rule
 ↓ batch                                 ↓
Router · hash by series                 Query engine
 ↓                                       ↓ label index: selectors → series IDs
Ingester shard                           ↓
 ↓ WAL append                           Head (recent, RAM) + Blocks (old, S3)
 ↓ head: per-series chunk append         ↓ fetch chunks, decompress
 ↓ (Gorilla compression)                 ↓ aggregate (push-down, then merge)
seal 2h → immutable block → S3          result (+ cache)

The walkthrough order for your whiteboard

1. Requirements        — series count is the question; alert latency budget
2. Capacity            — bytes are trivial; 50M series of index is the load
3. Evolution           — single TSDB → sharded + object storage; target L3
4. Core choice         — TSDB layout vs generic rows vs pre-aggregation
5. Write path          — series sharding, WAL, head, compression, 2h seal
6. Read path           — index intersection, chunk fetch, push-down aggregation
7. Deep dives          — cardinality defenses, alerting-as-queries, pull vs push
8. API                 — per-series rejections, explicit step
9. Data model          — chunks, blocks, inverted label index
10. Failure modes      — ingest / query / alerting (meta-monitoring!)

If the interviewer pushes deeper

13. Wrap-up

One crisp sentence before the interviewer’s next question:

This design treats time series as what they are — append-only, per-series, recency-biased — so writes collapse to in-memory appends with a WAL, history collapses to immutable compressed blocks on cheap storage, and the hard limit becomes cardinality, which is managed as a first-class quota rather than discovered as an outage.

What separates levels on this question

• SDE II pipes agents through Kafka into a wide-column store and puts Grafana on top — a workable shape that misses the series model, pays 10× on storage, and discovers cardinality in production.
• SDE III designs the head/block split, names Gorilla compression and why it requires series locality, shards by series rather than host, treats alert rules as continuous queries with a traced latency budget, and raises cardinality as the real capacity dimension with limits at ingest.
• Staff/Principal designs for the system’s own observability — the dead-man’s switch, the no-circular-dependencies rule, loud per-series rejections; pushes aggregation down to shards and recording rules; reasons about cost per series and who pays it; and knows which corners are safe because it’s metrics (approximate counters, minutes of loss) and which never are (silent alerting failure).

The difference isn’t knowledge. It’s respecting the workload’s shape — weaker answers store samples in a database; stronger ones notice the data is series, the queries are aggregations, and the enemy is cardinality.

Further reading

• Gorilla: A Fast, Scalable, In-Memory Time Series Database — Facebook’s paper; the compression scheme and the recency-biased design argument underneath every modern TSDB.
• Prometheus TSDB format documentation — the head/WAL/block layout this walkthrough’s storage section describes, as actually shipped.
• Site Reliability Engineering, Chapter 6: Monitoring Distributed Systems — Google. What monitoring is for — the four golden signals and the alerting philosophy the requirements section assumes.
• Designing Data-Intensive Applications — Kleppmann. Chapter 3’s storage-engine material (LSM, column layouts, compression) is the theory behind the block design.
• System Design Interview Vol. 2 — Alex Xu. The metrics monitoring and alerting chapter is the interview-prep treatment of this question.
• Thanos documentation — the Level 3 evolution in the open: object-storage blocks, store gateways, and global query federation around Prometheus’s layout.