Reference: Time-Series Storage and Cardinality

What it is

A time series is a sequence of (timestamp, value) samples for one series, identified by a metric name and a set of labels (request_latency{service="api", region="eu"}). Metrics systems, monitoring, and IoT all store millions of series, each written continuously and almost never updated. That workload (append-only, ordered by time, queried by label and time range) rewards a purpose-built layout instead of a generic table.

When you care

“Design a metrics system” asks for a storage layout and a write path, and the follow-up is always “what breaks first?” The answer is cardinality, not volume.

Three layouts

Use pre-aggregation selectively, as recording rules for known-expensive dashboard queries, on top of raw storage.

The write path

1. Agents batch samples and send on each scrape interval. A router hashes the series identity (name plus sorted labels), so every sample of a series lands on the same ingester shard.
2. The ingester appends the batch to its write-ahead log (a sequential write, the only durability cost on the hot path) and applies it to the head: look up the series (creating it if new; this is where cardinality is born) and append to its active chunk.
3. Compression happens in the append. Timestamps are stored as delta-of-delta (for a 10-second scrape, almost always a few bits), and values are XORed against the previous float (slowly changing gauges compress to almost nothing). This Gorilla-style compression reaches about 1.5 bytes per sample, roughly a 10× saving over 16 bytes raw, and it only works if samples are stored per series.
4. Every two hours, the head seals into an immutable block (chunks plus the label index for that window) written to object storage; the log segments for that window are dropped and head memory is freed.
5. Each series is written to two or three ingesters, so losing one costs nothing and a restart replays only the log.

Worked example. 5 million samples per second at 10-second scrapes means about 50 million active series. That is 5,000,000 × 86,400 ≈ 4.3 × 10¹¹ samples a day: about 650 GB a day at 1.5 bytes per sample, versus about 6.9 TB a day at 16 bytes. The high write rate never touches a database; it is a hash lookup and an in-memory append, with one sequential log stream per shard. The expensive step (building immutable, indexed, compressed blocks) happens every two hours in bulk, off the hot path.

The read path

1. Resolve series. The label index is an inverted index: labels are terms and series are documents. A query’s selectors (service="api" and __name__="request_latency") are set intersections over posting lists, returning perhaps 12,000 series ids. This intersection is the query’s real cost driver, before a single sample is read.
2. Fetch chunks for those series and the time range: recent data from the heads (RAM), older data from blocks, with block-level caching (immutability makes caches trivially correct).
3. Aggregate with push-down: each shard pre-aggregates its share of series and the query node merges partial results, so the fan-in is shards × groups, not series × samples.

Downsampling. A compaction job rewrites sealed blocks at lower resolution: 10-second raw to 5-minute and 1-hour aggregates, storing min, max, sum, and count per window so averages and rates remain computable. A 13-month query reads the 1-hour blocks, about 360× fewer samples than raw, and the query engine picks the resolution by range. Retention becomes literal: delete raw blocks after 30 days, keep downsampled ones.

Cardinality: the failure dimension

A series exists for every distinct label combination. The system fails abruptly when that count explodes: one deploy adds a user_id label to a popular metric and 1,000 series become 50 million overnight, multiplying head memory, index size, and query fan-out at once. It is the metrics equivalent of a celebrity problem: series-per-metric is power-law distributed and one bad label is the whole tail.

Worked example. A request counter labeled by service (5), endpoint (20), status class (5), and customer (100) has 5 × 20 × 5 × 100 = 50,000 series. Drop the customer label (high-cardinality detail belongs in logs and traces) and it is 500.

Defenses, in the order to deploy them:

1. Per-tenant and per-metric series limits enforced at ingest, with new series beyond the cap rejected loudly.
2. A cardinality dashboard that shows the top metrics by series count, so a bad label is visible before it is an outage.
3. Label allow-lists and review for new labels.
4. Recording rules that pre-aggregate the shapes people actually query.

When to pick what

Related

• Walkthrough: Designing a Metrics & Monitoring System — the full metrics design this storage layout serves.
• Reference: Inverted Index and Segment-Based Indexing — the label index is an inverted index: labels are terms, series are documents.
• Reference: Stream Processing — Windows, Event Time, Watermarks, and Exactly-Once — windowed aggregation, the other half of a monitoring pipeline.