1 SERVICE · 30 MINUTES · 402,329 READINGS · 1 ALERT SENT

Prometheus:
how one bad deploy becomes one alert

Prometheus collects numbers from running software every few seconds and stores each one with the time it was read. You ask it questions about those numbers, and it can send a message when an answer looks wrong.

WHAT THIS TUTORIAL COUNTS

A small web service ran for 30 minutes while Prometheus read numbers from it, and from the programs around it, every 15 seconds. Each number saved with its time is one sample.

samples stored in those 30 minutes402,329
  • your service, 2 copies11,650 samples
    what your own code counts: requests, errors and how long each request took
THE OTHER 390,679 · FROM THE PROGRAMS AROUND IT
  • the machine it runs on172,080 samples
    CPU, memory, disk and network counters, read from Linux by a helper called node exporter
  • Prometheus, about itself122,646 samples
    its own request counts, timings and storage sizes
  • Alertmanager, about itself85,136 samples
    the program that turns alerts into messages also reports on its own work
  • Pushgateway6,619 samples
    keeps the result of a 3-second cleanup job, which exits before Prometheus can ask it
  • scrape health3,590 samples
    a record of each read: whether the source answered, how long it took, how many numbers came back
  • rules608 samples
    answers Prometheus computed from other samples and saved as new ones
MESSAGES SENT TO A PERSON · 2
the page, “Checkout is failing 12.7% of requests”19:40:39 UTC
the all-clear19:50:39 UTC
measured on 13 November 2026, 19:23:00 → 19:53:00 UTC · Prometheus 3.16.0

This tutorial follows one small web service, checkout, for 30 minutes. In that time a broken version is deployed to one of its two copies, every payment sent to that copy starts to fail, and the version is rolled back 8 minutes after the deploy began. Each number Prometheus saves with the time it was read is one . There are 402,329, and your own service produces 11,650 of them. Three alerts fire along the way and one reaches a person, 4 min 9 s after the first failed payment. Everything between the first sample and that message is a query run on a timer.

You need to know what an HTTP request and a percentage are; no monitoring experience is assumed. Every number is from that one run, where a small webhook logger stands in for a real pager. The scenes play themselves, so watch or just scroll. are tappable. The exam angle is the Prometheus Certified Associate curriculum: each chapter’s deep dive ends with the questions it answers.

One request leaves three kinds of record

A customer pays for a cart at 19:22:30 UTC, half a minute before Prometheus starts. The request takes 224 milliseconds and succeeds. What is left of it afterwards? Three kinds of record: a line in a log, a timed trace of the work, and a handful of numbers in the service’s memory that each went up a little. Prometheus handles only the numbers. This chapter compares the three and picks out the numbers worth keeping. A number becomes a sample when Prometheus saves it with the time it was read. Nothing has read these yet, so the sample count is 0.

STEP 01 / 04

ONE REQUEST, THREE RECORDS

LOG LINE · 1 record
{"level": "info", "msg": "request", "method": "POST", "path": "/checkout", "status": 200, "duration_ms": 224.1, "trace_id": "a3ad5d975a096a66d7c6968acd5460c5", "version": "2.4.0", "ts": "2026-11-13T19:22:30.314Z", "instance": "checkout-1"}
one JSON record for this request
TRACE · 2 spans
POST /checkout · 224.1 ms
payment.charge · 210.6 msthe payment call, inside the request
both spans carry trace ID a3ad5d97…
METRICS · 8 totals moved, 3 shown
  • http_requests_total{…}+1requests handled
  • http_request_duration_seconds_sum{…}+0.2241seconds spent on requests, added up
  • http_request_duration_seconds_bucket{le="0.25",…}+1requests done within 0.25 s
running totals only: the request itself is not recorded
Fig 0.1Read the three cards left to right: the accent card holds no record of this request, only running totals that moved.

At 19:22:30.090 UTC, POST /checkout reaches checkout-1, one of the service’s two copies, and takes 224.1 ms. The service writes one log line for it: a JSON record with the path, status 200, the duration and a trace ID.

It also records two spans, each a timed piece of work: the whole request at 224.1 ms and, inside it, the call payment.charge at 210.6 ms. Both carry the same trace ID, and together they are this request’s trace.

Last, the request changes numbers held in the process’s memory. A request counter rises by 1 and a duration total by 0.2241 seconds. Each of those is a : a named number that keeps a running total. Logs and traces keep one record per event. A metric keeps none, only the totals.

STEP 02 / 04

A LOG GROWS WITH TRAFFIC, A METRIC DOES NOT

logs23,060 lines · 5,792,779 bytes
one record or more per request: 14,401 requests handled
metrics5,950 samples · about 13,000 bytes
50 numbers kept from each of 119 reads of an 87-line page: the same at any traffic
bar length = bytes kept from checkout-1 over the 30 minutes
Fig 0.2Compare the two bars, drawn to the same scale: the accent bar, the stored samples, is the thin one.

Every request adds to the log, and no request adds a line to the metrics. In the 30 minutes of this run, checkout-1 handles 14,401 requests and writes 23,060 log lines: 5,792,779 bytes. Its metrics page, the text that lists its current numbers, stays 87 lines long the whole time.

Prometheus reads that page 119 times and keeps 50 numbers from each read: 5,950 samples, about 13,000 bytes on disk. Ten times the traffic would write ten times the log and the same 5,950 samples. A counter that reads 4,030 instead of 403 is still one number.

The price is detail. A total can show that requests are failing, and how many. It cannot say which request failed: that record is in the log line and the trace.

STEP 03 / 04

FOUR QUESTIONS WORTH A NUMBER

  • latencyhow long a request takeshttp_request_duration_seconds0.245 s99% of requests finish within this
  • traffichow many requests arrivehttp_requests_total16.0requests a second, both copies
  • errorshow many of them failhttp_requests_total{status="500"}0%of requests answered with status 500
  • saturationhow full the service ischeckout_inflight_requests0.9requests in flight on checkout-1, 5-minute average
read at +11:00, 11 min after Prometheus starts and 1 min before the deploy
Fig 0.3Each row pairs one signal with the metric that answers it; the accent row, errors, is what the alert is built on.

Which totals are worth keeping? Google’s Site Reliability Engineering book names four, the golden signals. Latency is how long a request takes. Traffic is how many requests arrive. Errors is how many of them fail. Saturation is how close the service is to full.

The checkout service answers all four with three metrics. http_requests_total gives traffic, and errors through its status label. http_request_duration_seconds gives latency. checkout_inflight_requests, the number of requests being handled right now, gives saturation. At +11:00, 11 minutes after Prometheus starts, they read 16.0 requests a second, 0% errors, 99% of requests done within 0.245 s, and 0.9 requests in flight.

Two shorter lists cover the same ground. RED (rate, errors, duration) is the list for request-driven services like this one. USE (utilisation, saturation, errors) is the list for resources such as CPU and disk.

STEP 04 / 04

A PROMISE WITH A NUMBER IN IT

error budget at 99.9%41,472 failed requests allowed in 30 days
1,661 failed requests · 4.0% of the budget, spent in 8 min39,811 still allowed this month
941 answered with status 500 + 720 refused while checkout-2 was stopped
  • SLIservice level indicatorwhat you measuresuccessful requests ÷ all requests
  • SLOservice level objectivethe target you set99.9% over 30 days
  • SLAservice level agreementthe contract you signedwhat you owe when the target is missed
Fig 0.4The long bar is one month’s error budget at 99.9%; the accent slice is what this run’s 8-minute incident spends.

An SLI (service level indicator) is a measured ratio: here, successful requests divided by all requests. An (service level objective) is the target you set for it: 99.9% over 30 days. An SLA (service level agreement) is a contract that says what you owe if the target is missed.

The error budget is what the SLO leaves over: the 0.1% of requests that may fail. Over 30 days that is 43 min 12 s of complete failure. At 16 requests a second it is 41,472 failed requests out of 41,472,000.

The incident in this tutorial fails 1,661 requests: 941 answered with status 500 and 720 refused outright. That is 4.0% of the month’s budget, spent in 8 minutes.

WHAT YOU NOW KNOW

  • One request left a log line, two spans and eight changed totals; Prometheus stores only the totals.
  • In 30 minutes one copy wrote 5.8 MB of logs and 5,950 samples; the sample count does not grow with traffic.
  • Measure latency, traffic, errors and saturation; a 99.9% SLO at 16 requests a second allows 41,472 failures a month. The sample count is still 0.
deep dive: events, structured logs, OpenTelemetry and exemplars

An event is one discrete thing that happened: a request, a deploy, a crash. A log line is how most programs write an event down. A structured log writes it as named fields, as the JSON record in step 1 does, so a tool can filter on status or trace_id without parsing sentences. A per-event record costs storage in proportion to traffic, and a total costs the same at any traffic. That is why metrics can cover every request while logs and traces are often kept for only some.

A trace needs every service on a request’s path to use one trace ID. The caller sends it in an HTTP header, traceparent, defined by the W3C Trace Context standard, and each service passes it on. That is context propagation, and the load generator in this run sends the header on every request. Storing every trace is expensive, so tracing systems usually sample: they keep a fraction of them.

OpenTelemetry is a vendor-neutral project with one SDK and one wire protocol, OTLP, for traces, metrics and logs. Prometheus 3 can receive OTLP metrics at /api/v1/otlp/v1/metrics when it is started with --web.enable-otlp-receiver. An exemplar is a trace ID attached to a histogram bucket: the link from a point on a graph to one request that landed there.

Monitoring asks questions you already had, on a timer. Observability is being able to ask new ones of a running system without shipping new code. Burn rate says how many times faster than allowed the error budget is being spent: at a burn rate of 1 the budget lasts exactly the SLO’s 30 days, and at 10 it is gone in 3 days.

ON THE EXAM

  • “the three pillars of observability” → metrics, logs, traces
  • “numeric, aggregated, cheap to keep” → metrics
  • “one record per event” → logs
  • “follows one request across services” → a trace, made of spans
  • “one timed operation with a parent” → a span
  • “four golden signals” → latency, traffic, errors, saturation
  • “RED” → rate, errors, duration
  • “USE” → utilisation, saturation, errors
  • “SLI, SLO, SLA” → measurement, target, contract
  • “error budget” → 1 minus the SLO

Your code keeps the count, and a page at /metrics shows it

The checkout service works, and it reports nothing about itself: no request count, no timings. Instrumenting it means adding a few lines that count as the code runs, plus one URL, /metrics, that prints the current counts as text. This chapter adds both, then reads the same kind of page from a Linux machine. Nothing is sent anywhere and Prometheus has not started, so the sample count stays 0.

STEP 01 / 06

FOUR LINES OF PYTHON

ONE REQUEST THROUGH handle()
  1. in flight +1IN_FLIGHT.track_inprogress(), on the way in
  2. work, 224 msdo_work("POST", "/checkout")
  3. in flight −1the with block ends
  4. requests{POST, /checkout, 200} +1REQUESTS.labels(…).inc()
  5. duration observe 0.2241DURATION.labels(…).observe(elapsed)
change numbers in
PROCESS MEMORY · 4 METRICS DEFINED
  • http_requestscounter · 5 label sets
  • http_request_duration_secondshistogram · 4 label sets
  • checkout_inflight_requestsgauge · no labels, 1 number
  • checkout_buildinfo · 1 label set
+ 10 metrics the library adds about the process itself: process_*, python_*
Fig 1.1Follow one request down the left column: the two accent rows record it, and both change numbers in memory, nothing else.

A client library is a package that gives your program metric objects, held in the program’s own memory. Counter("http_requests", …, ["method", "path", "status"]) defines a counter that keeps a separate number for each method, path and status. .labels("POST", "/checkout", "200").inc() adds 1 to one of those numbers. It makes no network call and writes no file.

Prometheus maintains official libraries for Go, Java, Python, Ruby and Rust. This service uses the Python one, prometheus_client 0.26.0, and defines 4 metrics; the sample below shows three of them. The library adds 10 more about the process itself, with names starting process_ and python_: 14 in all.

At startup the service asks the library for each label combination once, which creates every number at 0. The count of failed checkouts, status 500, is therefore on the page, reading 0, before the first failure happens.

CHECKOUT.PY: THREE METRICS DEFINED, AND THE HANDLER THAT UPDATES THEM
import time
from prometheus_client import Counter, Gauge, Histogram

REQUESTS = Counter("http_requests", "HTTP requests handled.", ["method", "path", "status"])
DURATION = Histogram(
    "http_request_duration_seconds",
    "Time spent handling an HTTP request.",
    ["method", "path"],
    buckets=[0.05, 0.1, 0.25, 0.5, 1, 2.5],
)
IN_FLIGHT = Gauge("checkout_inflight_requests", "Requests being handled right now.")

def handle(method, path):
    start = time.time()
    with IN_FLIGHT.track_inprogress():      # gauge +1 now, -1 on the way out
        status = do_work(method, path)
    elapsed = time.time() - start           # 0.2241 for the request in chapter 00
    REQUESTS.labels(method, path, str(status)).inc()   # one number in memory, +1
    DURATION.labels(method, path).observe(elapsed)     # le="0.25" and every larger bucket
    return status
STEP 02 / 06

A COUNTER ONLY GOES UP

counter · http_requests_total{path="/checkout",status="200"}
01,0002,000
1,995 requests
+0:20+5:00+11:00
gauge · checkout_inflight_requests
012
1 in flight
+0:20+5:00+11:00
both read from checkout-1 every 15 seconds · the counter already read 489 at its first reading, because the process started before Prometheus did
Fig 1.2The accent line climbs and never falls; the grey line under it, requests in flight, moves both ways.

A only goes up, and starts again from 0 when the process restarts. A car’s odometer works the same way: the reading alone tells you little, and the difference between two readings is the distance driven.

The counter of successful checkouts on checkout-1 reads 403 when its page is captured, 2 min 43 s after the process started. At +10:50, counted from the second Prometheus starts, it reads 1,995. Neither number means much alone, because it depends on when you look. Only the change is useful. By convention a counter’s name ends in _total.

A moves both ways. checkout_inflight_requests, the number of requests being handled at this instant, reads 0, 1 or 2 in this run, and its current value is meaningful on its own. The rule for choosing between the two: if the value can go down, it is a gauge.

STEP 03 / 06

A HISTOGRAM IS SEVEN COUNTERS

http_request_duration_seconds_bucket{method="POST",path="/checkout",le="…"}
  1. le="0.05"0 requests
  2. le="0.1"0 requests
  3. le="0.25"403 requests
  4. le="0.5"403 requests
  5. le="1.0"403 requests
  6. le="2.5"403 requests
  7. le="+Inf"403 requests
the 224.1 ms request added 1 to each accent bucket
_sum · every duration added up73.898 seconds
_count · requests observed403 requests
7 bucket counters + _sum + _count = 9 series per label set
Fig 1.3Each bar is one bucket counter for POST /checkout; the five accent bars are the ones the 224 ms request raised.

A counts observations into buckets. Each bucket is a counter with a label le, for “less than or equal”: le="0.25" counts every request that took 0.25 s or less. A fast request is therefore counted in every larger bucket too: buckets are cumulative.

This service sets six boundaries: 0.05, 0.1, 0.25, 0.5, 1 and 2.5 seconds. The library adds a seventh bucket, +Inf, that counts every request. With a _sum of all the durations and a _count, that is 9 numbers for each method and path.

The 224 ms request added 1 to five buckets: 0.25, 0.5, 1.0, 2.5 and +Inf. When the page is captured it shows 403 checkouts: none within 0.1 s, all within 0.25 s, 73.898 s in total. A summary, the other type for durations, computes quantiles (the median, for one) inside the process instead, as go_gc_duration_seconds{quantile="0.5"} on Prometheus’s own page does. Quantiles cannot be added across copies; bucket counters can.

STEP 04 / 06

THE PAGE AT /metrics

GET checkout-1:8000/metrics
  1. # HELP http_requests_total HTTP requests handled.
  2. # TYPE http_requests_total counter
  3. http_requests_total{method="GET",path="/cart",status="200"} 585.0
  4. http_requests_total{method="POST",path="/cart/items",status="200"} 291.0
  5. http_requests_total{method="POST",path="/checkout",status="200"} 403.0
  6. http_requests_total{method="POST",path="/checkout",status="500"} 0.0
  7. http_requests_total{method="GET",path="other",status="404"} 18.0
  8. … 80 more lines
87 lines · 5,641 bytes · 59 samples · 14 metric families
Fig 1.4Read the accent line as name, labels, value: everything after the closing brace is one 64-bit float.

GET /metrics returns the current value of every metric as plain text. On checkout-1 that is 87 lines and 5,641 bytes: 59 values in 14 metric families, a family being one metric name with all of its label combinations. Each family starts with a # HELP line (a description) and a # TYPE line (the type), then one line per value: a name, labels in braces and a number.

This layout is the Prometheus text , version 0.0.4. The lines carry no timestamps: Prometheus adds the time when it reads the page, and that pairing is what makes a value a sample. Asked for the newer OpenMetrics format, the same service returns 88 lines, the last one # EOF.

The page is not a file. The library builds it from memory each time it is asked, so it shows the present and never the past. Keeping history is the job of whatever reads the page, and chapter 02 starts there.

STEP 05 / 06

NAMES CARRY THE UNIT

  • http_request_duration_seconds
    • httpwhat it belongs to
    • request_durationwhat it measures
    • secondsthe unit: a base unit, never milliseconds
  • http_requests_total
    • httpwhat it belongs to
    • requestswhat it counts
    • totalsuffix: this metric is a counter
  • checkout_build_info
    • checkoutthe application
    • buildwhat it describes
    • infosuffix: the value is always 1, the labels carry the facts
A VALUE IN THE NAME
http_500_requests_totalone more metric for every status
write instead
THE VALUE IN A LABEL
http_requests_total{status="500"}one metric, filtered by status
Fig 1.5Read the accent name in parts: what it belongs to, what it measures, the unit; the second name’s suffix marks a counter.

A metric name is lowercase words joined by underscores, in a fixed order. First comes a prefix for the application or domain: checkout_ or http_. Then what is measured. Then the unit, always a base unit: seconds and bytes, never milliseconds or megabytes. A counter ends in _total.

_info marks a metric whose value is always 1 and whose labels carry the facts: checkout_build_info{version="2.4.0"} 1.0. A is a key and value attached to a metric, and whatever varies belongs in one, never in the name. Write http_requests_total{status="500"}, not http_500_requests_total.

Every distinct label value creates one more number to store, so keep the set of values small. Here path holds the route, and any URL the service has no route for is counted as path="other". The 18 requests for /favicon.ico before the capture share one line on the page instead of adding a line each.

STEP 06 / 06

EXPORTERS: NUMBERS FROM CODE YOU CANNOT EDIT

EXPORTER · FOR CODE YOU CANNOT EDIT
Linux kernelits own counters, as files under /proc and /sys
read by
node exporter :9100a separate process, version 1.12.1
served as
/metrics2,052 lines · 1,434 samples
DIRECT INSTRUMENTATION · FOR CODE YOU WRITE
checkout.py + client librarycounts as it runs, in its own memory
served as
/metrics87 lines · 59 samples
Fig 1.6The accent box translates: kernel counters go in on the left, a /metrics page comes out on the right.

You cannot import a client library into the Linux kernel, a database or a network switch. An is a separate process that reads such a system’s own counters and serves them as a /metrics page.

node exporter 1.12.1 does this for a Linux machine. It reads two kernel directories, /proc and /sys, and serves 2,052 lines: 1,434 values in 309 families. One counter alone, node_cpu_seconds_total, accounts for 96 of them: 12 CPUs times 8 modes, such as idle and user.

Programs written with Prometheus in mind need no exporter. They instrument themselves directly, as checkout does, and as Prometheus (802 values on its own page) and Alertmanager (677) do. Common exporters are node for hosts, blackbox for probing a URL from outside, mysqld and postgres for databases, and cAdvisor for containers.

WHAT YOU NOW KNOW

  • A client library keeps metric objects in the process’s memory; inc() and observe() change numbers there and send nothing.
  • Counters only go up, gauges move both ways, and a histogram is 7 bucket counters plus a sum and a count.
  • /metrics prints the current values as text: 87 lines and 59 samples here, 1,434 samples from node exporter. Nothing is stored yet.
deep dive: bucket choices, native histograms and the textfile collector

Default buckets differ by library: Python’s has 14 finite boundaries from 0.005 to 10 seconds, and Go’s DefBuckets has 11. Neither set is tuned to your service, so put one boundary exactly at your SLO threshold; the share of requests that met it is then a ratio of two counters. Choose a histogram unless you need an exact quantile for a single process, which is the one thing a summary does better.

Native histograms replace the 9 series this histogram needs per label set with one series that carries many automatically sized buckets. They have been stable since Prometheus 3.8 (November 2025), and you opt in per scrape job with scrape_native_histograms: true. The Python library can also expose a _created series holding each metric’s creation time; this run switches it off with PROMETHEUS_DISABLE_CREATED_SERIES.

Label names must match [a-zA-Z_][a-zA-Z0-9_]*, and names starting with __ are reserved for Prometheus’s own use. UTF-8 metric and label names have been allowed since 3.0. OpenMetrics 1.0, published in 2020, is the first format Prometheus 3 asks a target for. The OpenMetrics project was archived at the CNCF on 2024-07-02, and its specification now lives under Prometheus, where version 2.0 is a release candidate.

node exporter also has a textfile collector: it serves the metrics it finds in text files in a directory you name, which is the usual way to publish the result of a cron job. Exporters listen on registered default ports: node on 9100, blackbox on 9115.

ON THE EXAM

  • “metric type that can decrease” → gauge
  • “type for durations you want to aggregate across instances” → histogram
  • “quantiles computed in the client, cannot be aggregated” → summary
  • “series a histogram exposes” → _bucket{le}, _sum, _count
  • “suffix on counters” → _total
  • “unit for durations” → seconds
  • “where does a varying dimension go” → a label
  • “monitor software you cannot change” → an exporter
  • “host metrics” → node exporter
  • “probe an endpoint from outside” → blackbox exporter
  • “lines starting with # on a metrics page” → HELP and TYPE
  • “official client libraries” → Go, Java, Python, Ruby, Rust

Prometheus fetches each page every 15 seconds

The counts exist now, inside two Python processes and four other programs, and nothing has read them. Each page shows only its current values, so no history exists anywhere yet. Prometheus reads the counts by fetching each /metrics page over HTTP on a timer, which is called scraping. It starts at 19:23:00 UTC with an empty database. After 20.2 seconds it has read all 6 pages once, and the sample count has gone from 0 to 3,088.

STEP 01 / 05

PROMETHEUS MAKES THE CALL

Prometheusholds the target list · scrapes every 15 s
GET /metrics
REQUEST HEADERS
  • User-Agent:Prometheus/3.16.0
  • Accept-Encoding:gzip
  • Accept:application/openmetrics-text;version=1.0.0first of 4 formats
  • X-Prometheus-Scrape-Timeout-Seconds:10seconds Prometheus will wait
200 OK · 5,576 bytes · scrapes average 1.1 ms
checkout-1:8000your service · answers a GET · nothing in it names Prometheus
Fig 2.1Watch the request leave Prometheus and the page come back; the accent header tells the target how long Prometheus will wait.

Prometheus sends GET /metrics to every address on its list and stores what comes back. One such request is a , and each address is a . This is the pull model: the monitoring system asks and the service only answers. Nothing in the checkout service’s code or config names Prometheus.

The first request to checkout-1 carried User-Agent: Prometheus/3.16.0, Accept-Encoding: gzip, an Accept header listing 4 formats with OpenMetrics first, and X-Prometheus-Scrape-Timeout-Seconds: 10. That last header is how long Prometheus will wait, in seconds. The answer was 5,576 bytes, and scrapes of this target average 1.1 ms.

Pull has a consequence that push does not. Prometheus holds the list of what should exist, so a page that does not answer is itself a result: the scrape fails, and the failure is recorded. With push, a service that has stopped sending looks the same as a service with nothing to report.

STEP 02 / 05

ONE FILE SAYS WHAT TO READ

  • jobprometheusinstancelocalhost:9090/metrics802 samples on the page
  • jobalertmanagerinstancealertmanager:9093/metrics677 samples on the page
  • jobnodeinstancenode:9100/metrics1,434 samples on the page
  • jobcheckoutinstancecheckout-1:8000/metrics59 samples on the page
  • jobcheckoutinstancecheckout-2:8000/metrics59 samples on the page
  • jobpushgatewayinstancepushgateway:9091/metrics45 samples on the page
5 jobs · 6 targets · every sample read from a target gets its job and instance
Fig 2.2Each card is one target; its two accent fields, job and instance, are labels Prometheus adds to everything it reads there.

What to scrape is written in one file, prometheus.yml. global.scrape_interval: 15s sets how often. The default is 1 minute, with a 10-second timeout. scrape_configs is a list of jobs, and a job is a named group of targets scraped the same way. This run has 5 jobs and 6 targets, because the checkout job has two copies.

Prometheus attaches two labels to every sample it reads from a target, called . job is the job’s name from the config. instance is the target’s host and port. All 59 samples on one checkout page arrive stamped job="checkout", instance="checkout-1:8000", which is how you later tell the two copies apart.

You did not write the path or the scheme. /metrics and http are the defaults, so a job needs only a name and its targets.

PROMETHEUS.YML: BOTH INTERVALS AND THREE OF THE FIVE JOBS
global:
  scrape_interval: 15s        # default is 1m
  evaluation_interval: 15s    # default is 1m

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: node
    static_configs:
      - targets: ["node:9100"]
  - job_name: checkout
    file_sd_configs:
      - files: ["targets/checkout.json"]
    metric_relabel_configs:       # runs on samples, after the scrape
      - source_labels: [__name__]
        regex: python_gc_.*
        action: drop              # 59 samples on the page, 50 stored
STEP 03 / 05

TARGETS FROM A FILE THAT CHANGES

DISCOVERED · 7 LABELS
__address__="checkout-1:8000"one entry of targets
job="checkout"from job_name
env="prod"from the file’s labels
__meta_filepath="/etc/prometheus/targets/checkout.json"the file it came from
__metrics_path__="/metrics"default
__scheme__="http"default
__scrape_interval__="15s"from global
relabeling
TARGET LABELS · 3 LABELS
instance="checkout-1:8000"copied from __address__
job="checkout"kept
env="prod"kept
dropped before the scrapethe other 4 all start with two underscores, and so does __address__: such labels exist only during discovery
Fig 2.3Left is what discovery found, right is what the target keeps; the accent row is the address becoming the instance label.

static_configs writes addresses into the config file. Addresses that change come from . file_sd_configs reads a JSON file and reads it again whenever it changes, with no restart. kubernetes_sd_configs, ec2_sd_configs, consul_sd_configs, dns_sd_configs and http_sd_configs ask a platform for the list instead.

Discovery hands Prometheus each target as a set of labels, and the ones it creates start with two underscores. For checkout-1 they include __address__="checkout-1:8000", __metrics_path__="/metrics", __scheme__="http", __scrape_interval__="15s" and __meta_filepath, the file the target came from.

turns that set into the target’s final labels. By default __address__ is copied to instance, and env="prod" stays because the file set it. Every label that still starts with two underscores is then removed, so none of them reaches a stored sample.

TARGETS/CHECKOUT.JSON: PROMETHEUS RE-READS THIS FILE WHEN IT CHANGES
[
  {
    "targets": ["checkout-1:8000", "checkout-2:8000"],
    "labels": { "env": "prod" }
  }
]
STEP 04 / 05

THE FIRST 3,088 SAMPLES

0 s = Prometheus starts, 19:23:00 UTC · one scrape interval is 15 s
0 s5 s10 s15 s20 s
checkout-2+7.5 s · 55 samples
prometheus+13.4 s · 807 samples
node+14.0 s · 1,439 samples
alertmanager+14.5 s · 682 samples
pushgateway+18.0 s · 50 samples
checkout-1+20.2 s · 55 samples
samples stored0
Fig 2.4Each mark is one target’s first scrape on the 15-second ruler; the count under it reaches 3,088 as the last one lands.

Prometheus does not scrape all 6 targets at the same instant. Each target gets a fixed offset inside the 15-second interval, which spreads the work out. In this run the first scrapes land at +7.490 s (checkout-2), +13.430 s (Prometheus itself), +14.040 s (node exporter), +14.531 s (Alertmanager), +17.998 s (Pushgateway) and +20.243 s (checkout-1).

Those pages yield 50 + 802 + 1,434 + 677 + 45 + 50 = 3,058 samples. Prometheus then adds 5 of its own per scrape, which appear on no page: up (1 if the scrape worked, 0 if it failed), scrape_duration_seconds, scrape_samples_scraped, scrape_samples_post_metric_relabeling and scrape_series_added. Across the 6 scrapes that is 30 more, so the sample count is 3,088.

A checkout page has 59 samples, yet only 50 are stored. A metric_relabel_configs rule in the config removes the 9 python_gc_* samples after the scrape and before storage. A dropped sample is never stored, so it uses no memory and no disk.

STEP 05 / 05

THE JOB THAT EXITS BEFORE ANYONE ASKS

+0:00 = Prometheus starts, 19:23:00 UTC
+9:45+10:00+10:30+11:00
runs 3.2 s, pushes, exits
cart-cleanupthe batch job
push arrives +10:03.6: 3 values held from here on
Pushgatewayits /metrics page
+10:18.0 first scrape that returns the result
Prometheusscrapes the gateway every 15 s
HELD BY THE PUSHGATEWAY FROM +10:03.6 ON
cart_cleanup_carts_deleted37 carts
cart_cleanup_duration_seconds3.2004 s
cart_cleanup_last_success_timestamp_seconds1794598383.594, which is 19:33:03 UTC
scraped 79 times by +30:00, the same values every time
Fig 2.5The job’s whole life is the short accent bar; the marks after it are Prometheus reading the stored result every 15 seconds.

The batch job cart-cleanup starts at +10:00, ten minutes after Prometheus did, runs for 3.2 seconds and exits. A scrape every 15 seconds would almost never find it running, and its numbers disappear with the process. So the job pushes them: one HTTP request to the , a small server that keeps pushed metrics on its own /metrics page.

Prometheus scrapes the gateway like any other target and stores three values: 37 carts deleted, a duration of 3.2004 s, and cart_cleanup_last_success_timestamp_seconds, the Unix time the job finished (19:33:03 UTC). honor_labels: true in the scrape config keeps the pushed label job="cart-cleanup", which Prometheus would otherwise replace with job="pushgateway".

There are two catches. The gateway keeps serving the last push until someone deletes it: by the end of this run Prometheus has scraped the same three values 79 times. And up describes the gateway, not the job, so a cleanup that never runs again still looks healthy. Chapter 05 turns the timestamp into an age to catch that.

CART-CLEANUP: SET THE GAUGES, PUSH ONCE, EXIT
from prometheus_client import CollectorRegistry, Gauge, push_to_gateway

registry = CollectorRegistry()
last_success = Gauge(
    "cart_cleanup_last_success_timestamp_seconds",
    "Unix time the cart cleanup last finished successfully.",
    registry=registry,
)
deleted = Gauge("cart_cleanup_carts_deleted", "Abandoned carts deleted.", registry=registry)

deleted.set(delete_abandoned_carts())   # 37 carts, 3.2004 s of work
last_success.set_to_current_time()      # 1794598383.59, which is 19:33:03 UTC
# one HTTP request; the process exits right after
push_to_gateway("pushgateway:9091", job="cart-cleanup", registry=registry)

WHAT YOU NOW KNOW

  • Prometheus pulls: one GET /metrics per target every 15 seconds, set in scrape_configs; the default interval is 1 minute.
  • Targets come from static lists or service discovery; relabeling turns __address__ into instance, and each scrape adds up and four scrape_* series.
  • All 6 targets answered within 20.2 seconds, taking the sample count from 0 to 3,088. Jobs too short to scrape push to the Pushgateway.
deep dive: relabel actions, scrape protocols and reloading the config

A relabel rule has an action. replace writes a label value, keep and drop filter by a regular expression, labelmap copies label names that match a pattern, labeldrop and labelkeep remove labels, and hashmod splits targets across several servers. relabel_configs act on targets before the scrape. metric_relabel_configs act on samples after it, which is how this run drops python_gc_*.

Each scrape starts with content negotiation. Prometheus 3 asks for OpenMetrics 1.0 first and accepts the text format 0.0.4 as a later choice. It fails a scrape whose Content-Type it cannot parse unless fallback_scrape_protocol is set. scrape_timeout must not exceed scrape_interval, and sample_limit fails a scrape that returns more samples than you allow. honor_labels keeps the labels a page already carries when they clash with target labels, and honor_timestamps keeps the timestamps a page supplies. TLS settings and credentials are set per scrape job.

The config can be reloaded without a restart: send the process SIGHUP, or POST /-/reload when it was started with --web.enable-lifecycle. Check the file first with promtool check config prometheus.yml. In this run it printed SUCCESS: 1 rule files found and SUCCESS: 8 rules found. On Kubernetes, discovery has six roles: node, pod, service, endpoints, endpointslice and ingress.

Pull detects a dead target, because a failed scrape is recorded, and it needs a network route from Prometheus to every target. Push suits jobs behind a firewall or too short to scrape. The Pushgateway documentation gives three warnings: it is a single point of failure, there is no up per job, and it “never forgets” the series pushed to it until you delete them. Agent mode (--agent) scrapes, forwards every sample to another system with remote_write and stores nothing locally.

ON THE EXAM

  • “how Prometheus collects metrics” → it pulls over HTTP
  • “label taken from job_name” → job
  • “label holding host and port” → instance
  • “series that shows a target answered” → up
  • “default scrape_interval” → 1m
  • “default metrics path” → /metrics
  • “add targets without a restart” → file_sd_configs or another discovery mechanism
  • “drop a metric after scraping” → metric_relabel_configs with action: drop
  • “change target labels before scraping” → relabel_configs
  • “short-lived batch job” → Pushgateway
  • “Pushgateway pitfall” → keeps old values, no per-job up, single point of failure
  • “keep the pushed job label” → honor_labels: true
  • “validate the config” → promtool check config

A sample is 16 bytes, stored in about 2

Every 15 seconds about 3,300 more numbers arrive. Prometheus keeps each one as a sample, a timestamp and a value, filed under the series it belongs to, in a database built into the server. Ten minutes in, at +10:00, that is 131,801 samples in 3,329 series, and no database file has been written yet. This chapter covers what a series is, what a sample costs to keep, where it is kept and what one server cannot do. The sample count goes from 3,088 to 131,801.

STEP 01 / 05

A SERIES IS A NAME PLUS LABELS

ONE SERIES = THIS EXACT SET OF 7 LABELS
__name__="http_requests_total"env="prod"instance="checkout-1:8000"job="checkout"method="POST"path="/checkout"status="200"
change one value and it is a different series
ITS SAMPLES · ONE EVERY 15 S
1900+10:05.2
1934+10:20.2
1966+10:35.2
1995+10:50.2
each dot is one sample: the requests counted so far, and under it the time Prometheus read that number
Fig 3.1The top row is one series’ identity, 7 labels; the accent dots under it are its samples, one every 15 seconds.

A is identified by its metric name and its full set of labels. The name is itself stored as a label, __name__. Change one label value and you have a different series. Each series holds samples, and a sample is two numbers: a timestamp in milliseconds and a 64-bit float.

At +10:00 Prometheus holds 3,329 series under 727 metric names, each with 39 or 40 samples. One of them counts the payments checkout-1 answered with 200: http_requests_total{env="prod", instance="checkout-1:8000", job="checkout", method="POST", path="/checkout", status="200"}. Four of its samples, read between +10:05 and +10:50, are 1900, 1934, 1966 and 1995.

STEP 02 / 05

LABELS MULTIPLY

node_cpu_seconds_total
12 CPUs →8 modes ↓
12 × 8 = 96 series
http_request_duration_seconds_bucket
8 route-and-copy pairs →7 buckets ↓
8 × 7 = 56 series4 routes × 2 copies of checkout = 8 pairs
alertmanager_notifications_failed_total
18 integrations →13 reasons ↓
18 × 13 = 234 seriesevery one of them reads 0
Fig 3.2Each grid is one metric, one cell per series; the largest, in accent, holds a zero in every cell.

The number of series a metric has is the product of its label values. node_cpu_seconds_total has a cpu label with 12 values and a mode label with 8, so 96 series. http_request_duration_seconds_bucket has 4 routes, 7 buckets and 2 copies of the service: 56 series.

The largest metric in this run is alertmanager_notifications_failed_total. It has 18 integrations (email, Slack, webhook and the rest) times 13 failure reasons, so 234 series, and every one reads 0. This count of series is called .

Each series costs memory whether or not its value ever changes. Now add a user_id label with 10,000 users to the request histogram: 56 × 10,000 = 560,000 series from one metric. That is why a label should take only a small, fixed set of values, and why checkout files unknown paths under path="other".

STEP 03 / 05

SIXTEEN BYTES IN, ABOUT TWO KEPT

raw6,658,960 bytes
416,185 samples × 16 bytes: an 8-byte timestamp and an 8-byte value
stored923,416 bytes · 2.2 bytes a sample
the same samples, as the chunks file of the block holds them
one chunk · up to 120 samples · 30 minutes at 15 s
2,933 of this run’s 3,399 series filled exactly one chunk
Fig 3.3The two bars are the same 416,185 samples: raw at 16 bytes each, and the accent bar as Prometheus wrote them.

A raw sample is an 8-byte timestamp and an 8-byte value. The , the time series database inside Prometheus, compresses both with the scheme from Facebook’s Gorilla paper (2015). Timestamps arrive 15.000 s apart, so it stores only the change in the gap, which is usually 0 and takes 1 bit. Each value is XORed with the one before, and an unchanged value also takes 1 bit.

Samples are packed per series into chunks of up to 120. At one sample every 15 seconds a chunk covers 30 minutes, so 2,933 series filled exactly one chunk in this run.

Measured after the run: 923,416 bytes of chunks hold 416,185 samples. That is 2.2 bytes each, 14% of the raw 6,658,960 bytes. The snapshot was taken shortly after the run ended, which is why it holds more than 402,329 samples. The documentation says 1 to 2 bytes on average, and longer, steadier series compress better than these.

STEP 04 / 05

MEMORY FIRST, A FILE EVERY TWO HOURS

scrapenew samples every 15 s
head blockmemory · 3,329 series · 131,801 samples at +10:00
write-ahead log (WAL)disk · 727,648 bytes at +10:00 · replayed after a restart
blockdisk · written every 2 hours · chunks + index
larger blocksolder blocks merged into one
deletedafter 15 days by default
Fig 3.4Follow a sample left to right; the accent box, memory, is where all 131,801 of this run’s samples still are at +10:00.

New samples go into the head block, the part of the database held in memory. Each one is also appended to the write-ahead log (WAL), a file on disk that a restarted Prometheus replays to rebuild the head. At +10:00 the WAL is 727,648 bytes for 131,801 samples. By the end of the run it is 2.2 MB.

Every 2 hours Prometheus writes the head’s data out as a block: a directory holding a chunks file, an index that maps labels to series, and meta.json. Blocks are later merged into larger ones and deleted once they are older than the retention time, 15 days by default.

This run never reached 2 hours, so Prometheus wrote no block of its own. The one listed below is a snapshot requested afterwards: 923,416 bytes of chunks and a 329,960-byte index.

THE DATA DIRECTORY AFTER THE RUN, AND THE SNAPSHOT BLOCK
du -sk /prometheus/wal /prometheus/chunks_head
# 2220  /prometheus/wal           kilobytes: the write-ahead log
# 132   /prometheus/chunks_head   chunks of the head block
curl -s -XPOST localhost:9090/api/v1/admin/tsdb/snapshot
# {"status":"success","data":{"name":"20261113T195400Z-666914b485fbbe16"}}
ls -l /prometheus/snapshots/20261113T195400Z-666914b485fbbe16/01M7B7JQJQVH7ZN9T92CD4Y1TP
# chunks/000001   923416 bytes   the samples, compressed
# index           329960 bytes   which labels belong to which series
# meta.json          306 bytes   416,185 samples in 3,459 series
# tombstones           9 bytes   deletions: none
STEP 05 / 05

WHAT ONE PROMETHEUS IS NOT

http_requests_total{instance="checkout-2:8000", path="/cart"}GET /cart requests counted by checkout-2, as Prometheus stored them
stopped +12:00 → +13:30
6 failed scrapesup = 0720 requests refused
2995+11:37.5
85+13:52.5
3046+11:52.5
29+13:37.5
90 seconds with no samples · each dot is one stored sample, labelled with the count it read
Fig 3.5The accent gap is 90 seconds with no samples; the 720 requests sent into it were counted only by their sender.

One Prometheus is one server with one local disk. It has no clustering and no replication. For availability you run two identical servers that scrape the same targets. It is also the wrong store for logs or events, and for labels whose values have no upper bound.

It is not exact either. The documentation says that if you need 100% accuracy, such as for per-request billing, Prometheus is not a good choice. This run shows why. While checkout-2 was stopped for 90 seconds, 720 requests were refused, and no counter recorded them because the process that counts was not running. The last 21 errors before the rollback happened after the final scrape and were never stored.

For long-term storage, or one view over many servers, remote_write sends each sample on to an external system such as Thanos, Cortex or Mimir.

WHAT YOU NOW KNOW

  • A series is a metric name plus a label set; a sample is a millisecond timestamp and a float. At +10:00: 3,329 series, 727 names.
  • Series count is the product of label values, and chunks of up to 120 samples compress 16 raw bytes to about 2.2.
  • Samples sit in memory and the write-ahead log, become 2-hour blocks, and are kept 15 days by default. The sample count is 131,801.
deep dive: retention, sizing a disk, remote write and running two

Retention is set by time, by size, or both: storage.tsdb.retention.time and storage.tsdb.retention.size in the config file since 3.8, where the older command-line flags are deprecated. Blocks are merged into larger ones that span up to 10% of the retention time or 31 days, whichever is smaller. The write-ahead log is kept in segments of 128 MB.

To size a disk, the documentation gives needed_disk_space = retention_time_seconds × ingested_samples_per_second × bytes_per_sample. This run ingests 223.5 samples a second, so 15 days at 2.2 bytes a sample is about 640 MB. Memory follows the number of active series, not the number of samples. To find the expensive metrics, read /api/v1/status/tsdb or run promtool tsdb analyze.

remote_write sends samples to another system as they arrive, and remote_read queries one. Agent mode keeps only the scraping and the forwarding. Federation runs the other way: one Prometheus scrapes /federate on another to pull in aggregated series. For availability, run two identical servers. Both send the same alerts, and Alertmanager removes the duplicates.

Prometheus started at SoundCloud in 2012 and has been public since January 2015. It was the second project to join the CNCF (2016-05-09) and the second to graduate (2018-08-09). The storage layer was rewritten for 2.0 (2017-11-08), and 3.0 arrived on 2024-11-14. The name is the Titan’s, and the design follows Google’s Borgmon. Some releases get long-term support: 3.13 is the current one.

ON THE EXAM

  • “what identifies a time series” → metric name plus label set
  • “value type” → 64-bit float, millisecond timestamp
  • “default retention” → 15 days
  • “protects recent data across a crash” → the write-ahead log
  • “block duration” → 2 hours
  • “is Prometheus clustered” → no, a single node
  • “high availability” → run two identical servers
  • “long-term storage” → remote_write to an external system
  • “what drives memory use” → active series, so label cardinality
  • “not suitable for” → logs, events, 100%-accurate billing
  • “bytes per sample” → 1 to 2

A selector, a window and a rate

131,801 samples are stored and none of them answers a question yet. PromQL, the Prometheus query language, gets answers out of them in a fixed order: first it picks series, then how much time to look at, and only then does arithmetic. This chapter asks six questions between +11:00 and +15:00. In the middle of them, at +12:00, the deploy restarts one copy of checkout, and its counters start again from 0. Prometheus keeps scraping while you ask: the sample count goes from 131,801 to 198,891.

STEP 01 / 06

A SELECTOR RETURNS THE NEWEST SAMPLE

EXPRESSIONevaluated at +11:00
http_requests_total{job="checkout",path="/checkout"}
RESULT · 4 OF THE 3,345 SERIES STORED
http_requests_total{env="prod", instance="checkout-2:8000", job="checkout", method="POST", path="/checkout", status="200"}2091requests counted
http_requests_total{env="prod", instance="checkout-2:8000", job="checkout", method="POST", path="/checkout", status="500"}0requests counted
http_requests_total{env="prod", instance="checkout-1:8000", job="checkout", method="POST", path="/checkout", status="200"}1995requests counted
http_requests_total{env="prod", instance="checkout-1:8000", job="checkout", method="POST", path="/checkout", status="500"}0requests counted
read 4 of the 145,165 samples stored at +11:00
each value is the newest sample of its series, looked for up to 5 minutes back
Fig 4.1The accent matchers pick 4 of 3,345 series; each result row is that series’ newest sample, not a sum.

The simplest query is a metric name followed by matchers in curly braces. A matcher is a test on one label. http_requests_total{job="checkout",path="/checkout"} asks for every series of that metric whose job label is checkout and whose path label is /checkout. At +11:00 Prometheus holds 3,345 series, and 4 of them pass: two copies of checkout, each with one series for status 200 and one for status 500.

The values are 2091 and 0 for checkout-2, then 1995 and 0 for checkout-1. Each is the newest sample of its series at or before the evaluation time, and Prometheus looks back at most 5 minutes for it (the lookback). A result of this shape, one sample per series and all read at one instant, is an . The query read 4 of the 145,165 samples stored so far.

There are four kinds of matcher. = and != compare a label with an exact value. =~ and !~ compare it with a regular expression, which has to match the whole value. The web UI, Grafana and the promtool command all send their queries to one HTTP endpoint, /api/v1/query. The sample below makes that call by hand with curl.

THE QUERY API: EVERY TOOL SENDS THIS REQUEST, AND VALUES COME BACK AS STRINGS
curl -s localhost:9090/api/v1/query \
  --data-urlencode 'query=http_requests_total{job="checkout",path="/checkout"}' \
  --data-urlencode 'time=1794598440'      # +11:00
# {"status": "success",
#  "data": {"resultType": "vector",
#           "result": [
#             {"metric": {"__name__": "http_requests_total", "env": "prod",
#                         "instance": "checkout-1:8000", "job": "checkout",
#                         "method": "POST", "path": "/checkout", "status": "200"},
#              "value": [1794598440, "1995"]},
#             three more series follow, with values "0", "2091" and "0"
#           ]}}
STEP 02 / 06

SQUARE BRACKETS RETURN A WINDOW

http_requests_total{instance="checkout-1:8000",path="/checkout",status="200"}[1m]
[1m] · +10:00 → +11:00
1831
1864
1934+10:20.247
1995+10:50.249
1900+10:05.247
1966+10:35.247
evaluated at +11:00: the window holds 4 samples, each with its own timestamp, the oldest 55 s oldthe 2 grey samples are older than the window: stored, and not returnedoffsets count from +0:00, the second Prometheus started (19:23:00 UTC)
Fig 4.2The accent bracket is the [1m] window ending at +11:00; it catches 4 samples, and the oldest is 55 seconds old.

Put a duration in square brackets after a selector and the result changes shape. With [1m], Prometheus returns every sample from the last minute, where the plain selector returned only the newest. That result is a . For successful checkouts on checkout-1 it holds 4 samples: 1900 at +10:05.247, 1934 at +10:20.247, 1966 at +10:35.247 and 1995 at +10:50.249.

The samples keep their own timestamps. Nothing is moved to +11:00 or spaced out evenly: the window holds whatever the scrapes left in it, and the oldest sample is 55 seconds old. A range vector cannot be drawn on a graph or compared with a number. It exists to be handed to a function, and the next step does that.

Two modifiers shift a selector in time. offset 5m reads the series as it was 5 minutes earlier, and returns 1302 here. @ followed by a Unix time pins the query to that moment. PromQL has four data types in all: instant vector, range vector, scalar (one plain number) and string.

STEP 03 / 06

RATE: TWO READINGS AND A CLOCK

requests counted · checkout-1 · POST /checkout · status 200
1900193419661995
+10:00+10:30+11:00
slope of the accent line: 95 requests ÷ 45.002 s = 2.111 a second
EXPRESSIONevaluated at +11:00
rate(http_requests_total{instance="checkout-1:8000",path="/checkout",status="200"}[1m])
RESULT
ratelast sample minus first, over the time between them2.111a second
irate in its placethe last two samples only: 29 requests ÷ 15.002 s1.933a second
increase in its placethe rate × the 60 s window: not a whole number126.66requests
each read 4 of the 145,165 samples stored at +11:00
Fig 4.3The accent line joins the first and last sample in the window; its slope, 95 requests in 45 seconds, is the rate.

takes a range vector and returns how fast each counter in it grew, per second. On the 4 samples from the last step it returns 2.111 requests a second. Check it by hand: the last sample minus the first is 1995 − 1900 = 95 requests, and the two are 45.002 seconds apart. 95 ÷ 45.002 = 2.111. It is the same sum you would do with two odometer readings and a clock.

Prometheus then stretches the result to cover the whole 60-second window, because the samples span only 45 seconds of it. That is why increase(), which is the rate multiplied by the window, returns 126.66 requests: a fraction, although the counter moved by exactly 95. irate() uses only the last two samples, 29 requests in 15.002 seconds, and returns 1.933 a second.

A rate needs at least two samples. A [20s] window at a 15-second scrape interval holds 1 sample, and the query returns nothing at all. The rule of thumb is a window at least 4 times the scrape interval, which here means [1m] or longer.

STEP 04 / 06

THE COUNTER THAT WENT BACKWARDS

http_requests_total{instance="checkout-2:8000",path="/cart",status="200"}
304629
6 scrapes failed
new process: 29 … 180
+11:00+12:00+13:00+14:00
3046 at +11:52.5, then 29 at +13:37.5: the counter went backwards
checkout-2 was stopped from +12:00 to +13:30; on each dashed line the scrape failed and up was 0
EXPRESSIONevaluated at +15:00
sum(rate(http_requests_total{job="checkout",path="/cart"}[5m]))
RESULT
rate first, then sum · correct5.94a second
read 34 of the 198,891 samples stored at +15:00
EXPRESSIONevaluated at +15:00
rate(sum(http_requests_total{job="checkout",path="/cart"})[5m:15s])
RESULT
sum first, then rate · wrong17.39a second
read 34 of the 198,891 samples stored at +15:00
Fig 4.4Follow the accent dots: the counter climbs to 3046, is absent for 6 scrapes, and starts again at 29.

The deploy at +12:00 stopped checkout-2. Its /cart counter read 3046 at +11:52.494. The next 6 scrapes failed, and the new process reported 29 at +13:37.494. A counter that falls has been reset: its process restarted and began again from 0. rate() treats every fall as a reset and adds the earlier value back, so the result carries on as if the counter had never restarted.

That correction works on one series at a time, so take rate first and sum second. At +15:00, sum(rate(…)) over both copies’ /cart series is 5.94 requests a second. rate(sum(…)) is 17.39, 3 times too high. The sum lost checkout-2’s 3046 when that series went away, and rate() treated the fall as a reset of the whole total.

is how a series goes away. At +12:05 the selector still returned 3046. At +12:10, after the failed scrape at +12:07.494, it returned nothing: a failed scrape marks every series of that target stale at once. The rollback at +20:00 was different. The process restarted in 1.4 seconds, between two scrapes, so up stayed 1 while the counter fell from 1337 to 28. resets() over the run returns 2.

STEP 05 / 06

SUM BY: FEWER LABELS, FEWER ROWS

INPUT · 10 SERIES · REQUESTS A SECOND
rate(http_requests_total{job="checkout"}[5m])
checkout-2 · /cart · 2003.34
checkout-1 · /cart · 2003.68
checkout-2 · /cart/items · 2002.00
checkout-1 · /cart/items · 2001.88
checkout-2 · /checkout · 2002.49
checkout-2 · /checkout · 5000.00
checkout-1 · /checkout · 2002.31
checkout-1 · /checkout · 5000.00
checkout-2 · other · 4040.17
checkout-1 · other · 4040.12
EXPRESSIONevaluated at +11:00
sum by (path) (rate(http_requests_total{job="checkout"}[5m]))
RESULT · 4 SERIES, ONE FOR EACH VALUE OF PATH
{path="/cart"}7.03a second
{path="/cart/items"}3.88a second
{path="/checkout"}4.80a second
{path="other"}0.29a second
read 200 of the 145,165 samples stored at +11:00
sum(rate(http_requests_total{job="checkout"}[5m]))16.00 a second
no by clause: every label is dropped, and one row is left
Fig 4.510 input rows collapse into the 4 accent rows: one per value of path, each the sum of its group.

An aggregation operator combines many series into fewer. rate(http_requests_total{job="checkout"}[5m]) returns 10 series at +11:00, one for each combination of copy, path and status. Wrap it in sum by (path) (…) and Prometheus groups the 10 by their path label, adds up each group and returns 4 rows: /cart 7.03, /cart/items 3.88, /checkout 4.80 and other 0.29 requests a second.

Read by (path) as “keep only path”. without (instance, status) is the mirror image: it drops those two labels, keeps every other one, and here returns the same 4 sums. A bare sum() keeps no label and returns one row: 16.00 requests a second, which is the 16 the load generator sends. The query read 200 samples, 20 for each series.

The other aggregation operators take the same two clauses: avg, min, max, count, topk, bottomk, quantile, stddev, count_values and group. topk(2, …) keeps the 2 largest rows, /cart and /checkout. count by (job) (up) counts series and returns 2 for checkout: its two targets.

STEP 06 / 06

GAUGES GET AVERAGED, NOT RATED

avg_over_time(checkout_inflight_requests[5m])
max_over_time(checkout_inflight_requests[5m])
each read 40 of the 253,167 samples stored at +19:00 · a dot is the requests in flight at one scrape
checkout-2:8000 · version 2.4.1
03.15
5-minute average
maximum
+14:00+15:00+16:00+17:00+18:00+19:00
average over the 5 minutes3.1 in flight
largest sample in them5 in flight
checkout-1:8000 · version 2.4.0
00.72
+14:00+15:00+16:00+17:00+18:00+19:00
average over the 5 minutes0.7 in flight
largest sample in them2 in flight
Fig 4.6The accent line is the 5-minute average of a gauge that only reads 0 to 5; the dots are its raw samples.

A gauge goes up and down, so how much it grew means little and rate() is the wrong tool. Functions whose names end in _over_time summarise one series along its window instead. checkout_inflight_requests is the number of requests a copy is handling at the moment of the scrape. At +11:00, avg_over_time(checkout_inflight_requests[5m]) returns 0.9 and 1.0 for the two copies, and max_over_time returns 2 for both.

At +19:00 the broken version is live. checkout-2 averages 3.1 requests in flight and peaks at 5, while checkout-1 stays at 0.7. Keep the two families apart. An aggregation operator such as sum works across series at one moment. An _over_time function works across time on one series.

A gauge with a trend gets two more functions. deriv() returns its slope, and predict_linear() extends that slope into the future. node exporter reports the machine’s free disk space: 156.57 GB at +11:00, falling by 9,250 bytes a second. predict_linear(…[10m], 4 * 3600) returns 156.43 GB for 4 hours later.

WHAT YOU NOW KNOW

  • A selector returns each matching series’ newest sample, up to 5 minutes old; [1m] returns the raw samples in a window.
  • rate() is last minus first over elapsed time, extrapolated and reset-corrected: 95 requests in 45 s is 2.111 a second. Take rate before sum.
  • sum by (path) collapses 10 series to 4, and _over_time functions average a gauge along time. The sample count is 198,891.
deep dive: range queries, subqueries, offset and the full function list

The 5-minute lookback is a server flag, --query.lookback-delta. Since Prometheus 3.0 a window is open on its left: a sample exactly on the left edge is left out. A [1m] window at a 15-second interval therefore holds at most 4 samples, never 5.

An instant query, /api/v1/query, evaluates an expression at one time. A range query, /api/v1/query_range, takes start, end and step, evaluates the same expression at every step and returns one line per series. That is what a graph is. A subquery does the same inside an expression. max_over_time(sum(rate(http_requests_total{job="checkout"}[1m]))[10m:15s]) evaluates the inner sum every 15 seconds for 10 minutes and keeps the largest result: 16.04 here, from 1,600 samples read.

offset 1h moves a selector back an hour. @ pins it to a Unix time, or to the edges of a range query with @ start() and @ end(). Durations use the units ms, s, m, h, d, w and y. A # starts a comment.

More functions worth recognising: delta and idelta (the gauge versions of increase and irate), changes and resets (how often a value changed or fell), absent_over_time, label_replace and label_join (rewrite the labels of a result), clamp_min, round and sort_desc. limitk and limit_ratio are experimental aggregators that return a sample of the series. The flag --query.max-samples caps how many samples one query may load into memory.

ON THE EXAM

  • “type a selector returns” → instant vector
  • “type metric[5m] returns” → range vector
  • “can a range vector be graphed” → no
  • “per-second average increase of a counter” → rate
  • “uses the last two samples” → irate, for graphs of fast-moving counters
  • “total growth over the window” → increase
  • “handle counter resets” → rate, irate and increase all do
  • “order of rate and sum” → rate first, then sum
  • “slope or forecast of a gauge” → deriv, predict_linear
  • “keep only these labels” → by
  • “drop these labels” → without
  • “the 3 largest series” → topk(3, …)
  • “average of a gauge over 5 minutes” → avg_over_time(x[5m])
  • “how old may a sample be for an instant selector” → 5 minutes
  • “the value an hour ago” → offset 1h

Two vectors, one ratio, and a percentile from seven buckets

At +13:30 the broken version starts answering, and every payment sent to it fails after 1.2 seconds. A count of errors cannot say whether that is a lot until you divide it by a count of everything. This chapter combines query results: one divided by another, joined to a third, cut down by a threshold and turned into a percentile. All six queries are evaluated at +16:00, 2 min 30 s into the failure. The sample count goes from 198,891 to 212,459.

STEP 01 / 06

DIVIDE ONE VECTOR BY ANOTHER

EXPRESSIONevaluated at +16:00
sum(rate(http_requests_total{job="checkout",status=~"5.."}[5m])) / sum(rate(http_requests_total{job="checkout"}[5m]))
THE TWO SIDES, THEN THE RESULT
left of the /errors: responses with a 5xx status1.136a second
right of the /every request13.209a second
the quotientthe share of requests failing0.0860= 8.6%
read 204 of the 212,459 samples stored at +16:00: 34 for the left side, 170 for the right
Fig 5.1The two grey rows are the numerator and the denominator; the accent row is their quotient, the share of requests failing.

Arithmetic operators (+ - * / % ^) work between a vector and a plain number, or between two vectors. With two vectors, Prometheus pairs each series on the left with the series on the right that has exactly the same labels, then does the arithmetic pair by pair.

The error ratio is the standard case: errors a second divided by requests a second. The left side keeps only statuses that match 5.. and is 1.136 at +16:00. The right side is every request: 13.209. Both sides went through a bare sum(), so each is one series with no labels, and the two pair. 1.136 ÷ 13.209 = 0.0860: 8.6% of requests are failing.

The query read 204 samples: 34 for the left side and 170 for the right. A count of errors alone, 1.136 a second, could not say whether that was a lot. Divided by the traffic it is a share, and a share can be compared with a target.

STEP 02 / 06

LABELS MUST MATCH, OR NOTHING COMES BACK

SERIES BY SERIES, WITH NO SUM
EXPRESSIONevaluated at +16:00
rate(http_requests_total{job="checkout",status="500"}[5m]) / rate(http_requests_total{job="checkout",status="200"}[5m])
RESULT
Empty query result
read 136 samples and returned 0 series, with no error
a series left of the /{env="prod", instance="checkout-2:8000", job="checkout", method="POST", path="/checkout", status="500"}
its closest series right of the /{env="prod", instance="checkout-2:8000", job="checkout", method="POST", path="/checkout", status="200"}
5 labels are equal and status differs: no pair, so nothing is divided
AFTER SUM BY (INSTANCE) ON BOTH SIDES
a series left of the /{instance="checkout-2:8000"}
its closest series right of the /{instance="checkout-2:8000"}
identical label sets: a pair, and the division happens
EXPRESSIONevaluated at +16:00
sum by (instance) (rate(http_requests_total{job="checkout",status=~"5.."}[5m])) / sum by (instance) (rate(http_requests_total{job="checkout"}[5m]))
RESULT
{instance="checkout-2:8000"}0.218= 21.8% of its requests failing
{instance="checkout-1:8000"}0none of its requests failing
read 204 of the 212,459 samples stored at +16:00
Fig 5.2Compare the label sets on the two sides: the accent label, status, differs in every pair, so the first query pairs nothing.

Now divide series by series, without the sum: the rate of status 500 over the rate of status 200. The result is empty, and there is no error message. Every series on the left carries status="500" and every series on the right carries status="200", so no two have identical labels. This pairing rule is , and a series without a partner is dropped.

There are two fixes. The first removes the label that differs, on both sides. sum by (instance) (…) / sum by (instance) (…) leaves one series per copy on each side, and they pair. It returns 0.218 for checkout-2 and 0 for checkout-1.

The second fix keeps the series and changes what is compared. on (instance) after the operator means: compare the instance label and nothing else. ignoring (status) means: compare every label except status. Either way, a label left out of the comparison is also left out of the result.

STEP 03 / 06

A JOIN THAT ADDS THE VERSION

EXPRESSIONevaluated at +16:00
sum by (instance) (rate(http_requests_total{job="checkout",status=~"5.."}[5m])) * on (instance) group_left (version) checkout_build_info
left of the *errors a second, one series per copy
{instance="checkout-2:8000"}1.136a second
{instance="checkout-1:8000"}0a second
* on (instance)
checkout_build_infoan info metric: its value is always 1
{env="prod", instance="checkout-2:8000", job="checkout", version="2.4.1"}1always
{env="prod", instance="checkout-1:8000", job="checkout", version="2.4.0"}1always
group_left (version)
resultthe left value × 1, with version copied on
{instance="checkout-2:8000", version="2.4.1"}1.136a second
{instance="checkout-1:8000", version="2.4.0"}0a second
read 36 of the 212,459 samples stored at +16:00
Fig 5.3The accent label, version, exists only on the right-hand series; group_left copies it onto each result row.

checkout_build_info is an info metric: a series whose value is always 1 and whose labels are the facts. On checkout-2 they include version="2.4.1". Because the value is 1, a multiplication by it leaves the other side’s numbers as they were. The multiplication is there to carry a label across.

The query multiplies each copy’s errors a second by that series. on (instance) names the one label to pair on. group_left (version) copies the version label from the right-hand side onto the result. It returns 1.136 for checkout-2 with version 2.4.1, and 0 for checkout-1 with version 2.4.0. Every error comes from the new version.

group_left also lets several series on the left pair with one on the right, which is called many-to-one matching. Leave it out when the left side has several series per instance, and the query fails with an error: many-to-one matching must be explicit (group_left/group_right).

STEP 04 / 06

A COMPARISON IS A FILTER

LEFT OF THE > · 2 SERIES GO IN
{instance="checkout-2:8000"}0.218above the line: kept
{instance="checkout-1:8000"}0not above it: dropped
↑ the threshold, 0.05
EXPRESSIONevaluated at +16:00
sum by (instance) (rate(http_requests_total{job="checkout",status=~"5.."}[5m])) / sum by (instance) (rate(http_requests_total{job="checkout"}[5m])) > 0.05
RESULT · 1 SERIES COMES OUT, VALUE UNCHANGED
{instance="checkout-2:8000"}0.218the value it came in with
read 204 of the 212,459 samples stored at +16:00
EXPRESSIONevaluated at +16:00
sum by (instance) (rate(http_requests_total{job="checkout",status=~"5.."}[5m])) / sum by (instance) (rate(http_requests_total{job="checkout"}[5m])) > bool 0.05
RESULT WITH BOOL · NOTHING IS DROPPED
{instance="checkout-2:8000"}1passed the test
{instance="checkout-1:8000"}0failed the test
read 204 of the 212,459 samples stored at +16:00
Fig 5.4Two rows go into the comparison and only the accent row comes out, still carrying 0.218, not “true”.

Add > 0.05 to the ratio per copy. The comparison does not return true or false. It drops every series that fails the test and returns the rest with their values unchanged. Here one row is left: checkout-2, still carrying 0.218. Write > bool 0.05 and nothing is dropped: each series returns 1 if it passed and 0 if it did not.

Three set operators combine whole vectors by their labels. and keeps the left series that have a partner on the right. unless keeps the ones that have none. or returns the left side plus any right series without a partner. up{job="checkout"} unless on (instance) checkout_build_info{version="2.4.1"} returns only checkout-1.

absent() covers the case a filter cannot: a series that does not exist. absent(up{job="checkout",instance="checkout-3:8000"}) returns 1 because nothing matches. The next chapter depends on all of this. An alert expression is a filter that returns nothing while all is well, and every series it does return is an alert.

STEP 05 / 06

A PERCENTILE FROM SEVEN BUCKETS

EXPRESSIONevaluated at +16:00
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job="checkout"}[5m])))
RESULT
the 99th percentile, estimated from the buckets2.33s
read 952 of the 212,459 samples stored at +16:00
requests a second that finish within each bound (the le label)
up to 0.05 s7.13 a second
up to 0.1 s9.25 a second
up to 0.25 s12.07 a second
up to 0.5 s12.07 a second
up to 1 s12.07 a second
up to 2.5 s13.24 a second
any duration (+Inf)13.24 a second
request number 13.11 = 0.99 × 13.24 ↑
inside the accent bucket, from 1 s to 2.5 s
↓ where they are: about 1.2 s
1 s
2.5 s
the estimate: 2.33 s ↑
1 + 1.5 × (13.11 − 12.07) ÷ (13.24 − 12.07) = 2.33 s, as if requests were spread evenly across the bucket
Fig 5.5The accent bucket, 1 to 2.5 seconds, holds the 99th-percentile request; the two markers show the estimate and the real time.

The 99th percentile is the time that 99% of requests beat. A histogram keeps no individual times, only a count per bucket, so histogram_quantile() has to estimate it. histogram_quantile(0.99, sum by (le) (rate(…_bucket[5m]))) returns 2.33 seconds at +16:00. Yet the failing requests take 1.2 seconds: their measured mean is 1.216.

Here is the estimate by hand. Every bucket carries a label le, for “less than or equal”: its upper bound in seconds. Per second, 7.13 requests finish within 0.05 s, 9.25 within 0.1 s, and 12.07 within 0.25 s, 0.5 s and 1 s alike. All 13.24 finish within 2.5 s. The 99th-percentile request is number 0.99 × 13.24 = 13.11, so it lies in the bucket from 1 to 2.5 seconds.

Inside a bucket the function draws a straight line, as if requests were spread evenly across it: 1 + 1.5 × (13.11 − 12.07) ÷ (13.24 − 12.07) = 2.33. The real requests all sit near 1.2 seconds, at the low end. Two more facts. Drop le from the sum and the result is empty. Before the deploy, at +11:00, the same query returned 0.245 seconds.

STEP 06 / 06

SUBTRACT A TIMESTAMP FROM NOW

EXPRESSIONevaluated at +16:00
time()
time() - cart_cleanup_last_success_timestamp_seconds
time() - process_start_time_seconds{job="checkout"}
RESULTS, ONE EXPRESSION AFTER ANOTHER
time()now, in seconds since 1 January 1970 UTC1794598740Unix seconds
since cart-cleanup last succeeded1794598740 − 1794598383.59356.4s
uptime of checkout-21794598740 − 1794598589.67150.3s
uptime of checkout-11794598740 − 1794597602.091,137.9s
the two subtractions read 3 of the 212,459 samples stored at +16:00
+0:00 = Prometheus starts, 19:23:00 UTC
+10:00+12:00+14:00+16:00 now
+10:03.6 cleanup finished · 356.4 s ago
cart-cleanupthe batch job
+13:29.7 checkout-2 started · up 150.3 s
checkout-2the restarted copy
Fig 5.6Each row subtracts a stored Unix time from now; the accent rows show checkout-2 has been up 150 seconds and checkout-1 1,138.

Some metrics hold a moment in time, written as Unix seconds. cart_cleanup_last_success_timestamp_seconds is 1794598383.59, the moment the batch job last finished. time() returns the evaluation time in the same form. Subtract one from the other and you have an age: time() - cart_cleanup_last_success_timestamp_seconds is 356.4 seconds at +16:00. The rule CartCleanupStale alerts when that passes 3,600.

The same subtraction gives uptime. time() - process_start_time_seconds is 150.3 seconds on checkout-2 and 1,137.9 on checkout-1, which shows which copy restarted. timestamp() returns the time a sample was scraped, where a selector returns its value. time() - timestamp(up) says the two copies’ newest samples are 7.5 and 9.8 seconds old.

This is why a job should export the time something happened, and never the seconds since. Prometheus can do the subtraction at any moment. A stored age was right when it was written, and it is out of date by the next scrape.

WHAT YOU NOW KNOW

  • Dividing two rates gives the error ratio: 1.136 ÷ 13.209 = 8.6% at +16:00. Series pair only when their labels are identical.
  • on, ignoring and group_left control the pairing; a comparison filters series, and an empty result means nothing matched.
  • histogram_quantile estimates inside a bucket (2.33 s for requests that took 1.2 s); time() minus a timestamp is an age. The sample count is 212,459.
deep dive: operator precedence, group_right, native histograms and Apdex

Operators bind in this order, tightest first: ^, then * / %, then + -, then the comparisons, then and and unless, then or. Matching comes in three shapes: one-to-one, many-to-one with group_left and one-to-many with group_right. An expression that may return nothing can end in or vector(0), which turns an empty result into a 0.

A histogram answers more than percentiles. The average duration is the rate of _sum divided by the rate of _count: 0.173 seconds here. The share of requests under a threshold comes straight from one bucket: 91.2% finished within 0.5 seconds at +16:00, and that ratio is the SLI itself. Apdex is a score built the same way from two buckets: requests under a target count in full, and requests under a looser limit count half.

Native histograms keep all their buckets in one series with no le label, so the query loses its by (le): histogram_quantile(0.99, sum(rate(x[5m]))). They add histogram_fraction, histogram_count and histogram_avg. A summary works out its quantiles inside the process and exports each as a series with a quantile label. Those cannot be averaged across copies, which is the usual reason to choose a histogram.

Calendar functions return parts of the evaluation time, in UTC: hour(), day_of_week() and days_in_month(). They let a rule apply only in business hours. The times in this run are UTC too: +16:00 is 1794598740 as time() returns it.

ON THE EXAM

  • “which labels are compared by default” → all of them
  • “compare on only some labels” → on (…)
  • “leave labels out of the comparison” → ignoring (…)
  • “many series on the left, one on the right” → group_left
  • “make a comparison return 0 or 1” → bool
  • “what a comparison returns without bool” → the series that pass
  • “series in A but not in B” → unless
  • “detect a missing series” → absent()
  • “percentile from a histogram” → histogram_quantile
  • “label that must survive aggregation of buckets” → le
  • “average request duration” → rate of _sum over rate of _count
  • “seconds since the last success” → time() minus the timestamp metric
  • “what limits histogram_quantile’s accuracy” → the bucket boundaries

A query on a timer, and a name for when it returns something

It is 19:38 UTC on a Friday and nobody is typing queries. The error ratio from the last chapter is only useful if something computes it without you and tells a person when it goes wrong. Prometheus does the first half itself: since it started, it has evaluated that ratio every 15 seconds from a rules file and saved each answer. At +15:09.6 the answer is 5.75%, over the 5% line for the first time, and at +17:09.6 an alert fires. Telling a person is the next chapter. The sample count goes from 212,459 to 226,211.

STEP 01 / 05

A QUERY ON A TIMER

+0:00 = Prometheus starts, 19:23:00 UTC
job:http_errors_per_request:ratio_rate5mthe value the third rule saved at each evaluation
  1. 0%
  2. 0.36%+13:39.6
  3. 1.09%
  4. 2.03%+14:09.6
  5. 2.90%
  6. 4.09%+14:39.6
  7. 4.83%
  8. 5.75%+15:09.6
one tick = one evaluation of all 8 rules, every 15 s+15:09.6 is the first value over 5%
rounds in this run120 rounds × 8 rules = 960 evaluations
evaluations that failed0 evaluations
time for one roundabout 1.1 ms
saved by the 4 recording rules476 samples
Fig 6.1Each tick is one evaluation of the group, 15 seconds apart; the accent row is the ratio the third rule saves.

Rules live in groups: named lists in a rules file. Prometheus evaluates every rule in a group, in order, once per evaluation interval. The setting is evaluation_interval, 15 seconds here and 1 minute by default. This run made 120 rounds of 8 rules: 960 evaluations, none of them failed, and a round took about 1.1 ms.

A saves the result of its expression as a new series. The third rule in the file saves the error ratio as job:http_errors_per_request:ratio_rate5m{job="checkout"}. The name follows a convention, level:metric:operations: the labels kept by the aggregation, the metric, then what was done to it.

The four recording rules wrote 476 samples in this run. A later rule in the group can read an earlier one, which is how the third rule divides the first two. A dashboard or an alert then reads one stored sample per evaluation, where the raw query in chapter 05 read 204 samples.

RULES/CHECKOUT.YML: THE GROUP AND ITS FOUR RECORDING RULES
groups:
  - name: checkout
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))
      - record: job:http_errors:rate5m
        expr: sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
      # reads the two rules above; 0.0575 at +15:09.6
      - record: job:http_errors_per_request:ratio_rate5m
        expr: job:http_errors:rate5m / job:http_requests:rate5m
      - record: job:http_request_duration_seconds:p99_rate1m
        expr: >-
          histogram_quantile(0.99,
            sum by (job, le) (rate(http_request_duration_seconds_bucket[1m])))
STEP 02 / 05

AN ALERT RULE IS A QUERY THAT SHOULD RETURN NOTHING

rules/checkout.yml · group checkout
  1. - alert: CheckoutHighErrorRate
    namebecomes the label alertname
  2. expr: job:http_errors_per_request:ratio_rate5m{job="checkout"} > 0.05
    conditionevery series it returns is one alert
  3. for: 2m
    how long it must hold8 evaluations in a row
  4. labels:severity: pageservice: checkout
    labels · for routingadded to every alert the rule returns
  5. annotations:summary: "Checkout is failing {{ $value | humanizePercentage }} of requests"runbook_url: https://runbooks.example.com/checkout/high-error-rate
    annotations · for peopletext, filled in from a template
the summary as rendered when the alert fired at +17:09.6, with $value = 0.127“Checkout is failing 12.7% of requests”
Fig 6.2Read the rule top to bottom: the accent line is the whole condition, last chapter’s ratio compared with 0.05.

An has a name, alert:, and an expression, expr:, which can be any PromQL. Every series the expression returns is one alert, so a healthy system returns nothing. for: 2m says how long the expression must keep returning a series before that alert fires.

labels: are added to the alert and are what routing matches on later: here severity: page and service: checkout. annotations: are text for people and can hold a template. Here {{ $value | humanizePercentage }} turned the value 0.127 into “Checkout is failing 12.7% of requests”.

The file holds four alerting rules. For the first 12 minutes of the run all four returned nothing at every evaluation, which is the normal state of an alerting rule.

RULES/CHECKOUT.YML: TWO OF THE FOUR ALERTING RULES
- alert: CheckoutHighErrorRate
  expr: job:http_errors_per_request:ratio_rate5m{job="checkout"} > 0.05
  for: 2m                       # 8 evaluations in a row
  labels:
    severity: page              # used for routing
    service: checkout
  annotations:
    summary: "Checkout is failing {{ $value | humanizePercentage }} of requests"
    runbook_url: https://runbooks.example.com/checkout/high-error-rate

- alert: TargetDown
  expr: up == 0
  for: 30s
  labels:
    severity: warning
    service: "{{ $labels.job }}"
  annotations:
    summary: "{{ $labels.instance }} has not answered a scrape for 30 seconds"
STEP 03 / 05

PENDING FOR EIGHT EVALUATIONS

+0:00 = Prometheus starts, 19:23:00 UTC
+12:00 deploy+16:00+20:00 rollback+24:00
pending +12:09.6
firing +12:39.6
resolved +13:39.6
TargetDownfor: 30s · severity warning
pending +13:54.6
firing +18:54.6
resolved +20:54.6
CheckoutHighLatencyfor: 5m · severity warning
pending +15:09.6
firing +17:09.6
resolved +23:09.6
CheckoutHighErrorRatefor: 2m · severity page
Fig 6.3Three lanes, one per alert: hollow bars are pending, filled bars are firing. The accent bar is the alert that paged.

An alert is in one of three states. Inactive: the expression does not return it. Pending: the expression returns it, and the for time has not passed yet. Firing: it has been returned at every evaluation for the whole for time. A single evaluation under the line puts a pending alert back to inactive.

At +14:54.6 the ratio was 4.83%: inactive. At +15:09.6 it was 5.75%: pending. It stayed above 5% for 8 evaluations, and at +17:09.6, two minutes after the first, the alert fired at 12.70%. While an alert is pending or firing, Prometheus also writes a series for it, ALERTS{alertname="CheckoutHighErrorRate", alertstate="pending"} with value 1: 8 pending samples here, then 24 firing ones.

TargetDown has for: 30s: pending at +12:09.6, firing at +12:39.6. CheckoutHighLatency has for: 5m: pending at +13:54.6, firing at +18:54.6.

STEP 04 / 05

THE WINDOW IS PART OF THE WAIT

+0:00 = Prometheus starts, 19:23:00 UTC
checkout-1checkout-2
+12:00deploy+16:00+20:00rollback+24:00
5-minute window ending+13:09.6error ratio0.00%
0%5% · the rule’s line16%
under 5%: no alert
  • requests answered
  • stopped for 1 min 30 s
  • every payment fails
first failed payment +13:30.3page sent +17:39.6
1:395-minute window filling
2:00for: 2m
0:30group_wait
= 4:09 from the first failed payment to a person
Fig 6.4Watch the 5-minute window slide over the failure; the accent number is the ratio it reports, crossing 5% at +15:09.6.

The first payment failed at about +13:30. The rule’s 5-minute window still held mostly healthy requests, so the ratio rose slowly: 0.36% at +13:39.6 and 5.75% at +15:09.6. That is 1 min 39 s before the alert was even pending.

Then for: 2m ran, to +17:09.6. Then Alertmanager, the program in the next chapter, waited 30 seconds before sending, to +17:39.6. From the first failed payment to a person: 4 min 9 s. Recovery lags the same way. The ratio peaked at 15.39% and, after the rollback at +20:00, took until +23:09.6 to fall under 5%.

Each part is a trade. The latency rule uses a 1-minute window and crossed its line at +13:54.6, after 24 seconds of failures. A short window reacts sooner and flaps more: it goes over and under the line on brief spikes. A long window and a long for give fewer false alarms and a later page.

STEP 05 / 05

PAGE ON WHAT THE CUSTOMER FEELS

SYMPTOMS · WHAT A USER EXPERIENCES
  • CheckoutHighErrorRate
    detectsover 5% of requests failing
    severitypage
    for2m
    in this runfired +17:09.6 · paged
  • CheckoutHighLatency
    detects99th percentile over 0.5 s
    severitywarning
    for5m
    in this runfired +18:54.6 · muted
CAUSES · WHY IT MIGHT BE HAPPENING
  • TargetDown
    detectsa target not answering scrapes
    severitywarning
    for30s
    in this runfired +12:39.6 · muted
  • CartCleanupStale
    detectsno cart-cleanup success for 1 hour
    severitywarning
    fornot set
    in this runnever active
the page line, 5% of requests failing50 × what a 99.9% SLO allows
requests failed in this incident1,661 of a 41,472 monthly budget · 4.0%
Fig 6.5Four rules sorted by what they detect; the accent one is the only rule allowed to page a person.

Alert on symptoms, the things a user experiences, and page a person only for those. A cause, such as one copy being down, is recorded for information. Of the four rules, CheckoutHighErrorRate is a symptom with severity: page and CheckoutHighLatency is a symptom with a warning. TargetDown and CartCleanupStale are causes with warnings, and the second was never active in this run.

A page should be urgent, should need a person, and should say what to do. This one carries a runbook_url annotation: a link to the written steps for this alert.

The line is set against the SLO from chapter 00, 99.9% of requests succeeding over 30 days. An error ratio of 5% is 50 times the failure rate that target allows. The incident failed 1,661 requests, which is 4.0% of the month’s error budget of 41,472.

WHAT YOU NOW KNOW

  • Prometheus evaluated 8 rules every 15 seconds, 960 evaluations; a recording rule saves its result as a new series named level:metric:operations.
  • An alerting rule fires for each series its expression returns, once that has lasted for: pending at +15:09.6, firing at +17:09.6.
  • The 5-minute window and for: 2m make up 3:39 of the 4:09 between the first failure and the page. Sample count: 226,211.
deep dive: testing rules, keep_firing_for and burn-rate alerts

Rule files are listed under rule_files in the main config, and a group may set its own interval. promtool check rules validates a file before you load it. promtool test rules is the unit test: you write input_series, it replays them through the rules, and it asserts which alerts fire and when.

keep_firing_for holds an alert in the firing state for a set time after its condition clears, which stops an alert that hovers around its line from resolving and firing again. ALERTS_FOR_STATE is a second series Prometheus writes for each active alert. Its value is the Unix time the alert became active: 1794598689, which is 19:38:09 UTC, for this one. A restarted Prometheus reads it and resumes the for clock.

Templates in labels and annotations can read $labels and $value, and format them with helpers such as humanize, humanizeDuration and humanizePercentage. The external_labels in the main config are added to every alert Prometheus sends, so a receiver can tell which Prometheus it came from.

A fixed 5% line is the simple form. The SRE workbook’s burn-rate alerts compare the error ratio with the SLO directly: for a 99.9% target, page when the error budget is burning 14.4 times too fast over 1 hour, which means 2% of the month’s budget is gone. A rule can also alert on a series that is missing, with absent(). Prometheus reports on its own rules in prometheus_rule_group_last_duration_seconds and prometheus_rule_evaluation_failures_total.

ON THE EXAM

  • “precompute an expensive query” → a recording rule
  • “recording rule naming” → level:metric:operations
  • “field that delays firing” → for
  • “state before firing” → pending
  • “where severity goes” → labels
  • “where a description or runbook link goes” → annotations
  • “template variable for the expression’s value” → $value
  • “labels of the alerting series” → $labels
  • “how often rules run” → evaluation_interval, default 1m
  • “validate rules” → promtool check rules
  • “unit-test rules” → promtool test rules
  • “series Prometheus writes for active alerts” → ALERTS
  • “symptoms or causes” → page on symptoms
  • “keep firing after the condition clears” → keep_firing_for

Three alerts go in, one message comes out

At +17:09.6 an alert is firing and no person has been told. Prometheus does not send messages to people. It hands each firing alert to a second program, Alertmanager, which groups alerts, mutes some and sends the rest. In this run 3 alerts reach it, and one of them becomes a message to a person, at +17:39.6. The rollback follows at +20:00. By the end of the run the sample count has gone from 226,211 to 402,329.

STEP 01 / 06

THIRTY-FIVE DELIVERIES, TWO MESSAGES

targets6 /metrics pages
checkout-1checkout-2node exporterPushgateway
GET /metrics every 15 s
Prometheus3 alerts fired
scrapestoragerules
35 deliveries
Alertmanagera separate program
deduplicategrouproutemutenotify
2 notifications
pager2 notifications
team-chat0 notifications
you · queries and dashboards read from Prometheus with PromQL
deliveries to /api/v2/alerts35 = 8 as firing + 27 as resolved
deliveries with the same labels are one alert3 alerts
notifications Alertmanager sent2 notifications
Fig 7.1Read the arrows left to right: 35 deliveries go into the accent box and 2 notifications come out, both to the pager.

Prometheus sends each firing alert to over HTTP, at /api/v2/alerts. It sends the alert again while it keeps firing, at least once a minute, and for a while after it resolves, marked as resolved. In this run that made 35 deliveries: 8 as firing and 27 as resolved.

Alertmanager counts deliveries that carry the same labels as one alert, so 35 deliveries were 3 alerts. It then groups them, routes each group to a receiver (a named destination such as the pager), mutes any that it has been told to mute, and sends a notification for the rest. It sent 2 notifications, both to the pager.

The split is fixed. Rules and alert states are Prometheus’s half. Who hears about an alert, and how often, is Alertmanager’s half.

STEP 02 / 06

A SILENCE WRITTEN BEFORE THE DEPLOY

+0:00 = Prometheus starts, 19:23:00 UTC
silence · +11:30 → +21:30 · 10 min
+11:00+15:00+19:00+22:00
↑ TargetDown firing +12:39.6 → +13:39.6
↑ +13:09.6 muted · reason=silence
THE SILENCE, AS ALERTMANAGER STORED IT
alertname="TargetDown"instance="checkout-2:8000"
by deployer · “deploy checkout 2.4.1”
deploy at +12:00 · the alert was received and shown as suppressed · 0 notifications sent for it
Fig 7.2The accent band is the silence, +11:30 to +21:30; the alert bar inside it fired and produced no message.

A is a set of matchers (label conditions) with a start, an end, an author and a comment. It is stored in Alertmanager, and you create one in its web UI or with the command-line tool amtool. At +11:30, 30 seconds before the deploy, the deployer silenced alertname="TargetDown", instance="checkout-2:8000" for 10 minutes.

Prometheus has no part in a silence. TargetDown went pending, fired on schedule at +12:39.6, and was delivered. At +13:09.6 Alertmanager’s log reads Notifications will not be sent for muted alerts with reason=silence. The alert existed and showed as suppressed in the UI. No message was sent.

The silence expired at +21:30. By then the alert had been resolved for 7 min 51 s.

THE SILENCE, WRITTEN WITH AMTOOL BEFORE THE DEPLOY
amtool silence add alertname=TargetDown instance=checkout-2:8000 \
  --duration=10m --author=deployer --comment="deploy checkout 2.4.1" \
  --alertmanager.url=http://localhost:9093
# 7825b316-9f18-4ff4-9964-71055bf30f92
#
# what Alertmanager stored for it:
#   matchers   alertname="TargetDown", instance="checkout-2:8000"
#   starts     19:34:30 UTC   (+11:30)
#   ends       19:44:30 UTC   (+21:30)
#   createdBy  deployer
STEP 03 / 06

ONE TREE, TWO RECEIVERS

root routereceiver: team-chat · group_by: [service]every alert enters the tree here
the child route matches
CheckoutHighErrorRate · severity="page"
child routematchers: [severity="page"]receiver: pager
↓ to its receiver
pagerwebhook · /pager2 notifications sent
no child route matches: the root’s receiver
TargetDown · severity="warning"CheckoutHighLatency · severity="warning"
team-chatwebhook · /chat0 notifications sent: both alerts were muted
Fig 7.3Follow each alert down the tree; only the accent path, severity page, ends at the pager.

Every alert enters Alertmanager’s routing tree at the top, the root route. A route names a receiver and may have child routes, each with matchers. Alertmanager checks the children in order and uses the first one that matches. An alert that matches none stays with the root.

A receiver is a named list of integrations: email, Slack, PagerDuty, a webhook and others. Here the root’s receiver is team-chat, and its one child route is matchers: [severity="page"] with receiver pager. Both receivers are webhooks in this stack.

CheckoutHighErrorRate carries severity="page" and goes to pager. The two warnings match no child and stay with the root’s team-chat.

ALERTMANAGER.YML: THE ROUTING TREE AND ITS TWO RECEIVERS
route:
  receiver: team-chat          # the root: every alert starts here
  group_by: [service]
  group_wait: 30s              # default
  group_interval: 5m           # default
  repeat_interval: 4h          # default
  routes:
    - matchers: [severity="page"]
      receiver: pager

receivers:
  - name: pager
    webhook_configs:
      - url: http://webhook:5001/pager
        send_resolved: true
  - name: team-chat
    webhook_configs:
      - url: http://webhook:5001/chat
        send_resolved: false
STEP 04 / 06

THIRTY SECONDS OF WAITING ON PURPOSE

+0:00 = Prometheus starts, 19:23:00 UTC
+17:00+20:00+23:00+26:00+29:00
firing +17:09.6
resolved +23:09.6
CheckoutHighErrorRatethe alert, in Prometheus
+17:39.6
sent · firing
group_wait · 30 s
+22:39.6
checked · no change
group_interval · 5 m
+27:39.6
sent · resolved
the next check
Alertmanagerthe group service="checkout"
the page, +17:39.61,044 bytes · 1 alert
the all-clear, +27:39.61,055 bytes · 1 alert
from the alert resolving to the all-clear4 min 30 s
Fig 7.4The two accent marks are the messages; the grey tick between them is a group check that found nothing new to send.

Alerts on the same route that share the group_by labels, service here, form a group and go out as one notification. This is . group_wait: 30s makes a new group wait before its first notification, so related alerts can join it. The page left at +17:39.6, 30.0 s after the alert fired: 1,044 bytes of JSON carrying 1 alert, accepted in 4.4 ms.

group_interval: 5m sets how often the group is checked again for changes. At +22:39.6 nothing had changed and nothing was sent. The alert resolved at +23:09.6, and the all-clear went out at the next check, +27:39.6. repeat_interval: 4h re-sends a notification for an alert that is still firing.

The all-clear still read “Checkout is failing 5.797% of requests”. That is the value at the last evaluation where the condition was true, 15 seconds before the alert resolved.

THE PAGE AS THE WEBHOOK RECEIVED IT: 1,044 BYTES AT 19:40:39.612 UTC
{
  "receiver": "pager",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "CheckoutHighErrorRate",
        "job": "checkout",
        "service": "checkout",
        "severity": "page"
      },
      "annotations": { "summary": "Checkout is failing 12.7% of requests" },
      "startsAt": "2026-11-13T19:40:09.599Z"
    }
  ],
  "groupLabels": { "service": "checkout" },
  "groupKey": "{}/{severity=\"page\"}:{service=\"checkout\"}"
}
STEP 05 / 06

THE WARNING NOBODY RECEIVED

  1. TargetDownseverity warning · fired +12:39.6
    muted at +13:09.6reason=silence · written by deployer
    stops here: nothing sent
  2. CheckoutHighLatencyseverity warning · fired +18:54.6
    muted at +19:24.6reason=inhibition · the page was firing for the same service
    stops here: nothing sent
  3. CheckoutHighErrorRateseverity page · fired +17:09.6
    sent at +17:39.6matched no silence and no inhibition rule
    pager2 notifications: the page, then the all-clear
3 alerts fired · 1 silenced · 1 inhibited · 1 delivered
Fig 7.5Three alerts enter on the left; the accent one is the only one with a line through to a receiver.

An rule mutes one alert while another is firing. The rule here has source severity="page" and target severity="warning", and it applies when both alerts carry the same service label. In words: while a page is firing for a service, warnings for that service are not sent.

CheckoutHighLatency, a warning for service="checkout", fired at +18:54.6 while the page was already firing. When its group’s 30-second wait ended, at +19:24.6, Alertmanager’s log reads reason=inhibition. The alert resolved at +20:54.6 and was never sent.

The run’s total: 3 alerts, 1 silenced, 1 inhibited, 1 delivered. The two kinds of mute differ in who makes them. A person writes a silence for a span of time. An inhibition is a standing rule in the config, and it applies only while its source alert fires.

ALERTMANAGER.YML: THE ONE INHIBITION RULE
inhibit_rules:
  # while a page is firing for a service…
  - source_matchers: [severity="page"]
    # …warnings for the same service are not sent
    target_matchers: [severity="warning"]
    equal: [service]
# CheckoutHighLatency: fired +18:54.6, muted +19:24.6
STEP 06 / 06

THE GRAPH YOU OPEN NEXT

  • checkout-2
  • checkout-1
requests a second · checkout-2 peak 8.0
0.04.08.0
deploy
rollback
+0:00+12:00+20:00+30:00
share of requests failing · checkout-2 peak 33%
0%20%40%
deploy
rollback
+0:00+12:00+20:00+30:00
99th percentile, seconds · checkout-2 peak 2.46 s
0 s1 s2 s3 s
deploy
rollback
+0:00+12:00+20:00+30:00
requests in flight · checkout-2 peak 5
0246
deploy
rollback
+0:00+12:00+20:00+30:00
4 range queries · 121 points a series, one every 15 s · 36,113 samples read
Fig 7.6Four panels, the same 30 minutes: the accent line is checkout-2, and it leaves the grey line in each after the deploy.

The page says that something is wrong. A dashboard says where. This one has four panels: request rate, error ratio, 99th-percentile duration and requests in flight, each split by (instance). Each panel is one range query: the same expression evaluated at many instants, through /api/v1/query_range. Here that is one point every 15 seconds, 121 points a series. Drawing all four read 36,113 samples.

The panels show what the page did not. checkout-1 stays flat at 0% errors. checkout-2 climbs to 33%, starting right after the gap at +12:00, which is where version 2.4.1 went out. Grafana is the usual tool for dashboards: it uses Prometheus as a data source and holds a PromQL expression in every panel.

The other two signals from chapter 00 finish the diagnosis. A failing request from the same version left the log line payment timed out after 1.2s, and a trace whose payment.charge span took 1,200.1 ms.

WHAT YOU NOW KNOW

  • Prometheus delivers firing alerts to Alertmanager, which deduplicates, groups, routes and mutes them: 35 deliveries became 2 notifications.
  • A silence muted TargetDown, an inhibition rule muted the latency warning, group_wait held the page 30 s and group_interval held the all-clear 4.5 minutes.
  • A dashboard of four range queries showed the fault was one copy on version 2.4.1. The run ends with 402,329 samples.
deep dive: Alertmanager clusters, templates, amtool and dashboard habits

For availability you run several Alertmanagers as a cluster. They share silences and the record of what was already sent over a gossip protocol on port 9094, and Prometheus sends every alert to all of them. Never put a load balancer in front: the cluster only removes duplicates correctly when every instance receives every alert.

resolve_timeout, 5m by default, is when Alertmanager treats an alert as resolved if its sender gave no end time; Prometheus always gives one. send_resolved on a receiver controls whether the all-clear is sent. Notification text comes from Go templates with fields such as .GroupLabels, .CommonAnnotations and .Alerts.Firing.

In the routing tree, continue: true makes an alert keep matching sibling routes after its first match. group_by: ['...'] groups by every label, which turns grouping off. Named time intervals can mute a route for set hours, such as nights and weekends. amtool covers the daily work: check-config, alert query, silence add, silence expire and config routes test severity=page, which prints the receiver a label set would reach.

Dashboard habits: one row of RED panels per service and USE panels per machine, with variables for job and instance. Grafana’s $__rate_interval keeps a rate window from being shorter than 4 scrape intervals. Prometheus’s own UI graphs one expression at a time and still serves console templates. Perses is the CNCF’s dashboard project.

ON THE EXAM

  • “component that sends notifications” → Alertmanager
  • “what Prometheus does with alerts” → evaluates the rules and sends firing alerts to Alertmanager
  • “one notification for many similar alerts” → grouping, group_by
  • “wait before a new group’s first notification” → group_wait, 30s
  • “wait before notifying about changes to a group” → group_interval, 5m
  • “wait before re-sending” → repeat_interval, 4h
  • “mute during maintenance” → a silence
  • “mute because another alert is firing” → inhibition, inhibit_rules with source_matchers, target_matchers and equal
  • “where notifications go” → a receiver
  • “what picks the receiver” → the routing tree and its matchers
  • “default port” → 9093
  • “command-line tool” → amtool
  • “several Alertmanagers” → a gossip cluster, with no load balancer in front
  • “most used dashboard tool” → Grafana
  • “API behind a graph” → query_range

402,329 samples, three alerts, one page

Thirty minutes of one service, counted once more. The board sorts the 402,329 samples by where they came from and by metric type, then follows them to a person: rule evaluations, alerts, deliveries and two notifications. The last two blocks give the clock, and what was never stored at all.

checkout · 2026-11-13 19:23:00 → 19:53:00 UTC · Prometheus 3.16.0
+0:00 = Prometheus starts, 19:23:00 UTC · hollow bars are pending or muted, filled bars are firing
+11:30+13:30+17:10+20:00+23:10+27:40
TargetDown muted
silencewritten by the deployer
firing +12:40
TargetDownseverity warning
firing +18:55
CheckoutHighLatencyseverity warning
firing +17:10
CheckoutHighErrorRateseverity page
the page
the all-clear
notificationssent to the pager
SAMPLES STORED, BY SOURCE
node exporter, the machine172,080 samples
Prometheus, about itself122,646 samples
Alertmanager, about itself85,136 samples
your service, checkout, 2 copies11,650 samples
Pushgateway and the cart-cleanup job6,619 samples
scrape health, up and four scrape_* series3,590 samples
rules, recorded results and alert states608 samples
SAMPLES STORED, BY METRIC TYPE
counters187,241 samples
histograms, buckets with their sums and counts101,698 samples
gauges85,033 samples
summaries18,046 samples
everything else: untyped, scrape health, rules, info10,311 samples
FROM SAMPLES TO A PERSON
rule evaluations, 8 rules every 15 seconds960 evaluations
alerts that fired3 alerts
alert deliveries to Alertmanager35 deliveries
muted by a silence1 alert
muted by an inhibition rule1 alert
notifications sent, a page and an all-clear2 notifications
THE CLOCK
first failed payment to the page4 min 9 s
5-minute window filling 1 min 39 s · for 2 min · group_wait 30 s
rollback to the alert resolving3 min 8 s
alert resolving to the all-clear4 min 30 s
NEVER STORED
requests refused while checkout-2 was stopped720 requests
errors after checkout-2’s last scrape, before the rollback21 requests
samples stored in 30 minutes402,329
Read next. Kubernetes basics: what happens after kubectl apply for where targets like these usually run, and how a cluster replaces the copy that stopped.
job:http_errors_per_request:ratio_rate5m{job="checkout"} > 0.05
Empty query result