A latency page is almost never about latency. It is about saturation, or a slow dependency, or a change in what the traffic is asking for — and the p95 is just the first place that showed up. Which makes this the alert class where gathering evidence matters most and where a single number tells you least.
Why the percentile alone is not actionable
"p95 is 975ms, threshold is 500ms" tells an engineer almost nothing they can act on. The same reading is produced by at least five different situations, and the response to each is different.
| What is actually happening | The tell | Response |
|---|---|---|
| More traffic than usual, same work per request | Throughput up, latency up together | Scale, or accept it |
| Same traffic, a dependency got slower | Throughput flat, dependency latency up | Chase the dependency's owner |
| Resource saturation — CPU, connections, threads | Throughput flat or down, saturation metric pinned | Scale or fix the limit |
| A release changed the work per request | Step change coinciding with a deploy | Roll back or accept |
| A small number of pathological requests | p99 far worse than p95; p50 unchanged | Find the query, not the service |
Notice that distinguishing them requires three or four signals compared against each other, not one signal read more carefully. That is precisely the work that is mechanical, repeated identically every time, and eats the first twenty minutes of every latency incident.
The runbook
p95
A range query rather than an instant one — the shape over thirty minutes distinguishes a step change from a gradual climb, and that distinction picks out the deploy case immediately.
rps
The single most diagnostic comparison in this incident class. Latency up with throughput is capacity; latency up without it is something else.
recent_deploys
A step change in the p95 that lines up with a release is the easiest of these to resolve, and the easiest to miss without looking.
summary
Writes the comparison out in a sentence, so the engineer reads a picture rather than four panels.
Puts the summary in the channel where the on-call already is. A mutating step — the dry run names it before activation.
Five steps and no remediation. That is deliberate: there is no safe automatic action for a latency alert. Scaling might be right, or might mask a leak. Rolling back might be right, or might be unrelated. What the runbook does is remove the twenty minutes of assembly and hand over a comparison.
The comparison that does the work
more requests, same work each.
Scale, or accept it."] Q -->|"flat"| Q2{"Dependency
latency"} Q -->|"down"| C2["Saturation or a stall
serving fewer and slower is
a queue that is not draining."] Q2 -->|"also up"| C3["Upstream
not your problem to fix,
but your problem to chase."] Q2 -->|"flat"| Q3{"Step change at
a deploy?"} Q3 -->|"yes"| C4["Release
the work per request changed."] Q3 -->|"no"| C5["Look at the distribution
p99 vs p50 — a few pathological
requests, not a sick service."] style C2 fill:#fde8e8,stroke:#e57373 style C4 fill:#fff4e2,stroke:#f4a03a
That decision tree is not encoded as branching steps, and the reason is worth explaining. Each branch needs a judgement about what "up" means for a noisy series, and encoding those thresholds per service is a maintenance burden that outweighs the benefit. Instead the runbook gathers all four signals and the summarise step states the comparison; the engineer walks the tree in three seconds with everything in front of them.
Where we drew the line
This is a case where automating the gathering is clearly right and automating the conclusion is marginal. Latency has too many causes that look alike on the metrics, and a classification here would be assumed more often than established — which the verification step would correctly cap at 0.4, at which point the confidence is doing no work. Better to gather well and hand over.
Two extensions worth making
Add the dependency you actually suspect. The seeded runbook is generic. Every
real service has two or three downstreams that are the usual cause, and a
prometheus.instant against each of their latencies turns "flat throughput,
something slower" into "the payment service p95 tripled at 02:14".
Add the distribution, not just the percentile. Querying p50, p95 and p99 together separates "everything is slower" from "a few requests are pathological", and those are different investigations. It costs one extra step and removes a whole category of wild-goose chase.
| Signature | What it means |
|---|---|
| p50 up, p95 up | Everything is slower. Capacity, saturation or a dependency in the common path. |
| p50 flat, p99 way up | A minority of requests are pathological. Find the query or the tenant, not the service. |
| p50 flat, p95 up modestly | Often a traffic-mix change — more of an expensive endpoint. Not a fault at all. |
Should it watch instead?
A reasonable question given the lag runbook: latency spikes often recover on their own too, so why not park and re-check?
Sometimes yes, and the pattern is easy to add — a monitor step on the p95
with the alert's own threshold. But be careful about the difference. A backlog draining is
a system recovering from a known perturbation. A latency spike recovering may mean the load
passed, or may mean a dependency's own incident resolved, or may mean an autoscaler caught up
— and the last two are worth knowing about even though the symptom cleared.
The rule we would suggest
Watch it when the alert is reliably self-clearing and nobody would act on the cause anyway. If the cause matters even after the symptom passes — and for latency it usually does — gather, hand over, and let a person decide whether to chase it.
Next in this series
Service down — where speed matters most, and where the agent should still ask before restarting anything.