Below is a single incident, timed. Nothing exotic: an error-rate page on a payments-adjacent service, at 02:14, handled by a competent engineer following a runbook that existed and was roughly accurate. It resolved in forty-one minutes, which most teams would call a good night. Thirty-four of those minutes produced no diagnosis and no fix.
The point of writing it out minute by minute is that summary metrics hide the shape. "MTTR of 41 minutes" tells you nothing about which minutes to attack.
The incident, as it happened
| Elapsed | What happened | Produced |
|---|---|---|
| 00:00 | Prometheus fires HighErrorRate, routed via Alertmanager to PagerDuty. Push notification. | — |
| 00:00–04:30 | Engineer wakes, finds laptop, connects VPN, acknowledges the page. | Nothing |
| 04:30–06:00 | Reads the alert. Learns the service name and that the 5xx rate crossed five per cent. | The service name |
| 06:00–09:30 | Opens Grafana, finds the right dashboard on the third attempt, confirms the rate is still elevated at about twenty-five per cent. | Still happening |
| 09:30–13:00 | Opens the logs tool. Constructs a query. Sees connection refused against a downstream dependency. | A symptom |
| 13:00–17:00 | Checks whether the dependency is itself alerting. It is not. Checks its dashboard. It looks healthy. | A contradiction |
| 17:00–21:00 | Opens the deploy pipeline. Finds a release to the alerting service forty minutes before the page. | A suspect |
| 21:00–26:00 | Opens the pull request. Reads the diff. A connection-pool setting changed. | A hypothesis |
| 26:00–30:00 | Posts in Slack to check nobody is mid-experiment. Waits. No reply at 2:40am. | Nothing |
| 30:00–33:00 | Decides to roll back. Finds the rollback procedure. Runs it. | An action |
| 33:00–39:00 | Watches the error rate. It falls below threshold. | Confirmation |
| 39:00–41:00 | Resolves the page. Writes two sentences in the ticket. Goes back to bed. | A thin record |
Where the time actually went
Sort those rows by what they were for rather than when they happened, and the shape becomes obvious.
Twenty-one and a half minutes — more than half the incident — was spent locating and reading information that already existed, in systems that were already running, about a service that was already instrumented. Nothing was being discovered. It was being fetched.
Seven minutes went on the part that genuinely needed a human: connecting a pool-size change to a connection-refused error, and judging that a rollback was safer than a forward fix at 2:40 in the morning. That judgement is worth paying an experienced engineer for. The twenty-one minutes of fetching that preceded it are not.
The four minutes nobody counts
Waking up, finding the laptop and connecting the VPN is four and a half minutes in this timeline, and it is unavoidable for a human. It is not unavoidable for the gathering: there is no reason the evidence cannot already be assembled by the time the engineer's screen comes on.
What the record did not capture
The ticket says, in full: "Error rate spike caused by pool size change in v2.4.1, rolled back." That is accurate and almost useless.
It does not record what the error rate actually was, or that the dependency looked healthy from the outside — which was the confusing part and the thing most worth knowing next time. It does not record which log query surfaced the symptom. It does not record that the engineer waited four minutes for a Slack reply that never came, which is a process finding, not a technical one.
So when the same alert fires in five weeks and a different engineer picks it up, almost none of this is available to them. They will spend their own twenty-one minutes fetching. The organisation learned nothing, not because the engineer was careless, but because writing all that down at 02:55 is not a reasonable thing to ask of a person who has to work tomorrow.
The arithmetic, with your own numbers
One incident is an anecdote. The reason to time one carefully is to multiply it.
Take a mid-sized estate: 40 actionable pages a month reaching L2 or above, and assume the shape above holds — roughly half the elapsed time spent fetching. That is an assumption, not a measurement, and it is the one number worth replacing with your own before you trust anything below it. At 21 minutes of fetching per incident:
- 40 incidents × 21 minutes = 14 hours a month of pure lookup.
- At a fully loaded L2 cost of $95/hour, that is about $16,000 a year in direct labour.
- Roughly a third of those pages fall outside business hours, so some of that is at out-of-hours rates or in time off in lieu.
Sixteen thousand dollars is not a number that funds a platform purchase, and we would rather say so than inflate it. The direct labour saving is real but modest. The case is elsewhere, and it is worth being precise about where.
| Effect | Why it is larger than the labour saving |
|---|---|
| Elapsed time, not effort | Those 21 minutes are minutes the customer is also experiencing the incident. For a revenue-bearing service, the cost of the outage window usually dwarfs the cost of the engineer's hour. |
| Escalations avoided | A meaningful share of pages escalate only because the person holding them could not reach the evidence to be sure. Each avoided escalation removes a whole handoff, not just a few minutes. |
| The record that now exists | The second occurrence of a known incident is dramatically cheaper than the first — but only if the first was written down properly. Most are not. |
| Rota sustainability | Harder to put on a spreadsheet, and the thing your senior engineers will actually notice. A 3am page that arrives with the evidence attached is a different experience from one that arrives as a service name. |
The same incident, with the fetching already done
Run the same page through a system that gathers first. The alert fires; before anyone has acknowledged it, a runbook has confirmed the error rate, pulled the pod status, listed releases in the window, fetched the error log lines, and checked whether the dependency is healthy.
Twenty-one minutes instead of forty-one, and — more interesting than the total — the human minutes are now almost entirely the ones that needed a human. The engineer reads, judges, acts and confirms. They do not fetch.
Note what has not changed. The rollback is still a person's decision. The agent gathered the deploy correlation but did not act on it, because correlation inside a time window is not proof of causation — a distinction we treat as load-bearing rather than decorative, and which we cover in the agent that says "I don't know".
Two ways to read this
If you run the support org
Time three incidents yourself this month, by hand, with a stopwatch and a notepad. Not a survey — an observation. The fetching share is the single most useful number you can take into a budget conversation, and almost nobody has it.
Then resist quoting the labour saving as the headline. It is the weakest part of the case and the easiest for finance to discount. Lead with elapsed customer-facing minutes and with escalations avoided.
If you are evaluating automation
The gathering has to happen before the human arrives, or it saves nothing. A system that starts gathering when someone clicks a button has moved the work, not removed it. Ask when execution begins relative to the page.
And check what the record looks like afterwards. If the output is a chat transcript rather than a structured artifact with the evidence attached, the second-occurrence saving — the biggest one — never materialises.
Next in this series
Where AI actually helps with incidents — and where it does not. The fetching is mechanical and should be automated. The rollback decision at 2:40am is not, and pretending otherwise is how these projects fail.