The most common production incident, and the one where automation is most tempted to overreach. The 5xx rate crosses five per cent, a release went out forty minutes ago, and the pattern is so familiar that both a human and a model will jump to the same conclusion — often correctly, and sometimes expensively.
This post is about running that incident end to end, and about the step that stops the familiar conclusion from being stated as more certain than it is.
The five questions, in order
Watch an experienced engineer handle this and they ask the same five things, in roughly the same order, every time. That consistency is what makes it automatable.
happening?"] --> B["2 Is the service
itself healthy?"] B --> C["3 Did anything
change?"] C --> D["4 Is a dependency
failing?"] D --> E["5 What do the
logs say?"] E --> F["A hypothesis"] F --> G{"Does the evidence
establish it?"} G -->|"yes"| H["Act, or close"] G -->|"only correlation"| I["Escalate, with
the gap named"] style G fill:#fff4e2,stroke:#f4a03a,stroke-width:2px style I fill:#f4f6ff,stroke:#a78bfa
Steps one to five are mechanical. Everything interesting is in the diamond, and the diamond is where most automation in this category simply asserts an answer.
The runbook
Stops the page escalating while the agent works. This is the one mutating step, and the dry run names it before activation.
error_rate
Is it still happening? A surprising share of pages have already recovered by the time anyone looks.
pod_status
Restart counts and readiness. Distinguishes “the service is failing” from “the service is fine and something it calls is failing”.
recent_deploys
The deploy-correlation question, with an explicit window so the answer does not depend on who asks.
dep_health
The hunch that it is always the payment service, made checkable.
error_logs
Conditional — run_if {{error_rate}} > 0.05. No point reading logs for an incident that already recovered.
classification
Weighs the five pieces of evidence and names a cause with a stated confidence.
review
The step this post is about. Says what the conclusion rests on and caps the confidence where a link is assumed.
Skipped for false positives. skip_if {{classification.category}} == 'false_positive'
Auto-resolves only when the reviewed confidence clears the floor and the cause is benign. Note it reads {{review.confidence}}, not the classifier's original number.
What the agent concluded, and what it admitted
Run against the demo stack with a 30% injected error rate and a "v2.4.1 deployed" marker, the classifier returned bad_deploy at 0.82. Reasonable. An experienced engineer would have said the same.
Then step eight checked it against the evidence alone, and the number came down to 0.75.
| Status | Claim | Established by |
|---|---|---|
| Established | A deployment happened inside the alert window | recent_deploys |
| Assumed | The errors are attributable to that deployment | error_rate — timing only |
And the gap, stated plainly on the incident: "The link is temporal. A rollback that clears the errors, or the same signature absent in the previous version, would settle it."
Why this matters more here than anywhere else
Bad-deploy is the classification an agent will reach most often, and it is the one where being confidently wrong is most expensive — because the obvious next action is a rollback, and rolling back the wrong release during an incident caused by something else costs you the rollback window and the trust of the team that shipped it.
Closing the gap properly
That "assumed" is not a limitation to be tolerated. It is a gap with a known fix, and this is the strongest argument for connecting a change system.
v2.4.1 at T-40min"] --> B1["Claim: the errors are
attributable to it"] B1 --> C1["Assumed
confidence capped at 0.75"] end subgraph With["With ServiceNow or the deploy history"] A2["Change CHG0031482 against
CI api-gateway, window T-45 to T-30"] --> B2["Claim: the errors are
attributable to it"] B2 --> C2["Established
confidence stands"] end style C1 fill:#fff4e2,stroke:#f4a03a style C2 fill:#e8f8f4,stroke:#22c1a4,stroke-width:2px
A change record naming the same configuration item inside the window turns the causal claim
from assumed into established, and the confidence ceiling lifts. servicenow.recent_changes
does this against the change calendar; GitHub, GitLab or Argo deployment history does the
same thing for teams that deploy from git. Either way, it raises the ceiling on every
RCA the product writes, which is why we built the ServiceNow connector before the
remaining clouds.
The three ways this incident ends
| Evidence | Classification | Outcome |
|---|---|---|
| Rate recovered before the agent looked; no deploy; pods healthy | transient_spike, high confidence | Auto-resolved. Above the floor and benign. Logged with the readings. |
| Deploy in the window, logs name a dependency the new version talks to | bad_deploy, capped at 0.75 | Escalated with the evidence, the assumed link and what would settle it. |
| No deploy; dependency returning connection refused; pods healthy | upstream_dependency_failure | Escalated to the dependency's owner, with their service named. |
Only the first closes on its own, and only because the evidence settles it without a causal leap. That ratio — roughly one in three of these in a typical estate — is the honest version of "auto-resolution rate" for this alert class.
Adapting it
- Replace step three with what you run. On ECS it is
ecs.services; on Azure,azure.metricsagainst the app service. The shape of the runbook does not change. - Point step five at your actual dependency. The seeded version checks a payment service because that is the demo estate. Yours has a different usual suspect, and the value of the step is entirely in naming it correctly.
- Keep the acknowledge step first. It stops the page escalating while the agent works, which is the difference between a useful assistant and one that runs alongside a paging storm.
- Read
{{review.confidence}}in the gate, not{{classification.confidence}}. Otherwise the verification runs and is ignored, which is worse than not running it.
Next in this series
Latency: when p95 moves and nothing else does — the incident where the alert is rarely about the thing it is named after.