The short version
- Read-only for the first four weeks, with no exceptions. The agent concludes; the humans decide; you compare. Nothing it says has any consequence yet.
- Trust is earned on disagreements, not agreements. The weeks where it matched your on-call prove nothing. The weeks where it did not, and said so, are the evidence.
- Write-access expands in the order of reversibility — annotate, then ticket, then close the benign, then remediate behind an approval. Never in the order of how often the agent is right.
- The most common failure is going too fast after a good month. One good month is not evidence; it is a sample.
The programmes that fail do not fail on technology. They fail because somebody switched on write access in week three, it did something surprising in week five, and the on-call team — who never agreed to this — spent the next year citing that incident whenever it came up.
This sequence is slower than it needs to be on purpose. Every phase is designed so that the people who have to live with the thing are the ones who decide it has earned the next step.
Weeks 1–2 · Measure before you install anything
Do this first, because afterwards you will never get a clean baseline and every number you quote will be arguable.
- Escalation rate from L1, and specifically the share escalated with no change made at L1. That is the escalation the information gap caused.
- Median gap between escalation and first action at the receiving tier, separate from handling time. Nobody reports this because it belongs to no team.
- Share of incidents closed with no change made to anything. Usually the surprise.
- Time three incidents by hand. A stopwatch, not a survey. The assembly share from three real incidents is worth more than any benchmark.
In parallel: stand the stack up, point one alert source at it, read-only. One source, because two is how you end up debugging the integration instead of evaluating the product.
Weeks 2–5 · Read-only, and the daily comparison
Convert your three most-fired alerts into runbooks. Not the most interesting ones — the wiki is longest where incidents were most unusual, and the value is highest where they are most frequent. Those are rarely the same pages.
Then run the comparison every day, and keep it to five minutes. For each incident: what did the agent conclude, what did the on-call conclude, and where they differ, who was right.
| Outcome | What it tells you |
|---|---|
| Agreed, agent confident | Little. This is the easy majority and it proves nothing. |
| Disagreed, agent was wrong, low confidence | The result you want. It did not know, and it said so. This is the whole basis for trusting anything else it says. |
| Disagreed, agent was wrong, high confidence | Stop and investigate. Either the evidence was misleading or the verification is not doing its job. Do not proceed until you understand it. |
| Disagreed, agent was right | Happens more than people expect, usually because it checked something the human skipped. Worth showing the team. |
Four weeks is the minimum, and here is why
You need to have seen the agent be wrong, and to have seen how it behaved when it was. A month where everything went well tells you nothing you can act on. If four weeks pass with no disagreements at all, your sample is too easy — point a noisier source at it before proceeding.
Weeks 6–8 · Reversible writes only
First write access, and the ordering is by blast radius, not by confidence: acknowledging a page, adding a note to the incident, opening a ticket. If every one of these goes wrong, the cost is noise.
Acknowledging is the one that changes how the rota feels, and it is worth doing early for that reason: the page stops escalating while the agent works, so the engineer wakes to an incident with evidence rather than to a second page.
Weeks 9–11 · Let it close things
Now the agent may resolve incidents on its own — but only in categories where the evidence settles the matter, and only above a confidence floor you set.
Start the floor high, around 0.9. It will close very little. That is correct: you are establishing that the ones it closes are right, not maximising throughput. Lower it in steps, on evidence from your own incidents, and record who lowered it and when — that single decision is what a reviewer will ask about first.
This is also the point to enable watch-and-wait on lag and backlog alerts, because it is the highest-value change with almost no risk: the worst case is the incident escalates a few minutes later than it would have.
The mistake to avoid here
Do not lower the floor because the auto-resolution rate looks low. The rate is an output, not a target. Teams that chase it end up with a floor tuned to a number rather than to the evidence, and the first confidently-wrong closure undoes the whole programme.
Weeks 12–13 · One gated remediation
One. Pick the safest thing your runbooks do — typically a restart of a stateless service, or a cache clear — and put it behind an approval gate. A named person answers; the agent resumes within seconds; the answer, who gave it and when are on the record permanently.
What this phase is really testing is not the remediation. It is whether your on-call team finds answering the approval faster than doing the work themselves. If they do, the programme has landed. If they are ignoring approvals and fixing it by hand, something earlier did not earn their trust and no further automation will fix that.
What to report at the end
| Number | Why this one |
|---|---|
| Escalation rate, before and after | The clearest evidence of the information gap closing. Yours, not a benchmark. |
| Runbook coverage | The share of incidents a runbook matched. Also tells you what to write next. |
| Agreement rate, split by confidence | The trust metric. High agreement when confident, honest uncertainty when not. |
| Incidents needing no action | Before and after. This is where watch-and-wait shows up. |
| Do not. It is unfalsifiable, everyone knows it, and including it undermines the four numbers above. |
Next in this series
How to tell whether it is working — the four numbers worth tracking after the ninety days, and the one almost nobody does.