The short version
- Your incident response depends on a handful of people who know things nobody wrote down. The org chart does not show this dependency, and neither does any risk register we have seen.
- The wiki was the attempt to fix it, and it failed structurally — not through indiscipline. Prose cannot be executed, so it cannot be tested, so it drifts one deploy at a time.
- An executable runbook is a different kind of object. It can be tested before it is trusted, it fails loudly when the estate moves underneath it, and it does not leave when its author does.
- This is the argument that ages best. Cost savings get re-litigated every budget cycle; a capability your organisation owns rather than rents from four individuals does not.
Every technology organisation has one: the engineer who, three minutes into an incident, says "check the payment gateway" and is right. Nobody taught them that. They learned it from an outage in 2023 that took six hours, and it has lived in their head ever since.
That person is one of your most valuable assets and one of your least documented risks. When they are on holiday the same incident takes four times as long. When they resign, the knowledge does not transfer — it evaporates, and you discover the shape of what you lost during the next outage rather than during the handover.
Why the wiki did not solve this
Every organisation has already tried to fix this, and the attempt is sitting in Confluence right now. It is out of date, everyone knows it is out of date, and nobody trusts it enough to follow it at 3am without checking everything anyway.
The usual explanation is discipline: people did not maintain it. That explanation is wrong, and acting on it — another documentation drive, another quarter of "runbook hygiene" — produces another wiki that decays on the same curve.
The real reason is structural, and it is worth stating as a chain because each link is load-bearing:
executed"] B --> C["Cannot be
tested"] C --> D["Nothing detects
when it goes stale"] D --> E["Drifts, one
deploy at a time"] E --> F(["Nobody trusts it,
so nobody maintains it"]) F -.->|"and the knowledge
returns to heads"| A style A fill:#fde8e8,stroke:#e57373 style F fill:#fde8e8,stroke:#e57373,stroke-width:2px
The loop closes on itself. A document that cannot tell you it is wrong will be wrong, and once it is known to be wrong it stops being maintained, which returns the knowledge to where it started. No amount of process discipline breaks that cycle, because the cycle is a property of the artifact rather than of the people.
What changes when the runbook is executable
An executable runbook is not a better-written document. It is a different kind of object, with properties prose cannot have.
| Property | A wiki page | A runbook held as data |
|---|---|---|
| Can be tested | No. You find out it is wrong during an incident. | Yes. A dry run walks every step against a sample alert and reports what would break. |
| Fails loudly | No. A renamed service silently invalidates a paragraph. | Yes. A step referencing a variable nothing sets is an error before activation, not a surprise at 3am. |
| Versioned with intent | A page history nobody reads. | A diff per change, with the reason, attached to the incidents that ran each version. |
| Accrues automatically | Only if someone writes it up afterwards, which at 02:55 they will not. | Every execution leaves its evidence and outcome on the record without anyone typing. |
| Survives its author | Depends entirely on how well they wrote it. | The steps are the knowledge. They run whether or not the author is still employed. |
That last row is the whole argument. The knowledge stops being a description of what someone would do and becomes the thing that is actually done.
What this does not mean
It does not mean every piece of judgement can be encoded. The engineer who decides a rollback is safer than a forward fix at 2:40am is exercising something a runbook cannot hold. What can be encoded is everything that engineer did before reaching that decision — which, when we timed it, was over half the incident.
The transfer, in practice
The question a CIO asks next is fair: does this actually happen, or does the knowledge simply move from a wiki nobody maintains to a runbook library nobody maintains?
The difference is where the maintenance pressure comes from. A wiki decays silently and nothing complains. A runbook that references a renamed service fails on its next execution, visibly, on an incident, with a message naming the step. The estate itself applies the pressure, which is the only kind of maintenance discipline that survives a busy quarter.
a service that no longer resolves" Eng->>RB: fixes the step, versioned with a reason Note over RB: the library is now correct again,
because being wrong was visible Est->>RB: a new failure mode appears Eng->>RB: adds the check they just ran by hand Note over RB: the library grew, because adding
a step is cheaper than remembering
There is a second-order effect worth naming. Once the checks an expert would run are written down as steps, the gap between your best on-call engineer and your newest one narrows. Not because the newest engineer got better, but because the starting position stopped depending on who happened to be holding the pager.
How to measure your exposure
Key-person risk in incident response is measurable, and most organisations have never tried. Three ways, in increasing order of how uncomfortable the answer tends to be.
- Bus-factor by alert. For your top ten most-fired alerts, how many people could handle each one without escalating? If the median is one or two, you do not have a support organisation for those alerts; you have a person.
- Holiday variance. Compare MTTR for the same alert class when your most experienced on-call engineer is available versus when they are not. The ratio is the value of what is in their head, and it is usually larger than anyone expects.
- Onboarding time to independent on-call. How long before a new hire takes the pager alone? That duration is how long it takes to rebuild the knowledge from scratch, one person at a time, for ever.
The question to ask in your next risk review
Not "do we have runbooks?" — you do, and they are stale. Ask: if the three people who know this estate best all left in the same quarter, what would the organisation still be able to do? The honest answer is usually the strongest argument for this work that anyone will make.
Three ways to read this
If you own the capability
This is the argument that survives a budget cycle. Cost cases are re-litigated every year; a capability the organisation owns rather than rents from four individuals is a structural change to your risk position.
It also answers a board question you are likely to be asked about AI adoption: what did we build that we keep? Executable runbooks are an asset with a name.
If you run the support org
Measure bus-factor per alert before anyone asks you for it. It is the number that makes the case for time to write runbooks, which is otherwise the first thing cut.
And make runbook authoring part of incident closure, not a separate initiative. The moment to write the step down is the moment someone just ran it by hand.
If you will implement it
The property that matters is whether a runbook fails loudly when the estate moves. Ask to see what happens to a runbook whose service was renamed — if the answer is a silent skip, you have rebuilt the wiki with extra steps.
Check that versions carry a reason and are linked to the executions that ran them. Without that, you cannot tell which change broke what.
Next in this series
The AI use case you can actually defend to your board — why incident triage is a better first bet than a chatbot, and what makes an AI decision auditable rather than merely explainable.