The short version
- Boards are asking for AI adoption and CIOs are being handed chatbots — which are hard to measure, easy to misuse, and produce nothing an auditor can read.
- Incident triage is a better first bet on four specific grounds: the scope is bounded, every conclusion cites evidence, the actions are reversible, and the process produces its own audit trail as a by-product rather than as an afterthought.
- "Explainable" is not the bar. An explanation is a story generated after the fact. An audit trail is a record of what was actually consulted. Regulators increasingly want the second.
- The governance questions are answerable here — who approved it, what it read, what it changed, what it would have done differently — which is not true of most AI pilots.
There is a specific conversation happening in a lot of boardrooms. The board has asked what the organisation is doing about AI. The CIO has a portfolio of pilots. Most of them are assistants — a copilot here, a chatbot there — and none of them has a number attached that would survive a follow-up question.
The difficulty is not that those pilots are bad. It is that they are hard to defend: hard to measure, hard to bound, and hard to explain to a risk function whose first question is what happens when the model is wrong.
What makes an AI use case defensible
Four properties, and they are worth testing any candidate against. Incident triage happens to score well on all four, which is why it is a better place to start than the obvious ones.
Can you state exactly
what it may touch?"] B["Attributable?
Does each conclusion
name its evidence?"] C["Reversible?
If it acts and is wrong,
what is the cost?"] D["Recorded?
Is there an artifact
an auditor can read?"] end A --> R["Incident triage:
yes to all four"] B --> R C --> R D --> R style R fill:#e8f8f4,stroke:#22c1a4,stroke-width:2px
| Property | A general assistant | Incident triage |
|---|---|---|
| Bounded | Open-ended. It can be asked anything, which means the risk surface is whatever your staff type into it. | A runbook is a finite list of steps against named systems. You can print the whole surface on one page. |
| Attributable | A fluent answer with no traceable source. Citations, when present, are often generated. | Each claim names the step that produced the evidence, and an assumed link is marked assumed. |
| Reversible | Depends entirely on what the human does with the output, which you do not control. | Read-only by default. Anything that writes is marked, gated, and bounded by a policy you set. |
| Recorded | A chat transcript, if anyone kept it. | A structured artifact per incident: every step, parameter, reading, claim and decision. |
Explainable is not the same as auditable
This distinction does more work than any other in the governance conversation, and it is worth being precise about.
An explanation is generated after the decision. Ask a model why it concluded something and it will produce a plausible account. That account may or may not correspond to what actually drove the output, and you have no way to tell which. It reads well and proves nothing.
An audit trail is captured during the decision. These six queries ran, against these systems, at these times, and returned these values. This conclusion was reached. This claim in it is established by that query; this other claim is assumed, and here is what would settle it. Confidence was reduced from 0.82 to 0.75 for that reason.
The test that separates them
Ask whether the record would be identical if the model had been replaced by a different model. The evidence, the queries, the readings and the timestamps would be. Only the interpretation would change — and it would change visibly, because it is recorded separately from the facts it rests on. That separation is what an auditor needs and what a generated explanation cannot provide.
Where the regulation is heading
Without overstating what any specific regime requires today, the direction across the emerging frameworks is consistent, and it favours this shape of use case.
- Risk tiering by consequence. Obligations scale with what the system can affect. A read-only triage agent that escalates to a human for anything consequential sits at the low end by construction, and you can demonstrate why.
- Human oversight that is real. Not a person nominally in the loop, but a named individual who approved a specific action and whose approval is recorded. An approval gate that suspends execution until someone answers is exactly this.
- Record-keeping and traceability. Inputs, outputs and the basis for decisions, retained. A structured incident artifact is already this; a chat log is not.
- Operational resilience. For financial services in particular, the question is what happens when the AI component is unavailable. A system that degrades to rules and says so answers it; one that cannot run without a model does not.
That last point deserves emphasis because it is the one most often missed in procurement. If the model endpoint is down at 3am, does incident response stop? If the honest answer is yes, you have introduced a dependency your resilience testing has to cover.
The questions to put to any vendor
These are the ones a risk function will eventually ask, so it is cheaper to ask them first.
complete record"] --> A1{"Is it a structured
artifact or a
chat transcript?"} Q2["Turn off the
model endpoint"] --> A2{"Does it stop, or
degrade and
say so?"} Q3["Show me a case
it got wrong"] --> A3{"Did its confidence
reflect that at
the time?"} Q4["Who approved
this action?"] --> A4{"A named person,
recorded — or
a policy default?"} style A3 fill:#fff4e2,stroke:#f4a03a,stroke-width:2px
Question three is the one that matters most and the one vendors are least prepared for. A system that is right eighty per cent of the time and confident about the other twenty is unusable in a regulated environment, because you cannot tell the two apart until after the consequence. One that is right sixty per cent of the time and flags its own uncertainty is something you can build a control around.
What to put in the board pack
Concretely, this use case gives you four things to say that most AI pilots cannot.
- A measurable before and after from your own ticket system — escalation rate, time-to-first-action, the share of incidents closed with no change made. Not a vendor benchmark.
- A stated blast radius. "It reads these systems. It may close incidents in these two categories above this confidence. Everything else waits for a named person." That sentence is a control, and it is auditable.
- An artifact per incident that a reviewer can read without needing the engineer who was on call.
- A degradation story. With no model available it falls back to rules, escalates more, and labels every artifact accordingly. Nothing stops.
What not to put in the board pack
A percentage MTTR improvement borrowed from a vendor's case study. You will be asked where it came from, and the honest answer — another company, another estate — will undermine the parts of the case that are genuinely yours.
Three ways to read this
If you answer to the board
This is a use case where the governance answer is stronger than the efficiency answer, which is unusual and worth exploiting. Lead with bounded, attributable, reversible and recorded; let the numbers follow.
It also gives your risk function something to say yes to early, which makes the next AI proposal easier rather than harder.
If you run the support org
The confidence floor for autonomous resolution is a policy you own. Start it high, and move it only on evidence from your own incidents.
Record who set it and when. That single decision is what a reviewer will ask about first.
If you will implement it
Build the degradation path before the happy path. What the system does with no model, an unreachable metrics store or an expired credential is what determines whether it is deployable in a regulated estate.
And keep the evidence separate from the interpretation in your data model. If they are one blob, you cannot answer the auditor's question.
Next in this series
Absorbing an estate — why post-acquisition tool consolidation never finishes, and what to do instead when the integration plan gets descoped in the second quarter.