This is the post about the step that decides whether anyone reads the other nine. An agent that gathers evidence beautifully and then states a confident wrong conclusion is worse than no agent, because the first time an engineer catches it they stop reading its output — permanently, and they tell their colleagues.
So the design question is not how to be right more often. It is: when the agent is wrong, does the record show that it did not know?
The problem with a confidence score
Ask a language model to classify an incident and return a confidence, and it will return one. It will be a plausible number. It will not be a measurement of anything.
This is not a criticism of models; it is a category error in how they are being used. The model is producing the most likely continuation of a prompt that asked for a confidence. It has no access to a calibration set, no record of how often conclusions of this shape have been right before, and no way to distinguish "the evidence strongly supports this" from "this is the kind of thing that is usually true".
The specific failure
"Root cause: database connection pool exhaustion. Confidence 92%." The engineer checks; the pool is at forty per cent. The wrong answer is forgivable. The 92% is not, because it was asserted rather than derived, and once an engineer knows that, every subsequent number from the system is worthless to them — including the correct ones.
Separating the claim from its support
The approach InsightResolve takes is to stop treating the conclusion as one thing. A classification is decomposed into the individual claims it depends on, and each claim is checked against the gathered evidence on its own.
Take the most common classification in production: bad deploy. It is not one assertion. It is two, and they are very different.
bad_deploy, stated at 0.82"] C --> C1["Claim 1:
A deployment happened
inside the alert window"] C --> C2["Claim 2:
The errors are attributable
to that deployment"] C1 --> V1["Check the evidence"] C2 --> V2["Check the evidence"] V1 --> R1["Established
recent_deploys shows
v2.4.1 at T-40min"] V2 --> R2["Assumed
the evidence shows timing,
not causation"] R1 --> O["Verdict: partly supported
confidence capped at 0.75"] R2 --> O O --> G["Gap stated:
"a rollback that clears the errors,
or the same signature absent in the
previous version, would settle it""] style R1 fill:#e8f8f4,stroke:#22c1a4 style R2 fill:#fff4e2,stroke:#f4a03a,stroke-width:2px style O fill:#f4f6ff,stroke:#a78bfa,stroke-width:2px
Claim one is checkable: either a deploy appears in the evidence inside the window or it does not. Claim two is not checkable from timing alone, and that is the entire point. Two things happening in the same ten minutes is correlation. The agent marks it assumed, and an assumed link caps the confidence at 0.75 regardless of what the model first said. A claim nothing in the evidence supports caps it at 0.4.
Why the verifier is shown less, not more
A design decision worth explaining because it is counter-intuitive: the verification step sees the evidence and the conclusion, but not the reasoning that produced the conclusion.
Give a critic the argument and it tends to agree with it — the reasoning is persuasive, that is what it was generated to be. Give it only the facts and the claim, and it cannot manufacture support that is not in front of it. Withholding context makes the check stronger, which is the opposite of the usual instinct with these systems.
And it works without a model at all
With no model configured, verification falls back to rules: does each thing this category of conclusion leans on actually appear in the evidence, and is the causal link stated or merely temporal? The rules are cruder and the confidences come out lower — which is the correct behaviour, not a degraded one. Every artifact records whether a model or the rules decided, so a reader always knows which they are looking at.
What the reader actually sees
On the incident, this renders as a block headed what this rests on. Not a percentage — a list.
| Status | Claim | Where it came from |
|---|---|---|
| Established | A deployment happened inside the alert window | recent_deploys |
| Assumed | The errors are attributable to that deployment | error_rate — timing only |
Followed by still worth checking: the specific thing that would settle it. That sentence is more useful to an on-call engineer than any score, because it tells them what to do next rather than how to feel about what they have been told.
What it costs to be this careful
Honesty has a price and it is worth naming, because a vendor that claims otherwise is not being straight with you.
- Fewer incidents auto-resolve. A capped confidence often falls below the floor, so the incident escalates that would otherwise have closed. That is a real cost in exchange for not being confidently wrong in front of your on-call team.
- The artifacts are longer. A bare classification is one line; this is a paragraph and a table. Engineers reading at 3am mostly want the one line, so the summary leads and the detail sits underneath.
- It makes the product look less capable in a demo. "Confidence 92%" demos better than "partly supported, 0.75, and here is what would settle it". We think the second is the one that survives contact with a real estate, but the first wins bake-offs.
The thing to ask any vendor
Not "how accurate is it". Ask to see a case it got wrong, and look at what its confidence said at the time. If they cannot produce one, they either have not looked or will not show you — and both answers tell you the same thing.
Next in this series
Not every alert needs an action — the discipline of waiting, and why the hardest thing to build into an automated system is the decision to do nothing.