The whole loop, and what happens at each step

Five stages, and the interesting decisions are not where you would expect. Receiving an alert is mostly plumbing. Matching it to a runbook is where teams usually over-engineer. Executing is where most products stop. Resolving is where honesty matters most — and learning is the stage almost everyone skips, which is why the second occurrence of an incident costs the same as the first.

1 · Receive

Twelve sources arrive at /api/webhooks/{source}: PagerDuty and Opsgenie for paging; Alertmanager, Grafana and Datadog for monitoring; CloudWatch, EventBridge, Azure Monitor, Cloud Monitoring and OCI for the clouds; ServiceNow; and a generic shape for anything else. Each is normalised to one alert — source, name, service, severity, labels, an external id and whether the source says it has cleared.

The part of this worth explaining is deduplication, because it is where the subtle failure lives. An incident is identified by a fingerprint over source, alert name, service and the key labels. That works for repeats. It does not reliably work for closes.

2 · Match

A match rule is a small set of conditions — alert name (globbed), service, severity, source, required labels, an optional expression. The rule with the most specific match wins, with rule priority and the runbook's historical effectiveness as tie-breakers.

The temptation here is to reach for similarity search: embed the alert, find the nearest runbook. We deliberately do not. Matching needs to be explainable and stable — an on-call engineer must be able to answer "why did this runbook run?" with a sentence, and the answer must not change because someone added an unrelated runbook last week. So the incident records the reasons: alert_name ~ HighErrorRate*, service ~ api-gateway, severity in [critical, high], labels ⊇ {env: production}.

When nothing matches, the incident is not dropped. It goes to AI triage with whatever context exists, and it appears on the Analytics page under alerts no runbook matched — which is the list of the next runbooks worth writing.

3 · Execute

The matched runbook is executed step by step, in order, with each step's output saved under a name that later steps can reference. Here is a real one from the demo stack.

Three things about execution are worth drawing out.

Steps can park. A monitor step that has not cleared yet, or a human.approve step waiting for a person, suspends the whole execution with a resume time. It does not hold a worker or a thread. The agent picks it back up by itself, rebuilding the context from the steps that already ran.

Steps degrade honestly. A step whose system this deployment was never given returns unavailable, says which system and which environment variable would switch it on, and is skipped. It does not guess a plausible pod list to fill the gap.

Conditions are explicit. A step can carry a run_if or skip_if over anything an earlier step saved, and the dry run will tell you before activation if a condition references something nothing sets.

4 · Resolve

When the steps are done the evidence is classified — by a model if one is configured, by rules if not, and the artifact says which. Then the classification is checked against the evidence before anything acts on it.

Two of those transitions deserve comment. A cleared watch resolves the incident without needing a confidence score at all — the condition the alert was about has gone away, which is the complete answer. And the confidence floor for autonomous resolution is a setting, not a constant: how bold to be is a decision for your change policy.

5 · Learn

The stage most systems skip. Each execution leaves a structured record: every step with its parameters and output, the readings a watch took, the classification, the review of that classification, and what the agent did about it.

That record is what makes the second occurrence cheaper than the first — the saving most organisations assume they already get and almost none actually do, because the human version of this step is "write two sentences in the ticket at 02:55".

Above the individual incident, the same records drive pattern analysis: alerts grouped on fingerprint, with a suggested action derived from the group's own history. An alert that fires often and is almost always benign gets tune the threshold. One that recurs with no runbook gets write one. One that recurs and usually ends with a person gets investigate the cause — and each suggestion states the rule that produced it, because a suggestion you cannot audit is only an opinion.

Keep reading