The page fires. The runbook runs. The incident closes — with the root cause written down

The agent gathers, concludes — then checks itself

• 1PagerDuty fires HighErrorRate on api-gateway, production, critical. Normalized, fingerprinted, matched to API Gateway High Error Rate Triage.
• 2pagerduty.acknowledge — the page stops escalating while the agent works.
• 3prometheus.instant — the 5xx rate, confirmed still above threshold.
• 4k8s.pods — skipped. Kubernetes is not configured in this deployment, so the step says so rather than guessing.
• 5prometheus.instant ×2 — recent deployments, and upstream dependency health.
• 6http.request — the service’s own recent error lines, a step the runbook only runs while the rate is still elevated. It was, so it ran.
• 7ai.classify — bad_deploy at 0.82: a deployment inside the window, and errors naming a dependency the new version talks to.
• 8ai.verify — checks that conclusion against the evidence alone. “A deployment happened inside the window” is established. “The errors are attributable to it” is assumed — the evidence shows timing, not causation. Confidence lowered to 0.75.
• 9Below the floor for autonomous resolution, so it escalates — with the evidence, the claims, and the gap: “a rollback that clears the errors would settle it.”

Steps, variables and conditions — not prose

Every step is an action with typed parameters. Outputs become variables the next step can use. Conditions decide what runs.

# variables
service_name = {{incident.labels.service}}
threshold    = {{incident.labels.threshold}}   # default 5.0

# steps
2. prometheus.instant  query: rate(http_requests_total{service='{{service_name}}',code=~'5..'}[5m])
                       save_as: error_rate
4. postgres.query      sql: SELECT version, deployed_at FROM deployments WHERE service='{{service_name}}' ...
                       save_as: recent_deploys
6. k8s.logs            run_if: {{error_rate}} > {{threshold}}
                       save_as: error_logs
7. ai.classify         categories: [bad_deploy, upstream_dependency_failure, capacity, false_positive]
                       context_vars: [error_rate, pod_status, recent_deploys, error_logs]
9. gate                {{classification.confidence}} > 0.85 AND category == 'false_positive'
                       true:  pagerduty.resolve   false: pagerduty.add_note + escalate

It knows when to do nothing

Consumer lag, queue depth, replication delay — a fifth to a third of most queues resolve with no change made. The monitor step samples the value, judges the direction from the series’ peak, and parks the execution between looks rather than holding a worker. If it clears, the incident closes saying no action was needed. If it stops recovering, that is when a person is paged.

It checks its own conclusions

Before acting on a classification the agent verifies it against the evidence alone, claim by claim. A link that is assumed rather than shown caps confidence at 0.75; one nothing supports caps it at 0.4. The incident shows this as what this rests on — each claim with the step that established it, and the gaps a person would still check.

You set the blast radius

Steps that change something outside the platform are marked and surfaced before activation. Anything consequential sits behind an approval that suspends the run until a named person answers — and if nobody does, it times out and the run continues without it rather than waiting for ever. The confidence floor for autonomous resolution is a setting you own, not a vendor default.

You can test it before you trust it

Test run walks a runbook against a sample alert built from its own match rule and reports what each step would do — an unknown action, a {{reference}} nothing fills, a system this deployment was never given. Nothing is executed, so a rehearsal can never page anyone. The summary names every step that would change something.

The executive whitepaper

Twelve pages for whoever owns the run-versus-change ratio. What is mechanical in incident response and what is not, the operating model that follows, and the sequence for adopting it.

Worked use cases

One alert, one runbook, start to finish — consumer lag, error rate after a deploy, latency, service down, a pipeline that failed at 2am, and capacity. With the real runbooks and the numbers the agent recorded.

On your cloud

What the wiring looks like on AWS, Azure, Google Cloud and OCI — including the parts the vendor documentation does not prepare you for, and where ServiceNow fits underneath all four.