"Can it restart things?" is the first question every platform lead asks, and the honest answer is: only if you say so, only for the steps you marked, and only after a named person answers. This post is about how that boundary is drawn, because a boundary you cannot inspect is not a control — it is a promise.
Reading is safe. Writing is not.
The single most useful division in this whole design is between steps that observe and steps that change something. It sounds obvious and it is routinely blurred, usually by systems that describe everything as an "action" and leave you to work out which ones have consequences.
Every step type in InsightResolve is classified, and the classification is visible before you activate anything.
| Class | Examples | Default posture |
|---|---|---|
| Observes | prometheus.instant, k8s.pods, cloudwatch.logs, azure.logs, servicenow.recent_changes, monitor | Runs freely. Read-only credentials are sufficient. |
| Reasons | ai.classify, ai.verify, ai.summarize, gate | Runs freely. Touches nothing outside the platform. |
| Changes something | pagerduty.resolve, opsgenie.close, alertmanager.silence, jira.create, slack.post, bash.exec, airflow.trigger, datadog.mute_monitor | Marked. Surfaced before activation. Gateable behind an approval. |
Where the credentials come from
Never from a step's parameters. A step names what to query; who we are is the deployment's business — environment, task role, instance role. This matters more than it sounds: if a step could carry credentials, then anyone who can edit a runbook could point the agent at another account, and runbook editing is a much lower bar than infrastructure access.
Three controls, in increasing order of strictness
walks every step against a sample alert
and reports what each would do"] C1 --> C2["2 · Mutating-step marking
the rehearsal names every step that
changes something outside the platform"] C2 --> C3["3 · Approval gates
the execution suspends until a
named person answers"] C3 --> A["Activated, with a stated blast radius"] C1 -.->|"never executes
anything"| N1(["a rehearsal cannot
page anyone"]) C3 -.->|"nobody answers"| N2(["times out and
escalates"]) style C3 fill:#fff4e2,stroke:#f4a03a,stroke-width:2px style A fill:#e8f8f4,stroke:#22c1a4,stroke-width:2px
The dry run is the cheapest and the one people skip. It builds a sample alert
from the runbook's own match rule, interpolates every variable, and reports a verdict per
step — without executing a single one. It catches the unglamorous failures: an action
type that does not exist, a {{reference}} nothing fills, a step whose system
this deployment was never given, a condition on a variable nothing sets.
Mutating-step marking turns the rehearsal into a blast-radius statement. The summary line is not a promise from a vendor; it is derived from the runbook: "All 10 steps would run. 1 of them changes something outside InsightResolve: Acknowledge incident."
Approval gates are the hard boundary. A human.approve step
suspends the whole execution — not a thread, the execution — and waits.
What an approval actually does
no worker held · survives a restart Inc->>Eng: "Waiting for you: restart orders-api in prod?" Eng->>Inc: Approve, with a note Inc->>Ag: resume_at = now Ag->>Ag: picks it up within seconds, runs step 6 Note over Ag: the answer, who gave it and when
are on the step for ever
Three properties of that flow are deliberate and worth checking in any system you evaluate.
- It suspends rather than blocks. A held thread dies with the process; a parked execution is a database row. Deploy in the middle of an approval and the approval is still there afterwards.
- Rejection is a real outcome, not a cancel. Rejecting stops the runbook there and escalates, with the reason recorded. The record shows a person actively declined, which is different from nobody having looked.
- Silence has a defined behaviour. If nobody answers within the timeout, the agent escalates by itself. The one outcome never allowed is an incident sitting in waiting because everyone assumed someone else had it.
What we do not offer, deliberately
There is no "autonomous mode" switch that lets the agent do anything it judges necessary. Not because it could not be built, but because the resulting blast radius could not be stated in a sentence — and a control you cannot state in a sentence will not pass a change-advisory board, so it would be a feature that demos well and never gets enabled.
Drawing the boundary for your own estate
This is a policy decision, not a technical one, and it belongs with whoever owns change management. A sequence that works:
- Start with everything read-only. No mutating steps at all. Let it gather and conclude for a few weeks and compare against what on-call actually did.
- Add the reversible writes first — acknowledging a page, adding a note, opening a ticket. If these go wrong the cost is noise, not an outage.
- Then the benign auto-resolutions, gated by a confidence floor you set. Start the floor high. Move it only on evidence from your own incidents.
- Then the safe remediations, behind approval — a restart, a scale, a cache clear, each with a named approver.
- Some things never move. Failing over a database, rolling back a release. Not because the agent could not, but because the accountability cannot be delegated to one.
The question to ask at each step
Not "is the agent usually right about this?" but "if it is wrong here, how long does it take us to notice, and what does it cost to undo?" Reversibility and detection time are the two variables that should set the boundary. Accuracy is the one that tempts people to move it too early.
Next
That completes the design series. Next: the use cases, one alert at a time, starting with Kafka consumer lag — the alert that should usually be waited out.