Deploy, connect & operate InsightResolve

InsightResolve is three containers and a database: the API, the console, and the InsightWorker engine that executes runbook steps. Alerts arrive as webhooks; everything the agent learns is written to Postgres and shown live in the console.

Architecture at a glance

Three tiers, the same shape as every Insight product: a React console, a FastAPI service, and the InsightWorker agent engine. State lives in Postgres; the console gets live updates over SSE.

Components

Demo stack — Docker Compose

The demo brings up everything the loop needs on one machine. PagerDuty is SaaS, so a stand-in with the same three surfaces (Events API in, incident REST out, webhook v3 events) takes its place.

git clone https://github.com/verticalserve/insightresolve-api
cd insightresolve-api/demo
cp .env.example .env
docker compose up -d --build        # ~3 minutes the first time
./demo.sh errors                    # api-gateway fails 30% of requests
./demo.sh payment-latency           # payment-service slows down
./demo.sh recover                   # sources send resolved; incidents close with MTTR

Within a minute of ./demo.sh errors, Prometheus fires HighErrorRate, Alertmanager routes it to PagerDuty, PagerDuty webhooks InsightResolve, and the incident appears matched to API Gateway High Error Rate Triage with its execution queued.

The incident loop

• Receive. A webhook is normalized into one NormalizedAlert: source, external id, alert name, title, severity, service, labels, URL, start time, resolved flag.
• Deduplicate. A fingerprint of source, alert name, service and key labels (env, namespace, cluster, region, instance, pod, host, dag_id, job). A repeat inside the window folds into the open incident; a resolved event closes it and records MTTR.
• Match. Every active rule of every active runbook is scored (see matching). No match → an ai_triage execution.
• Execute. An execution pins the runbook version and creates one row per step. InsightWorker runs them; each output is saved as evidence and becomes a variable.
• Resolve. The classification step produces category, confidence and reasoning; the gate step resolves the page or escalates with a note; the RCA artifact is written.
• Learn. Effectiveness and average resolution per runbook, per-incident metrics, and detected patterns with suggested actions.

Webhooks & sources

Four public endpoints. When RESOLVE_WEBHOOK_TOKEN is set, callers present it as ?token= or X-Resolve-Webhook-Token. A connection query parameter names the registered connection the sender belongs to.

Runbook model

A runbook is normalized into rows, not stored as one JSON document. That is what lets the console reorder a step with one update, two people edit different steps at once, and a version diff say “changed threshold in step 3”.

Variables resolve in order: incident context ({{incident.labels.service}}), then runbook variables, then the outputs of earlier steps ({{error_rate}}, {{classification.category}}).

Step action types

Matching & conditions

A rule scores one point for each field it sets that matches the alert; a field it sets that does not match disqualifies it. The best score wins; ties go to rule priority, then to the runbook’s effectiveness. Test any rule from the console against a past incident or a mock payload.

{{variable}} == 'value'        {{variable}} != 'value'
{{variable}} > 5.0             {{variable}} contains 'OOMKilled'
{{variable}} in ['critical', 'high']
{{variable}} matches 'error.*timeout'

Configuration

Identity, roles & API keys

Sign-in is local or any OIDC provider (Okta, Entra ID, Google). Group-to-role rules map directory groups onto roles at every sign-in. admin holds every role; curator edits runbooks; every signed-in person reads and acts on incidents. API keys are created under Users & access, shown once, and carry scopes as roles. Every change is in the audit trail.

API

Every response is the envelope {"status": "SUCCESS" | "FAIL", "data": ..., "error_message": ...}. Authenticate with a bearer token from POST /api/auth/login or an API key. The full OpenAPI document is at /docs on your instance.

POST /api/auth/login                        {"username": "...", "password": "..."}
GET  /api/dashboard
GET  /api/incidents?status=open&severity=critical
GET  /api/incidents/{id}                    executions, steps, RCA
POST /api/incidents/{id}/retry              match again, open a fresh execution
POST /api/incidents/{id}/escalate
GET  /api/runbooks   POST /api/runbooks     PUT/DELETE /api/runbooks/{id}
POST /api/runbooks/{id}/activate  /clone    POST /api/runbooks/from-template/{tid}
GET/POST /api/runbooks/{id}/steps|rules|variables|versions
POST /api/runbooks/{id}/steps/reorder       POST /api/runbooks/{id}/rules/test
POST /api/runbooks/match                    which active runbook takes this alert
GET  /api/user/getAll  /roles  /providers  /api-keys
GET  /api/connections  /connectors