A scrape target disappears. Prometheus has not heard from payment-service for thirty seconds and the page fires. This is the alert where minutes cost the most — and, awkwardly, the one where the obvious remediation is the thing an agent should least be trusted to do alone.
Three very different incidents, one alert
Before anything else: up == 0 does not mean the service is down. It means
Prometheus could not reach it, which is a weaker claim than it looks.
| What is true | How it looks | Urgency |
|---|---|---|
| The service is genuinely down | No pods ready, or all crash-looping | Highest. Customers are affected now. |
| The service is fine, monitoring cannot reach it | Pods ready, traffic still being served, error rate normal | Low. It is a monitoring incident wearing a service-down costume. |
| Mid-deployment | Old pods terminating, new ones not ready yet | None, usually. It resolves in ninety seconds. |
The second and third rows are common enough that paging a human immediately on
up == 0 is how teams train themselves to ignore the alert — which is the
worst possible outcome for the row that matters.
The runbook
pods
Ready replicas, restart counts, and whether anything is in CrashLoopBackOff. Separates row one from rows two and three immediately.
events
OOMKilled, FailedScheduling, image pull errors, or a rollout in progress. This is where the cause usually names itself.
restart_ok
Weighs pods and events against one question: would a restart plausibly recover this without losing anything? A stateful service mid-write is a different answer from a stateless one that OOMed.
The execution parks here. run_if {{restart_ok}} == 'yes' — the agent only asks when it has a reason to.
Runs only after a named person approves. kubectl -n {{namespace}} rollout restart deploy/{{service_name}}
Why the approval gate is here and not elsewhere
This is the runbook where the argument for full autonomy is strongest — the service is down, every minute counts, and a restart fixes it most of the time. We still gate it, and the reasoning is worth setting out because it is not squeamishness.
without asking?"] --> A{"How long before
we notice it was wrong?"} A -->|"immediately — it either
comes back or does not"| B{"What does being
wrong cost?"} B -->|"a few more seconds
of downtime"| C["Arguably safe"] B -->|"an in-flight write lost,
a queue re-consumed,
a stuck lock never diagnosed"| D["Gate it"] A -->|"not until much later"| D style D fill:#fde8e8,stroke:#e57373,stroke-width:2px
The deciding factor is the third branch. A restart does not merely fail to fix the wrong cases — it destroys the evidence. The pod that was about to be inspected, the thread dump nobody took, the stuck connection that would have named the deadlock: all gone, and the incident recurs in two hours with less to go on than the first time.
The real cost of an automatic restart
It is not the risk that the restart breaks something. It is that a restart converts a diagnosable incident into a recurring mystery. Teams that automate restarts on this alert frequently end up with a service that "just needs restarting every few days" and nobody who can say why.
What the approval looks like in practice
The point of the gate is not to slow things down. It is to put the decision in front of a person who is already holding a complete picture, so answering takes seconds rather than twenty minutes.
no deploy in the window Eng->>IR: Approve — "memory limit, restart then raise it" IR->>IR: resumes within seconds, runs the restart Note over IR: the answer, who gave it and the
note are on the step permanently
Compare that with the unautomated version: the engineer wakes, connects, finds the namespace,
runs kubectl get pods, runs kubectl describe, reads the events,
then decides. The decision is the same. The twelve minutes in front of it are gone.
And if nobody answers, the timeout escalates. The one outcome never permitted is an incident sitting in waiting because everyone assumed someone else had it.
The step most teams should add
The seeded runbook does not include it because the demo stack has no traffic source, but for a real estate it is the most valuable step in this runbook: check whether the service is actually serving traffic.
A prometheus.instant on request rate, or an elb.target_health against
the load balancer, answers row two of the table at the top directly. If the pods are ready,
the targets are healthy and requests are being served at the normal rate, then Prometheus has
a scrape problem and nobody needs to be woken up. That single step converts a meaningful
share of these pages from an escalation into a note.
The ordering that matters
Put the traffic check before the restart decision, not after. An agent that has already decided a restart is warranted and then discovers traffic is flowing has wasted the reasoning; one that checks first often never asks the question.
Adapting it beyond Kubernetes
- ECS:
ecs.servicesgives desired versus running counts, rollout state and the last five service events — the same information as pods plus events, in one step. - Behind a load balancer:
elb.target_healthis often a better first signal than the scrape, because it reflects what the traffic path thinks rather than what the monitoring path thinks. - Cloud-managed services: add
cloud.healthearly. "Is it us or the provider?" is a one-step question and occasionally the entire answer. - Anywhere: keep the approval gate on the remediation. The platform changes; the argument for a person owning the restart does not.
Next in this series
A data pipeline failed at 2am — a different shape of incident entirely: nobody is being paged about a customer, but the morning report is at risk.