Disk at 85%. Memory climbing steadily for six hours. Connection pool at eighty per cent of its limit. These alerts arrive with more warning than any other class, and they still routinely turn into outages — because the warning is delivered as a threshold crossing rather than as a deadline.
"Disk is at 85%" is not actionable. "Disk fills in four hours, which is 02:00" is.
The only question that matters
Every capacity alert reduces to one question, and almost no alert rule answers it: when does this become a problem, and is that inside working hours?
| Projection | What it means | Right response now |
|---|---|---|
| Never — it is flat | 85% is just where this volume lives. The threshold is wrong. | Close it. Fix the alert rule, not the disk. |
| Days away | Real, but not tonight. | Ticket it for business hours. Nobody wakes up. |
| Hours — inside working hours | Someone will be awake and can size it properly. | Ticket it, flagged, with the deadline stated. |
| Hours — lands overnight | It will fill while nobody is watching. | Act or page now, while there is still a choice. |
| Minutes | Something changed abruptly — a log loop, a runaway job. | Page, and look for the cause rather than adding space. |
Five very different responses, one alert, and the distinguishing information is not in the alert. It is in the slope, which nobody computes at 3am.
The runbook
used
The number in the alert, re-read. Capacity alerts often fire on a spike that has already passed.
trend
A range query, not an instant one. The slope is the whole diagnosis; a single reading cannot distinguish “flat at 85%” from “climbing fast”.
top_consumers
Largest directories, biggest tables, heaviest connections — whatever is appropriate to the resource. Turns “add space” into “this log is not rotating”.
changes
A step change in a growth curve usually has a cause with a change number. Optional, but it converts a capacity ticket into a root cause.
urgency
Given the slope and the current value, when does this exhaust — and is that inside working hours? The output drives the gate below.
Escalates now when exhaustion lands overnight; otherwise files a ticket with the projection stated and lets everyone sleep.
This one is not in the seeded set
Unlike the other runbooks in this series, we do not ship this one active in the demo stack — the projection step depends on which resource you are watching and what counts as "consuming it", and a generic version would be a worked example pretending to be a product feature. The shape is worth copying; the specifics are yours.
Projection, and where it goes wrong
The same trend machinery that powers the lag watch applies here, pointed the other way: instead of "when does this fall below a threshold", it is "when does this rise above one". Same least-squares fit, same caution about naming a direction when the movement is not meaningful.
Three ways a naive projection misleads, all of which we have watched happen:
a nightly job truncates it"] --> A2["A linear fit across 6h
projects exhaustion tonight"] A2 --> A3["Wrong. It resets at 01:00,
as it has every night for a year."] end subgraph B["Step change"] B1["Flat for weeks, then a jump
when a new tenant onboarded"] --> B2["A fit across the step
projects a runaway"] B2 --> B3["Wrong. It is flat again
at the new level."] end subgraph C["Accelerating"] C1["Growth rate itself increasing
— a retry storm, a log loop"] --> C2["A linear fit projects
hours"] C2 --> C3["Dangerously optimistic.
The real answer is minutes."] end style A3 fill:#fff4e2,stroke:#f4a03a style B3 fill:#fff4e2,stroke:#f4a03a style C3 fill:#fde8e8,stroke:#e57373,stroke-width:2px
The third is the one to design against, because its failure mode is the harmful direction. A practical guard: compare the slope over the last hour against the slope over the last six. If the recent slope is materially steeper, do not project linearly — say "growth is accelerating" and escalate. Being vague and alarmed is the right answer there; being precise and wrong is not.
State the projection as a window, never a timestamp
"Exhausts in roughly 3–6 hours, so between 01:00 and 04:00" is honest and actionable. "Exhausts at 02:47" is false precision, and the first time it is wrong by an hour the engineer stops believing the number entirely — which costs you every future projection too.
Why "what is consuming it" earns its place
Step three is the difference between a runbook that helps and one that files tickets. Without it the outcome is "disk at 85%, please add space". With it the outcome is often "the application log has not rotated since the last deploy and is 40GB" — which is a five-minute fix rather than a provisioning request, and it stops the alert recurring.
It is also the step that makes the difference visible on the Patterns page. A capacity alert that fires monthly on the same volume and is always resolved by adding space is telling you something the individual incidents cannot: the growth is structural and the sizing is wrong.
The honest recommendation
Of all the runbooks in this series, capacity is the one where the best outcome is often fewer alerts rather than better handling.
- Alert on the projection, not the level. If your monitoring can express
predict_linearor an equivalent, alerting on "will exhaust within four hours" removes most of what this runbook is compensating for. - Route by projected deadline. A capacity alert that will not bite until Tuesday should not page at all. It should open a ticket.
- Treat repeat capacity alerts as a sizing conversation. Three in a quarter on the same resource is not an incident-response problem.
That is a slightly awkward thing for a product in this category to say, and we would rather say it: automating the response to an alert that should not exist is a worse outcome than deleting the alert.
Next
That completes the use cases. Next: what this looks like on each cloud, starting with AWS — CloudWatch alarms, EventBridge events, and reading ECS and load-balancer health back.