02:14, and etl_orders_daily has failed on
load_orders. Nobody is being paged about a customer. No dashboard is red. The
consequence arrives at 08:00, when the finance team opens a report built on data that is
now a day stale — and by then the window to fix it quietly has closed.
Data incidents have a different shape from service incidents, and treating them the same way is why most on-call rotas handle them badly.
What makes pipeline incidents different
| Service incident | Pipeline incident | |
|---|---|---|
| Urgency | Now. Customers are affected while you read this. | Deadline-shaped. Nothing matters until the report is due — then everything does. |
| Retry | Rarely safe or useful. | Often the correct first action, and frequently sufficient. |
| The real question | What is broken? | Is the input even there? A large share of failures are upstream data that never arrived. |
| Escalation target | The service owner. | Often the upstream owner, who does not know they broke anything. |
| Cost of waiting | Linear and immediate. | Zero, then a cliff at the SLA. |
That third row is the one that reshapes the runbook. For a service incident you start with the service. For a pipeline, starting with the pipeline is frequently a waste of the first ten minutes — the task failed because the file it expected was not in the bucket, and no amount of reading the task logs will say so as clearly as checking the source.
The runbook
run
Which task failed, on which run, and whether earlier tasks in the same run succeeded. Establishes whether this is one bad task or a run that never really started.
logs
The stack trace. Referenced as {{run.failed_task}} so the step does not need to know which task failed in advance.
freshness
The step that changes the diagnosis most often. If the upstream source is stale, the pipeline did not fail — it was starved, and the fix is somebody else's.
Conditional on the failure looking transient: run_if {{logs}} contains 'timeout'. A mutating step, named by the dry run.
Carries the logs and the freshness check, so whoever picks it up in the morning starts from the evidence rather than from “the DAG failed”.
Why this one ships as a template, not an active runbook
DAG naming is entirely local. There is no globally sensible match rule for "a pipeline
failed" the way there is for HighErrorRate. So this arrives as a template
you clone and bind to your own naming — and the readiness check will refuse to
activate it until you have added a match rule.
The decision the runbook is really making
source fresh?"} S -->|"stale"| U["Not your incident
the pipeline was starved.
Escalate upstream, name the source."] S -->|"fresh"| L{"What do the
logs say?"} L -->|"timeout, connection reset,
throttled"| T["Transient
retry once, then decide"] L -->|"schema change,
constraint violation,
null where none expected"| D["Data contract broken
a retry will fail identically.
Ticket it with the evidence."] L -->|"OOM, disk"| R["Resource
retry will likely fail too.
Needs a size change."] T --> Q{"Retry
succeeded?"} Q -->|"yes"| C["Resolved. Note it —
a pattern of retries is a
signal in itself."] Q -->|"no"| D style U fill:#f4f6ff,stroke:#a78bfa,stroke-width:2px style D fill:#fde8e8,stroke:#e57373
The left branch is the valuable one. A meaningful share of pipeline failures are not failures of the pipeline at all — the upstream extract did not land, the partner file was late, the API returned an empty page. Discovering that at 02:20 automatically, and escalating to the team that owns the source, is worth more than any remediation the runbook could perform.
The right branch matters for the opposite reason: a schema change or a constraint violation
will fail identically on retry, so retrying is pure delay. The condition on step four
(run_if {{logs}} contains 'timeout') is a deliberately narrow gate —
retry only when there is positive evidence of transience, not by default.
Deadline awareness, which most runbooks lack
The one thing this runbook does not do out of the box, and which we would add for any real data platform, is reason about the deadline.
"The DAG failed at 02:14 and the report is due at 08:00" is a completely different incident from "the DAG failed at 07:30 and the report is due at 08:00", and the evidence is identical. Two ways to add it:
- An
assignstep computing hours-to-SLA from a variable on the DAG, feeding agatethat escalates immediately when the margin is thin and tickets it for the morning when it is not. - A second match rule on the same runbook scoped to a time window, with a higher priority, that skips the retry and pages directly. Cruder, but it needs no expression logic and it is easy to explain to whoever is on call.
The anti-pattern this replaces
Most data teams handle this with an email alert that goes to a shared inbox and is read at 08:15 — fifteen minutes after the report was due, and six hours after the window in which a retry would have quietly fixed it. The value here is almost entirely in closing that gap, not in the sophistication of the diagnosis.
Adapting it to your stack
- dbt Cloud instead of Airflow: swap steps one and two for
dbt.test_results. The shape is identical — what failed, why, and is the source fresh. - Managed orchestration: Step Functions and Cloud Composer both surface
execution state through their cloud APIs;
cloudwatch.logsorgcp.logsstands in for task logs. - No dbt: replace the freshness step with a direct query —
postgres.queryformax(updated_at)on the source table is crude and answers the same question. - Always keep step three before step four. Retrying a DAG whose input never arrived is the most common wasted action in data on-call.
Next in this series
Capacity exhaustion — the alerts that announce themselves hours in advance and still manage to surprise people.