Every support organisation above a certain size is tiered, and almost nobody can explain the tiers without reaching for a diagram. L1 triages. L2 investigates. L3 fixes. That description is tidy, widely repeated, and does not survive contact with a real incident queue — because the interesting question is not what each tier does, but what happens in the gaps between them.

What the tiers are actually for

Tiering exists for one reason: expertise is expensive and unevenly distributed. A person who can diagnose a distributed deadlock in a sharded database is scarce and costly, and if that person spends their afternoon confirming that a disk-space alert on a development box can be ignored, the organisation has wasted its scarcest resource on its least valuable work.

So you build a filter. The first tier is broad, cheaper, available around the clock, and its job is to make sure that only the incidents genuinely needing scarce expertise ever reach it. The economics are sound, and they are why the shape has survived as long as it has.

TierWhat it really doesTypical coverageFully loaded cost/hour
L1Acknowledge, classify, apply known procedures, decide whether to escalate24×7, often outsourced or follow-the-sun$25–60
L2Diagnose beyond the runbook, correlate across systems, apply judgementBusiness hours plus on-call rota$70–120
L3Engineering: read the code, fix the cause, ship the changeBusiness hours, escalation only$120–200
VendorThe thing you do not ownContractualSupport contract

Treat those hourly figures as placeholders, not data — they are order-of-magnitude ranges for fully loaded cost, and yours will differ enough that you should substitute them before drawing any conclusion. What the table is actually for is the ratio: L3 time costs several times what L1 time costs. That is the part that holds regardless of which numbers you put in, and it is the reason the tier structure exists at all.

The escalation, not the investigation

Here is the thing the tidy diagram hides. When you time a real incident from page to resolution, the largest single block of elapsed time is usually not spent investigating. It is spent waiting, and specifically waiting across a tier boundary.

sequenceDiagram participant A as Alert participant L1 as L1 (24×7) participant L2 as L2 (on call) participant L3 as L3 (engineering) A->>L1: page fires Note over L1: 4 min — acknowledge, open six tools Note over L1: 9 min — follow the wiki runbook L1->>L2: escalate "error rate high, runbook didn't help" Note over L1,L2: 11 min — waiting for L2 to wake up and read Note over L2: 6 min — re-gathers the same evidence Note over L2: 8 min — finds the deploy correlation L2->>L3: escalate "looks like release v2.4.1" Note over L2,L3: 14 min — waiting, and re-explaining Note over L3: 5 min — confirms, rolls back L3-->>A: resolved

Add that up. Fifty-seven minutes of elapsed time, of which twenty-five — nearly half — is handoff latency. And the handoff is not just waiting. Each one carries two other costs that are easy to miss.

The evidence is gathered twice. L1 opened the metrics dashboard, looked at the error rate, checked the logs. Then L1 wrote a three-line summary in the ticket. Then L2 woke up, read the three lines, did not trust them — reasonably, because three lines cannot carry what a dashboard shows — and opened the same three tools again. The work was done twice because the first time it was not captured in a form the second person could use.

The context degrades at every hop. What L1 saw becomes a sentence. What L2 concluded becomes another sentence. By the time it reaches L3 the original signal has been through two rounds of lossy summarisation, and L3's first question is almost always a request for something that was on L1's screen forty minutes earlier.

The metric that hides this

MTTR measured per tier looks healthy. L1's mean handling time is good, L2's is good, L3's is good. The handoff belongs to nobody, so it is measured by nobody, and it is the largest component of the number the business actually feels.

Why L1 escalates more than it should

Ask any L2 engineer and they will tell you a meaningful share of what reaches them should have stopped at L1. Ask L1 and they will tell you, correctly, that they escalated because they could not be sure. Both are right, and the reason is structural.

L1's runbook says: check whether the error rate is still elevated; if it has recovered, note it and close. To follow that instruction the L1 engineer needs access to the metrics tool, permission to read the right dashboard, knowledge of which query represents "the error rate" for this particular service, and the judgement to decide what "recovered" means for a number that is bouncing around. In practice at least one of those four is missing, so the safe action is to escalate — and escalating is always safe for the individual, whatever it costs the organisation.

This is worth stating plainly because it reframes the problem. L1 does not over-escalate because L1 is unskilled. L1 over-escalates because the information needed to stop is not available at L1. Fix the availability of the information and the escalation rate falls without anyone being retrained.

The three genuinely different kinds of work

Tiering by seniority obscures something more useful: incidents divide into three categories that have almost nothing in common, and each wants different handling.

Known and procedural. This has happened before, there is a correct sequence of checks, and following it produces the right answer. Somewhere between a third and half of a mature estate's alert volume is this. It needs no judgement at all — only reliable execution, which is precisely what humans are worst at doing consistently at 3am.

Known-shaped but novel. The symptom is familiar, the cause is not. Evidence gathering is mechanical; the interpretation is not. This is the category where an automated layer should gather everything and then hand over, rather than guessing.

Genuinely new. Nobody has seen this. Automation's only useful contribution is to have already collected everything relevant, so the expert starts from a complete picture instead of an empty one.

Notice that in all three categories the gathering is mechanical. The difference between the categories is entirely in what happens afterwards. That observation is the whole basis for automating incident response sensibly, and we develop it in where AI actually helps.

What changes when the gathering is already done

Return to the sequence diagram and remove one thing: the assumption that a human must gather the evidence before anyone can think about it.

sequenceDiagram participant A as Alert participant Ag as Agent participant L1 as L1 participant L2 as L2 A->>Ag: page fires Note over Ag: 40 s — runs the runbook: metrics,
pods, releases, logs, change calendar Ag->>L1: incident with evidence and a
classification, confidence stated Note over L1: 3 min — reads the record, agrees,
closes the benign ones L1->>L2: escalate the rest, with everything attached Note over L2: 7 min — starts from the evidence,
does not re-gather it L2-->>A: resolved

Two things changed. The first hop now carries a complete record rather than three lines, so L2 does not repeat L1's work. And the incidents that were only ever going to be closed at L1 now can be closed at L1, because the information needed to be sure is sitting on the incident rather than behind a tool L1 cannot reach.

There is a second-order effect worth naming. When the evidence is attached to every incident automatically, the difference in outcome between your best on-call engineer and your newest one narrows considerably — not because the newest engineer got better, but because the starting position stopped depending on who happened to be holding the pager.

What this does not do

It does not remove a tier. Any vendor promising to eliminate L1 is selling you a headcount argument you will have to defend later, and the first genuinely novel outage will make you regret it. What it removes is the rework — the same evidence gathered two or three times by people of increasing cost — and the waiting that surrounds it.

Where to look in your own data

If you want to know whether any of this applies to you, three numbers from your existing ticket system will tell you, and none of them requires new instrumentation.

  • Escalation rate from L1. Above about thirty per cent and the filter is not filtering; the tier is acting as a switchboard.
  • Time from escalation to first L2 action, separated from L2's handling time. Most ticket systems can produce this, and most teams have never looked at it.
  • Proportion of incidents closed with no change made. If a third of your pages resolve themselves, you are paying full escalation cost for alerts that only needed someone to wait and watch.

That last number is the one teams are most often surprised by, and it is the subject of not every alert needs an action.

Two ways to read this

If you run the support org

Tiering is not the problem and you should resist anyone who says it is. What is worth attacking is the handoff: the waiting, and the evidence gathered twice. Both are measurable from the ticket system you already have, and neither requires you to change the shape of your team.

Be wary of headcount-reduction framing in vendor conversations. The defensible business case is capacity released — the same team absorbing more volume without the rota degrading — because that survives the first outage the automation does not handle. A headcount promise does not.

If you are evaluating automation

The interesting question is what the system does at the boundary of its competence. An agent that handles the procedural third and hands over cleanly is valuable. One that attempts the novel third and is confidently wrong will cost you the trust of the on-call team permanently, and you will not get it back in that org.

So the thing to test is not the happy path. It is what lands on the incident when the agent cannot reach a system, or cannot support its own conclusion.

Next in this series

Anatomy of an incident, minute by minute — a single high-error-rate page, timed end to end, and the arithmetic of what those minutes cost across a year.

Keep reading