Six tools, four teams, one pager at 3am

Nobody sat down and chose to run six monitoring tools. Ask the platform lead how the estate ended up this way and you will get a history rather than a design: a decision made in 2019 by a team that has since been reorganised, a tool that arrived with an acquisition, a Prometheus that came free with Kubernetes and never left.

This matters more than it sounds. The cost of a sprawling monitoring estate is not the licence spend, which finance can see and argue about. It is paid at three in the morning by whoever is on call, in the minutes between a page arriving and that person knowing what is actually wrong — and it is invisible on every dashboard anyone looks at.

How the estate accumulates

The pattern is remarkably consistent across companies of very different sizes. It goes something like this.

The infrastructure team starts with whatever the cloud provider includes, because it is already there and already paid for: CloudWatch, or Azure Monitor, or Cloud Monitoring. It answers questions about instances and load balancers and managed databases perfectly well, and answers questions about application behaviour badly, because it was never meant to.

So the application teams buy something that does. Usually Datadog or New Relic, occasionally a self-hosted Elastic stack. This tool is good at traces and custom metrics, and it becomes the place engineers actually look during an incident — which means the cloud-native tool quietly becomes a billing artefact that still, inconveniently, sends pages.

Then the platform migrates to Kubernetes, and Prometheus arrives as part of the deal. Nobody decides to adopt Prometheus. It is simply what the Helm charts assume, what the operators expose, and what every piece of cloud-native documentation is written against. Now there are three metric stores, two of which have alerting rules nobody has audited in a year.

Somewhere in here, security requires log retention the application tool cannot economically provide, and Splunk appears. An acquisition brings a fourth monitoring tool, and the integration plan that promised to consolidate it was descoped in the second quarter. Compliance needs incidents recorded in ServiceNow. Someone builds a Grafana instance to put the important panels in one place, which is an admission that the tools have failed to do so themselves.

Every arrow into that box is a tool someone chose for good reasons. The problem is not any of them. The problem is the box.

Why consolidation projects keep failing

Every estate like this has had at least one consolidation initiative. Most of them fail, and they fail for three reasons that have nothing to do with the quality of the tools.

The migration is never finished, so the cost is never removed. Moving dashboards is easy. Moving three years of accumulated alert rules, each encoding a hard-won lesson about a specific failure mode, is not. Teams migrate the eighty per cent that is straightforward, keep the old system running for the remaining twenty, and end up paying for both indefinitely.

Each tool is genuinely better at its own job. Prometheus really is the right answer for Kubernetes. CloudWatch really is the only thing that sees inside a managed RDS instance. Standardising on one tool means being deliberately worse at several things, and the team that owns the thing you got worse at will object, correctly.

Ownership is distributed, and so is the budget. The tools are not owned by one team, which means consolidating them requires agreement from several, each of whom is being measured on something other than this.

What it actually costs

Here is the part that does not appear in any tooling review. When a page arrives, the engineer who receives it does not begin diagnosing. They begin assembling.

They open the alert to see which service and which threshold. They open the metrics tool to see whether it is still happening and how bad. They open the logs tool to see what the service is saying about itself. They open the deployment pipeline to see whether anything shipped. They open the cloud console to see whether the provider is having a bad day. They open the runbook wiki, which is usually out of date. They open Slack to ask whether anyone else is seeing it.

Seven tools, seven authentication sessions, seven mental context switches, and at the end of it the engineer knows roughly what an experienced colleague would have guessed in thirty seconds. None of that time produced a fix. All of it was lookup.

Tool	Who owns it	The question it answers	Time to answer it
Paging	SRE	What fired, and how urgent is it?	Seconds
Metrics	Application teams	Is it still happening? How bad?	1–3 minutes
Logs	Platform / security	What is the service saying?	2–5 minutes
Cloud console	Infrastructure	Is the platform healthy?	2–4 minutes
CI/CD	Engineering	Did anything change?	1–3 minutes
Wiki / runbooks	Nobody, really	What did we do last time?	3–10 minutes
ITSM	Service management	Is this recorded?	After the fact

The rows are not the problem. The sum is the problem, and the sum is paid on every single page, including the ones that turn out to be nothing.

The seam, not the tools

Once you look at it this way, the diagnosis changes. The monitoring tools are mostly fine. What is missing is the layer between the alert and the human — the thing that would do the assembling, so the person arrives at a picture rather than at a starting line.

That layer has traditionally been a person: the experienced engineer who knows that HighErrorRate on the checkout service at this time of day is almost always the payment gateway, and who checks that first. When that person is on holiday, the incident takes four times as long, which is the clearest possible evidence that the knowledge lives in a head rather than in a system.

Attempts to write it down produce wiki runbooks, and wiki runbooks decay. They decay because prose cannot be executed, and anything that cannot be executed cannot be tested, and anything that cannot be tested drifts away from reality one deploy at a time. We have written separately about why a runbook has to be data rather than prose before any of this works.

What would actually help

Not one tool. Not a migration. Something narrower: one place the alert lands, which knows how to go and ask the other six.

Concretely, that means a layer which takes a page from any source and normalises it to one shape, so a runbook can match on it regardless of which tool raised it. It means that layer holding credentials to read — not write — from the metrics store, the log store, the cloud API and the change calendar. It means the gathering happening automatically, in parallel, in the seconds between the page firing and the human opening their laptop.

And it means the result being written down in a form the next person can read, so that the knowledge stops living in one engineer's head.

This is deliberately a smaller claim than the consolidation pitch. It does not require anyone to stop using Datadog. It does not require the Prometheus rules to be rewritten. It sits beside the estate you have and does the assembling that a person currently does by hand.

Two ways to read this

Keep reading