SRE Practices

Why MTTR Is the Wrong Metric for Modern SRE Teams

8 min read By Marcus Chen
Abstract time-series visualization representing metrics and measurements in incident response

MTTR is the metric that engineering leaders love because it fits on a dashboard. It's a single number. It trends up or down. It looks like progress. The problem is it collapses two fundamentally different activities into one number: the time it takes to understand what's wrong, and the time it takes to fix it once you do.

We built Devtract because we kept watching teams with sub-10-minute MTTR for some incidents and multi-hour MTTR for others, and the split had nothing to do with fix complexity. The short incidents were ones where the on-call immediately understood the cause. The long ones were ones where they didn't. If you're tracking MTTR and it's not improving, you're probably not asking the right question about why.

Decomposing MTTR Into What Actually Varies

A more useful decomposition breaks incident time into three segments: alert-to-acknowledge, acknowledge-to-diagnosis, and diagnosis-to-resolution.

Alert-to-acknowledge is mostly a paging policy and staffing problem. Tuning it is usually about on-call rotation design, escalation policies, and ensuring you're not alerting outside business hours on things that can wait. Most teams have this reasonably under control within their first year of operating an on-call rotation.

Diagnosis-to-resolution is the fix itself. It depends on how quickly you can deploy a fix, rollback a bad deploy, update a config, or restart a service. This is a deployment pipeline and access-control problem. Again, most mature teams have invested here, and the improvement ceiling is real: you can't restart a Kubernetes pod faster than the Kubernetes API will let you.

The middle segment, acknowledge-to-diagnosis, is where almost all the variance lives for teams with well-tuned paging and fast deployments. We've analyzed incident timelines across our early user base and the diagnostic gap, the time between an on-call engineer acknowledging a page and forming a clear hypothesis about what's wrong, ranges from under 2 minutes to over 90 minutes for incidents of similar technical complexity. That variance is almost entirely explained by how much relevant context the engineer had immediately available.

Why Averaging the Diagnostic Gap Hides It

MTTR as a mean has a specific failure mode: it averages across incidents with different root causes. A fast-resolving incident (known problem, clear log signature, 8 minutes total) and a slow-resolving one (novel failure mode, ambiguous telemetry, 110 minutes total) average to 59 minutes. The 59-minute average tells you nothing useful. It doesn't tell you that you had one incident where the on-call was flying blind for 90 minutes.

A more diagnostic approach tracks MTTR distribution rather than mean, and specifically tracks the tail: the p95 or p99 of incident duration. Teams whose p95 incident duration is improving are actually getting better at handling the hard cases. Teams whose mean is improving but whose p95 is flat have probably just gotten more consistent at the easy cases while the hard cases remain hard.

The practical question is: what's causing the hard cases to be hard? Almost always it comes back to the diagnostic gap. The on-call spends 45 minutes grepping logs across five dashboards trying to figure out which of the four services that changed in the last two hours is the one that caused the problem.

The Diagnostic Gap as a First-Class Metric

We'd push teams to track time-to-hypothesis as a distinct metric: the duration from alert acknowledgement to the first written hypothesis about root cause, logged in the incident channel or on-call tool. This is the moment the on-call engineer types something like "I think this is the auth service DB connection pool exhaustion from the deploy at 14:30" in the Slack war room thread.

This metric is harder to instrument than MTTR because it requires humans to log the moment of hypothesis formation. But it's enormously informative. When we instrumented this manually for a logistics platform we worked with during Devtract's development, the data was stark: their median time-to-hypothesis was 34 minutes. Their median diagnosis-to-resolution, once they had a hypothesis, was 7 minutes. Their total MTTR was 41 minutes. Improving their deployment pipeline to cut resolution time from 7 to 4 minutes would reduce MTTR by 7%. Cutting time-to-hypothesis from 34 to 5 minutes would reduce MTTR by 70%.

The value is entirely in the diagnostic gap. But if you're only measuring MTTR, you have no way to see this.

What Causes a Long Diagnostic Gap

In our experience, the diagnostic gap is long for one of three reasons, and they have different fixes.

The first is context poverty: the on-call engineer doesn't have the relevant signals in front of them when they start investigating. They have an alert that says "error rate elevated on checkout service" but they need to know which dependency is the proximate cause, whether there was a recent deploy, and what the corresponding log pattern looks like. If they have to hunt for this across four different tools, the diagnostic gap accumulates from context-gathering time.

The second is signal noise: there's too much happening and the relevant signals are buried in irrelevant ones. High-traffic systems often generate thousands of log lines per second during an incident. Finding the 12 lines that actually explain the problem requires pattern recognition that takes time even for experienced engineers.

The third is unfamiliarity: the on-call engineer doesn't own the service they're investigating. Modern SRE rotations often include engineers who are on-call for services they didn't build and may rarely touch. Their diagnostic gap is longer because they lack the mental model of how the service's failure modes connect to its telemetry.

The first two are tooling problems. The third is a knowledge transfer problem. They have different solutions, but they share a common property: they're not visible if you're only looking at MTTR.

A Counterpoint: MTTR Isn't Worthless

We want to be clear that we're not saying MTTR should be abandoned as a metric. It's a reasonable lagging indicator of overall incident response health. If your MTTR is trending upward over several quarters, something systematic is getting worse. If it's stable while your system complexity is growing, you're probably improving relative to that complexity.

The argument is against MTTR as the only metric, or as the primary metric you optimize against. When you optimize directly for MTTR, you tend to optimize for the parts of it that are easiest to move: deployment speed, escalation path efficiency, playbook coverage. These are real improvements, but they address diagnosis-to-resolution, not acknowledge-to-diagnosis. The diagnostic gap gets ignored because it's harder to see and harder to instrument.

The better use of MTTR is as an outcome check. Measure the diagnostic gap directly. Implement tooling to reduce it. Then check whether MTTR improves as a result. If it does, you've validated that the diagnostic gap was the binding constraint.

Making the Diagnostic Gap Visible in Practice

The simplest starting point is manual annotation. Ask on-call engineers to log two timestamps in their incident thread: when they acknowledged the page, and when they had a working hypothesis. Export these from your incident management tool at the end of each month and compute the distribution. This takes 30 minutes to set up and will immediately show you where your time is going.

The more sophisticated approach is what we built into Devtract: correlate telemetry sources in real time so the on-call engineer receives a ranked hypothesis rather than raw signals. The goal is to deliver context-to-hypothesis time down to seconds rather than minutes. When you can show an engineer a view that says "most likely cause: auth-service database connection pool saturation, correlated with the deploy at 14:31, see log pattern below" within 30 seconds of the alert firing, the diagnostic gap collapses.

Improving MTTR is a legitimate engineering goal. But the path to it runs through the diagnostic gap, not around it.