We analyzed 200 incident timelines collected during Devtract's early development to answer a specific question: what information did the on-call engineer need in the first 90 seconds to form a useful hypothesis about what was wrong? We annotated each incident with the first coherent hypothesis the engineer typed in the incident channel, then worked backward to identify which context pieces were present before the hypothesis was formed, which had to be gathered during investigation, and which turned out to be irrelevant.
The findings shaped how we think about incident context delivery. This post is the writeup of what we found.
The 5am Cognitive Problem
A 5am page arrives in a specific cognitive state. The engineer is not at full alertness. Sleep inertia is real and measurable: in the first 15-30 minutes after waking, cognitive performance on complex tasks runs 20-40% below fully-alert baseline. The first few minutes of an incident response happen at reduced cognitive capacity.
This matters for tool design because it means context that requires active synthesis is more expensive at 5am than during business hours. Asking a groggy engineer to open four dashboards, correlate what they see, and form a hypothesis is a harder task than it would be at 10am. The cognitive budget for the first 90 seconds is limited. Good incident context design works with that constraint, not against it.
The best pages we saw in our analysis shared a structural property: they reduced the number of things the engineer needed to hold in working memory simultaneously. The hypothesis was almost pre-formed in the alert context. The engineer's job was verification, not construction.
What the 90-Second Context Window Needs
From the 200 incident timelines, we identified five context elements that were present in the fastest-hypothesis incidents (under 5 minutes to first hypothesis) and absent or incomplete in the slowest ones (over 30 minutes).
1. The Affected Service and Its User-Facing Impact
This sounds obvious, but many alert messages are metric-level rather than service-level. "Error rate exceeded 5% on checkout-api for 3 minutes" is better than "http_request_errors{service='checkout-api', status='5xx'} > threshold." The first tells you immediately what's broken and why it matters to users. The second requires the engineer to parse a PromQL expression while half asleep.
In 31% of the slow-hypothesis incidents in our analysis, the initial alert message required the engineer to look up which service the metric belonged to before they could orient themselves. That lookup adds 2-5 minutes to the cognitive orientation phase.
2. A Change Signal in the Recent Window
In 67% of the incidents we analyzed, the incident was caused by a change: a deploy, a config update, a database migration, a dependency version update. In the fast-hypothesis incidents, the recent change history was surfaced in the alert context. The engineer could see immediately that a deploy happened 25 minutes ago on the auth service and make the connection to the current error pattern.
In the slow-hypothesis incidents, the on-call had to remember to check the deploy log, find the relevant repository, and search through the deploy history manually. This is a retrievable fact that requires no human judgment to surface. It should be automatic.
3. The Specific Log Signature
The alert fires on a metric. The metric tells you something is elevated but not the specific mechanism. The log signature gives you the mechanism: a specific exception class, a specific error message, a specific timeout type. Having the most frequent new log error pattern (new meaning: appearing or significantly increasing since incident onset) alongside the metric alert eliminates the "what specifically is failing" question from the first 90 seconds.
We found that when this log context was present in the initial incident notification, time-to-hypothesis dropped by roughly 60% compared to incidents where the engineer had to open a log query UI to find it. The absolute numbers in our sample ranged from a 12-minute reduction at the median to over 40 minutes at the p75.
4. The Dependency Chain for the Alerting Service
One of the most common diagnostic detours we saw was the engineer assuming the alerting service was the source of the problem, spending 15-20 minutes investigating it, finding nothing obviously wrong, and only then looking upstream at its dependencies. A simple display of "checkout-api depends on: payment-service, auth-service, product-catalog-service" shifts the mental model from single-service debugging to call-chain debugging from the start.
This is information that doesn't change often and can be precomputed from service mesh telemetry, OpenTelemetry span parent relationships, or manual service dependency documentation. It's not complex to surface. It just needs to be surfaced.
5. The Current Status of Dependencies
Knowing the dependency chain is useful. Knowing that one of those dependencies is also showing elevated error rates or latency right now is more useful, because it immediately narrows the candidate causes. If checkout-api depends on three services and two of them are healthy while payment-service has elevated p99 latency, the engineer's prior for "payment-service is involved" should be high without needing to verify it manually.
In our analysis, engineers who had dependency health status surfaced in the initial context form their first hypothesis in an average of 3.8 minutes. Engineers who had to check dependency health manually take an average of 18.2 minutes. This is the largest single context element by impact on hypothesis formation time.
What the Context Does Not Need
Just as important as what to include is what to exclude. Alert notifications that include too much information are nearly as problematic as those that include too little. The on-call engineer at 5am cannot process a 30-item list of potentially relevant signals. Cognitive overload at reduced alertness leads to fixation on irrelevant signals or paralysis.
In the incident timelines we analyzed, alerts that included the full Prometheus query expression, the full alert history for the past 30 days, the complete list of active alerts across all services, and a link to the full Grafana dashboard were slower to hypothesis than alerts that included the 5 elements above and nothing else. More information is not better information when the recipient is working under time pressure and reduced cognitive capacity.
The design principle is minimum viable context: the smallest set of information that allows a competent on-call engineer to form a first hypothesis without additional lookups. Everything beyond that threshold should be accessible on demand but not front-loaded into the initial notification.
The Runbook Problem
Runbooks deserve a separate mention because they're widely used and consistently overestimated as incident context. In our analysis, runbook links were present in alert notifications for 58% of incidents. Engineers clicked through to the runbook in 23% of those cases. In almost every case where they clicked through, they spent 2-4 minutes reading the runbook before determining it either didn't apply to their specific situation or was outdated.
Runbooks are excellent reference documents for known failure modes with known fixes. They're poor incident context because they require the engineer to read, comprehend, and apply them under time pressure. The best use of a runbook isn't in the notification; it's linked from the RCA hypothesis after the hypothesis is already formed, as a "if this hypothesis is confirmed, here's the remediation procedure" resource.
Surfacing a runbook before the hypothesis is formed is premature. It anchors the engineer on a known failure mode before they've verified whether that's the actual failure mode. We've seen this lead to engineers following a runbook procedure for 20 minutes before realizing the current incident doesn't match the scenario the runbook describes.
Building Better Alert Context: Practical Starting Points
If you want to apply these findings without a full context correlation platform, there are a few incremental changes worth making to your current alert configuration.
First, enrich your PagerDuty or OpsGenie alert payloads with deploy events. Most CI/CD platforms support webhooks. Send a deploy event to your alert enrichment layer on every deploy, tagged by service. When an alert fires, look up deploys in the prior 2 hours for the alerting service and its known dependencies. Append them to the alert body. This is implementable in a day with a simple Lambda or Cloud Function.
Second, add the alerting service's known dependency list to the alert template. Even a static list maintained in a config file is better than nothing. It prompts the on-call to check dependencies early rather than late.
Third, include a direct link to the log query that surfaces new error patterns in the last 15 minutes, not a link to the log platform home page. The difference is whether the engineer has to construct a query at 5am or just click through to pre-constructed evidence.
These three changes won't match the automated correlation we've built into Devtract, but they address the most impactful gaps: change signal, dependency context, and log shortcut. In teams we've spoken with who have made these changes manually, the subjective improvement in early-morning incident experience is consistent. Engineers report feeling less confused in the first two minutes, which is exactly the cognitive window where the design has the most impact.