The Incidents That Never Paged: Treating Near Misses as Your Most Underused Reliability Data
Photo: Internet Archive Book Images, No restrictions, via Wikimedia Commons
Every reliability program has a number. Availability, error budget consumption, mean time between incidents. All of them share a property that makes them comfortable and slightly useless: they are retrospective. They describe the failures that already reached a customer, which is a strange thing to optimize for.
The data that predicts the next outage is usually discarded before anyone sees it. It arrives as a message in a channel at 4:40 p.m., a sentence in a code review, or a comment in a postmortem that says, in effect, that was close.
What a near miss actually is
A near miss is not a small incident. It is an event that would have become an incident but for a condition nobody designed: a retry that happened to succeed, a deploy that happened to land outside a batch window, an engineer who happened to remember an undocumented dependency.
The distinguishing feature is that the guard was accidental rather than structural. That is why the entry matters. An incident tells you a control failed. A near miss tells you a control was absent and luck covered the difference. Luck is not renewable at scale.
Count what paged, and you will manage what paged
Organizations measure what their tooling emits. Paging systems emit incidents, so incidents get counted, reviewed, and remediated. Near misses emit nothing, so they become folklore. Six months later the folklore is gone, and the same accident finally gets a ticket number and a customer impact statement.
This is not a failure of diligence. It is a failure of format. If the only reliable artifact is an incident record, then the organization will keep producing incident records and calling it learning.
Capture has to cost almost nothing
Most attempts at near-miss logging die because they are designed as a second incident process. They inherit severity levels, approval steps, and a form with twenty fields. Engineers stop filing.
The version that survives is deliberately thin. One line in a dedicated channel, or a single template with four fields:
- What almost happened, in one sentence.
- What prevented it.
- What would have made it worse.
- A guess at what would prevent it next time, even if the guess is wrong.
No severity, no owner required at capture time, no approval. The writer's name is optional and should be genuinely optional. Capture is a recording act, not a commitment.
The monthly review is a decision meeting
An archive nobody reads is a diary. The review should take thirty minutes, happen on a fixed date, and include engineering leadership, the on-call rotation, and one person from product. The agenda is not a discussion of each entry. It is a sorting exercise: which of these describe a structural gap, and which are already covered by planned work.
The output is short. Three to five items classified as structural risk, each with a rough size and a decision about where it goes: into the current quarter, into the backlog with a date, or into a documented acceptance of risk. Acceptance is a legitimate outcome and should be recorded as such. A risk that has been consciously accepted is a different thing from a risk nobody noticed.
Wiring it to the roadmap
This is where most programs stop short. A near-miss archive that produces observations but never alters a plan becomes a complaint file, and complaint files are ignored by the second quarter.
The connection point is the roadmap review, not the incident review. Bring two numbers: how many entries this quarter described a single point of accidental protection, and how many of those are now covered by scheduled work. That ratio is a crude but effective measure of whether the organization is learning or merely documenting. Product leaders respond to it because it is a risk statement with a trend line, which is more than most infrastructure work ever gets.
The failure mode to watch for
The archive turns toxic the moment it becomes evidence. If an entry can be retrieved during a performance review, a blame conversation, or a postmortem about a different incident, capture stops within a month, and it stops quietly, which is worse than not starting.
Three defenses are worth writing down. Entries are not attributable without the author's consent. Entries are never used to reopen a closed incident. And the review discusses patterns, not people. Leadership has to enforce these visibly, including the uncomfortable time a senior engineer's near miss appears in the pattern count.
What the method cannot do
A near-miss archive does not replace incident review, and it will not catch a failure mode nobody has approached yet. It is a record of the organization's own experience, which means it is biased toward the systems people already work in.
What it does do is convert private knowledge into institutional memory at a cost of roughly one sentence per event. Most reliability programs are starved of leading indicators and oversupplied with lagging ones. This is the cheapest correction available, and the reason it is rare has nothing to do with difficulty.
It has to do with the fact that the most valuable failure in your organization today will leave no trace unless someone writes it down before the luck runs out.