Running a blameless post-mortem

Every incident leaves behind two stories. The first is the one that gets told in the hallway: a short, satisfying tale with a culprit at its center — the engineer who pushed the bad config, the team that skipped the review, the vendor who went down. The second is the real story, which is almost always longer, stranger, and more uncomfortable: a chain of reasonable decisions, missing guardrails, and ambiguous signals that lined up just so. A post-mortem exists to surface the second story, and the entire reason it has to be blameless is that the second story only comes out when nobody is bracing to be the villain of the first.

"Blameless" is one of the most misunderstood words in this discipline. People hear it as a softness — a polite agreement not to point fingers, a way to spare feelings. It is not that at all. Blamelessness is a hard engineering stance with a specific claim behind it: that the people involved in an incident were, with the information and tools they had at the time, doing reasonable work. If that is true, then the interesting question is never "who messed up" but "what about our system made this failure possible, and likely, and hard to catch?" Blame ends an investigation. Blamelessness is what keeps it open long enough to find the mechanism.

Blame is a debugging tool that stops working too early

The reason to be ruthless about blamelessness is not sentiment — it is signal quality. The moment a post-mortem becomes a search for the responsible person, two things happen, and both of them destroy the data you came for. First, the person at the center stops explaining and starts defending. The most valuable testimony in any post-mortem is "here is what I was thinking when I ran that command, and here is why it seemed correct" — and that sentence is impossible to say in a room where the next question might end your week. Second, everyone watching learns the real lesson: that the safe move during an incident is to be cautious, quiet, and slow to act, because acting is what gets you named. You will have optimized your team for exactly the wrong behavior under pressure.

There is also a simple logical problem with blame. By the time you are holding a post-mortem, you already know the outcome was bad — which makes every decision that led to it look obviously wrong in hindsight. This is hindsight bias, and it is not a minor distortion; it is the default way human brains read the past. The engineer who "should have known" the config was dangerous did not have the alert you now wish had existed, did not see the graph you now stare at, and was weighing that change against ten other plausible things at the same moment. Judging that decision by its outcome teaches you nothing transferable. Understanding why it looked correct teaches you where to put the guardrail.

What a blameless investigation actually looks for

A good post-mortem walks the incident as a sequence of events and asks, at each step, a mechanical question: what did the system make easy, and what did it make hard? A deploy that took down production is rarely interesting because someone deployed; it is interesting because the deploy pipeline let a change reach every region at once with no canary, because the bad config passed validation, because the resulting errors took eleven minutes to page anyone, and because the rollback required a runbook nobody had touched in six months. Each of those is a property of the system, each is fixable, and not one of them is a person.

Replace person-shaped causes with system-shaped ones, relentlessly. "The on-call engineer didn't escalate fast enough" becomes "our escalation policy had no automatic time-out, so escalation depended on a tired human remembering to do it." "Someone merged without enough review" becomes "our review process treats a one-line config change and a thousand-line refactor identically, so risky changes get the same scrutiny as trivial ones." The test is simple: if your finding could be fixed by sending one person to training or telling them to be more careful, you have not finished. Keep going until you find the thing in the system you could change so that the next reasonable person, on a worse day, does not fall into the same hole.

Run it soon, run it on a template, run it for the reader

Timing matters more than teams expect. Hold the post-mortem while memory is fresh — within a few days, before the chat logs blur and people reconstruct a tidy narrative they half-believe. Pull the facts before the meeting, not during it: the timeline, the graphs, the exact commands, the moment each alert fired. A post-mortem that spends its hour arguing about what happened never gets to the more important hour about why.

Give it a consistent shape so writers are not starting from a blank page and readers always know where to look. A workable structure is a short summary anyone can read in thirty seconds, the customer-facing impact, a factual timeline, the contributing factors that made the failure possible, an honest note on what went well (your fast rollback, your clear communication — these are worth protecting deliberately), and the follow-ups that will change the system. Naming what went well is not a consolation prize; it is how you keep the parts of your response that worked from quietly eroding the next time someone "optimizes" them away.

Finally, remember who the document is for. A post-mortem read only by the people who were in the incident has wasted most of its value. The audience is the engineer two teams over who will hit the same class of bug next quarter, the new hire learning how your systems actually fail, the manager deciding where reliability investment goes. Write for that reader: enough context that an outsider can follow it, no jargon you would not explain, and a clear line from "this is what broke" to "this is what we are changing so it breaks less." Do that consistently and the post-mortem stops being paperwork you file after an outage and becomes the single most concentrated source of hard-won institutional knowledge your team has.