The Guide
Field-tested practices for running on-call, responding to incidents, and learning from them — distilled into short, opinionated reads.
On-call
- Building a humane on-call rotationHow to design an on-call schedule that protects your engineers, distributes load fairly, and still guarantees that every page reaches a human who can act.
- Writing alerts worth waking up forEvery alert is a promise that something needs a human right now. Learn to write alerts that page only when that promise is true — and to delete the ones that aren't.
- Designing escalation that always finds someoneA page that goes unanswered is worse than no page at all. Learn to build escalation paths that route around a sleeping responder, respect working hours across time zones, and guarantee that every alert eventually reaches a human who can act.
- Following the sun without burning anyoneRound-the-clock coverage does not have to mean round-the-clock suffering. Learn how follow-the-sun rotations, working-hours rules, and humane compensation let a global team hold the pager all night without anyone losing their nights.
Incident response
- What counts as an incident, and why to declare earlyThe cheapest mistake in incident response is declaring something that turns out to be nothing. The expensive one is the silent hour where everyone assumed someone else had it. Learn what actually warrants an incident and why the bias should always be toward declaring sooner.
- Severity levels that actually rank impactA severity scale exists to answer one question under pressure — how hard do we push, and who do we wake? Learn to build levels defined by impact rather than guesswork, anchored to observable signals, and free of the urgency-versus-importance confusion that breaks most scales.
- Response roles that keep a busy room coordinatedWhen a serious incident pulls in half the team, the failure mode is no longer too few hands — it's too many, all uncoordinated. Learn the small set of roles that turns a crowd into a response: a single accountable lead, a dedicated comms voice, and clean handoffs.
- The incident lifecycle, from triage to resolutionEvery incident moves through the same shape — get acknowledged, get understood, get mitigated, get resolved — even though the technical work is different every time. Learn the lifecycle that gives a chaotic response a backbone, and why mitigation and resolution are not the same milestone.
Communication
- Writing incident updates people trustAn incident update is a promise about cadence, a statement of impact, and a small act of honesty all at once. Learn to write updates that tell customers what they actually need — what's broken, who it affects, and when they'll hear from you next — without overclaiming, hedging, or going silent.
- Internal and external communication are different jobsThe conversation inside the war room and the message you send to customers serve opposite goals, and conflating them creates leaks, confusion, and contradiction. Learn to run two clean channels — fast and messy on the inside, calm and curated on the outside — and the single role that connects them.
- Running a status page customers trustA status page is only worth having if people believe it. Learn what makes a status page credible — honest component scoping, fast and human updates, a hostable history — and the failure modes that turn it into a green light nobody looks at.
- Communicating planned maintenance wellPlanned maintenance is the one disruption you get to communicate before it happens — so there's no excuse for it surprising anyone. Learn to announce, schedule, and report maintenance so that expected downtime never gets mistaken for an outage and never erodes the trust your incident comms work so hard to build.
Learning
- Running a blameless post-mortemA post-mortem is not a trial, and it is not a formality. It is the one moment when a team can look honestly at how its system actually behaved under stress. Learn what makes a post-mortem blameless in practice, not just in name, and why that is the only kind worth running.
- Writing post-mortems people actually readA perfect investigation buried in a document nobody opens has changed nothing. The hard part of a post-mortem is not finding the truth — it is writing the truth so that a busy stranger reads it, believes it, and acts on it. Learn to write for the reader you will never meet.
- Turning learnings into follow-ups that get doneThe most common failure in post-incident learning is not a bad investigation — it is a good one whose follow-ups quietly die in a backlog. Learn how to write follow-ups that change the system, give them an owner and a deadline, and make sure they actually ship before the next incident proves they should have.
- Building a culture that learns from failurePost-mortems and follow-ups are practices; a learning culture is the thing that makes them stick when no incident is forcing the issue. Learn how to make learning a habit rather than a reaction — through blameless norms, rehearsed failure, and shared knowledge that outlives any individual.
Insights
- The metrics that matter, and where they lieMTTA and MTTR are the two numbers every incident program eventually reports, and for good reason — they are the closest thing the discipline has to a universal vocabulary for speed. But a number you do not understand is a number that will mislead you. Learn what these metrics genuinely measure, the specific ways they distort, and how to read them so they sharpen your judgment instead of replacing it.
- The quality signals that speed hidesA response can be fast and still be bad — silent for twenty minutes in the middle, acknowledged by someone who then vanished, technically resolved but never explained to the people who were waiting. The metrics that capture this live beyond the stopwatch. Learn to measure update discipline and genuine engagement, and why these signals catch failures that speed metrics are structurally blind to.
- Reading the human cost of on-callEvery dashboard tells you how your systems are doing and almost none tell you how your people are doing — yet the sustainability of on-call is the constraint that decides whether your response practice survives. Learn the signals that reveal when a rotation is quietly burning people out, why out-of-hours load is the metric that matters most, and how to read the data before your best responders leave.
- Using data to improve, without gaming itThe moment a metric becomes a target, people start optimizing the number instead of the thing it was supposed to represent — and incident response is unusually easy to game. Learn how to run a measurement practice that actually drives improvement: which metrics to expose to whom, why trends beat thresholds, and how to keep the numbers honest by never turning them into a scoreboard.