Designing escalation that always finds someone

The primary responder is asleep. Their phone is on silent, or buried under a pillow, or dead on the nightstand. Meanwhile the thing they are meant to be watching is on fire. This is not an edge case — it is Tuesday. Any on-call system that assumes the first person paged will always answer is not a system at all; it is a hope. Escalation is the machinery that turns hope into a guarantee: a defined sequence of who to try next, and next, and next, until someone conscious and capable picks up the page.

An escalation path is an ordered list of fallbacks

The unit of escalation is the step: page this target, wait a bounded amount of time for an acknowledgement, and if none comes, move to the next step. String a handful of these together and you have a path that degrades gracefully. The primary responder gets the first shot. If they do not acknowledge within, say, five minutes, the page rolls to the secondary. If the secondary is also dark, it climbs to the team lead, then to the wider engineering group, then to whoever keeps the lights on at the very end. Each step buys the previous one a grace period and the whole sequence a guarantee: as long as one human in the chain is reachable, the page lands.

The two knobs on each step are who and how long. The "who" should get broader and more senior as you descend — you start with the person most likely to fix it fastest and end with the person you are most certain will answer. The "how long" is the acknowledgement timeout, and it is a genuine trade-off. Too short and you wake the secondary for a page the primary was already typing a response to; too long and you burn precious minutes of an outage waiting on someone who is never going to reply. A few minutes per step is the usual starting point — long enough to fish a phone out of a pocket, short enough that a three-step climb still finds someone inside a quarter of an hour.

Target roles, not names

The most common escalation mistake is hard-coding people. You wire the path to Sam, and then Sam goes on holiday, or changes teams, or leaves — and now your 3am safety net has a Sam-shaped hole in it that nobody noticed until the night it mattered. Point each escalation step at a role instead: the primary on-call slot, the secondary slot, the team's manager-on-call. Let the schedule decide who actually fills that role tonight. The path stays correct as people rotate through it, holidays and departures and reorgs included, because it never mentions a person by name. The schedule answers "who is the primary right now?"; the escalation path only ever asks the question.

Make the last step unconditional

Every clever thing you do to an escalation path — skipping a layer because it is the wrong time of day, routing around a team that is off-shift, suppressing low-urgency noise — introduces a way for a page to fall off the end of the path having reached no one. That failure mode is unacceptable. A page that quietly expires because every step was conditionally skipped is the worst outcome the system can produce: the alert fired, the machinery ran, and still nobody woke up.

Defend against it with a simple invariant: the final step is always unconditional. Whatever filtering, scheduling, or suppression happens above it, the bottom of the path pages someone no matter what. If every other layer declines the page for a good reason, the backstop layer takes it anyway. Treat this as a law, not a nicety — the one guarantee the whole escalation apparatus exists to make is that a real emergency always finds a human, and a backstop is how you keep that promise even when all your cleverness conspires against it.

Test the path before you need it

An escalation path is a piece of critical infrastructure that, by design, you hope never fully executes. That makes it exactly the kind of thing that rots unnoticed. The secondary's phone number is stale. The "manager on-call" role points at someone who changed jobs eight months ago. The timeout is set to an hour because someone was debugging once and never reverted it. None of these are visible until the night the primary is unreachable and the path has to do its job for real — the single worst moment to discover it is broken.

So exercise it deliberately. Run a periodic drill where you trigger a test page and let it climb a step or two, confirming each target actually receives it on the channel you expect. Make a missed test page a bug with an owner, the same as a missed deploy or a failing build. The cost of a quiet afternoon's drill is trivial; the cost of a path that silently fails mid-incident is measured in extended downtime and eroded trust. A safety net you have never tested is not a safety net — it is a story you tell yourself about one.

Build the path to fail safe

The throughline of good escalation design is that every decision should fail in the safe direction. Unsure whether to add another step? Add it — an extra fallback costs you a slightly noisier worst case and buys you another human in the chain. Unsure how long a timeout should be? Err shorter for the early steps, because waking the secondary a minute too eagerly is a far cheaper mistake than losing ten minutes of an outage. Unsure whether a clever skip rule might drop a page? Put a backstop beneath it so the answer does not matter. Designed this way, escalation stops being a hope and becomes what your users were promised all along: when something breaks, the page does not stop climbing until it reaches someone who can make it stop.