Writing alerts worth waking up for
Every alert is a promise that something needs a human right now. Learn to write alerts that page only when that promise is true — and to delete the ones that aren't.
An alert is a request to interrupt a human life. Held to that standard, most of the alerts in a typical system do not deserve to exist. They fire on conditions no one acts on, repeat until they are tuned out, and slowly teach the on-call engineer that the pager is mostly noise — which is exactly the lesson you cannot afford them to learn before the one alert that matters.
Page on symptoms, monitor on causes
The single most useful rule for alert design is to page on what the user feels and merely record what you suspect is behind it. A user does not care that a disk is 90% full; they care that checkout returns a 500. Alert on the elevated error rate — the symptom — and let the disk metric live on a dashboard the responder consults after they have been paged. Cause-based alerts multiply without bound, because there are a hundred ways to break checkout, and each new one earns its own 3am false alarm. Symptom-based alerts stay few, because there are only so many ways the user can be hurt.
Every alert needs an owner and an action
Before an alert ships, answer two questions: who is woken by this, and what do they do when it fires? If you cannot name a concrete first action — restart this, fail over to that, escalate to them — the alert is not actionable and should be a dashboard, not a page. Link the runbook directly from the alert so the responder is one click from knowing what to do, not one frantic search away. An alert without an action is just an anxiety with a pager attached.
Let severity decide whether to wake someone
Not everything that is worth knowing is worth waking someone for, and conflating the two is how pagers earn their reputation for crying wolf. Give every alert a severity that answers a single blunt question: does a human need to act on this right now? A checkout outage is a yes — page someone, immediately. A slow-but-stable background job is a no — record it, surface it on a dashboard, raise a ticket for business hours, but do not interrupt a life over it. The discipline is to be ruthless about what clears the bar for an interruption and generous about what merely gets logged. Most of what your systems emit is genuinely interesting and genuinely not urgent; sorting it by severity, and routing only the top of that pile to a pager, is what keeps the pager meaning something. When you find yourself tempted to page on a warning "just so someone sees it," that is the signal it belongs on a dashboard, not in someone's sleep.
Group related alerts so one failure is one page
A single underlying failure rarely trips a single alert. A database falls over and suddenly every service that touches it is firing — a dozen alerts, a dozen pages, one root cause. Waking a responder twelve times for one incident is not twelve times as helpful; it is one helpful page buried under eleven that bury the signal and fray the nerves. Group alerts that share a cause or a target into a single notification, so the responder is woken once, with the shape of the problem already drawn for them, rather than drip-fed fragments they have to reassemble at 3am. The aim is one incident, one page: the responder should be interrupted in proportion to the number of problems, never the number of symptoms those problems happen to throw off.
Tune ruthlessly, and delete without ceremony
Treat every false page as a bug in your alerting, not an unavoidable cost of running software. Hold a short review of what paged each week, and for anything that fired without needing a human, either raise the threshold, add a duration the condition must persist, or delete the alert outright. The graveyard of deleted alerts is a sign of a healthy system, not a careless one — each deletion is one fewer needless interruption, one more reason the responder trusts the next page is real. Pair the review with the inverse question, too: was there an incident this week that no alert caught, where a user told you before your systems did? A missing alert is as much a bug as a noisy one, and the weekly review is where both get found.
The reward for this discipline is the most valuable thing an on-call system can have: a responder who believes the pager. When alerts are rare, actionable, and true, the engineer who is woken does not groan and silence it — they get up, because they know it matters. That belief is fragile, expensive to rebuild, and worth more than any amount of coverage.