What is alert fatigue?
Alert fatigue is what happens when a team receives so many alerts, most of them not needing action, that people start ignoring, muting or skimming them. The cost is the real outage that arrives looking like the hundred false ones before it. It is fixed by alerting on symptoms and removing alerts nobody acts on.
Why alert fatigue matters
Every alert asks a person to stop what they are doing. When most alerts turn out to be harmless (a CPU spike that resolved itself, a check that failed once from one network path, a disk at 81% that has sat at 81% for a month), people learn that the cheapest response is to wait and see. That habit is exactly wrong for the alert that matters.
Fatigue also spreads quietly: noisy channels get muted, phones get set to silent, and new engineers inherit a rota they dread. Hospitals have long described the same pattern as alarm fatigue, which is why the idea is used well beyond software operations.
How to measure alert fatigue
actionable ratio = alerts that needed a human action ÷ all alerts sent alerts per person = all alerts ÷ people sharing the rota
Worked example
A team of two on a weekly rota receives 420 alerts in one week. Reviewing them on Friday, 38 needed someone to do something (restart, roll back, add disk); the other 382 resolved on their own or were duplicates.
- Actionable ratio = 38 ÷ 420 = 0.0905, so 9.0% of alerts were worth interrupting someone for.
- Noise = 420 − 38 = 382 alerts that trained the team to ignore the pager.
- Per person: 420 ÷ 2 = 210 alerts per person per week, 30 a day.
Result: A ratio this far below one half is a fatigue problem. Start by deleting or downgrading the rules behind the 382, not by asking people to try harder.
Common mistakes
- Alerting on causes instead of symptoms. CPU at 90% is a cause that may never hurt anyone; "checkout returns 5xx" is a symptom users feel. Page on symptoms and keep causes on a dashboard.
- Paging on a single failed check. One timeout from one network path is noise. Confirm with a retry or a second check before anyone is alerted.
- Thresholds copied from a blog post. A threshold should come from your own baseline. If an alert fires every day and nobody acts on it, it is wrong, whoever wrote it.
- Everything goes to everyone. Route by urgency: pages for what needs action now, a chat channel or a ticket for what can wait until morning.
- Never reviewing what fired. Without a regular look at which alerts fired and what was done about them, noise only grows.
How Vigil handles alert fatigue
- Every failure is confirmed by a second check about 12 seconds later before anyone is alerted, so a single blip does not page you.
- Flapping monitors (4 or more state changes in 30 minutes) send one flapping alert instead of a stream of down and up messages.
- Monitor dependencies suppress the alerts of everything behind a dependency that is already down, so one database outage is one alert, not twenty. Maintenance windows silence planned work, and metric triggers have a warm-up and a cooldown.
- Urgent alerts can go to on-call paging while the rest go to Slack, Telegram, Discord or email, so the phone only rings for what needs it.
Alert fatigue: frequently asked questions
What is a good actionable alert ratio?
There is no standard number, but if fewer than half of your pages lead to an action, the alert rules need work. The aim is that every page is something a person must act on now.
Is alert fatigue the same as alarm fatigue?
It is the same idea. Alarm fatigue is the older term from hospitals, where staff become desensitised to monitor alarms; alert fatigue is the usual term in software operations.
Should I just raise every threshold?
No. Raising thresholds hides noise but also delays real detection. Remove alerts that never lead to action, confirm failures before alerting, and move cause-level metrics to dashboards.