What is an incident?
An incident is an unplanned interruption or degradation of a service that needs someone to respond, such as an outage, a spike in errors, slow pages or a failed backup. Incident management covers detecting it, alerting the right person, acknowledging, communicating with users, fixing it, and learning from it afterwards in a postmortem.
Why an incident matters
Calling something an incident starts a process: one owner, one place for updates, a status page entry, and a record afterwards. Without that, outages are handled ad hoc in several chat threads and the same failure returns.
Incident timestamps are also where reliability numbers come from. Start, detection, acknowledgement and resolution times feed MTTD, MTTR and SLO reports, so recording them accurately matters.
How to measure an incident
time to detect = detected − started time to acknowledge = acknowledged − detected duration = resolved − started
Worked example
An API starts returning errors at 14:02. Monitoring confirms the failure and alerts at 14:05, the on-call engineer acknowledges at 14:09, and a rollback resolves it at 14:41.
- Time to detect = 14:05 − 14:02 = 3 minutes.
- Time to acknowledge = 14:09 − 14:05 = 4 minutes.
- Duration = 14:41 − 14:02 = 39 minutes, all of which counts as downtime against the SLO.
Result: Of the 39 minutes, 7 passed before anyone was working on it. That is the part monitoring and paging can shrink; the other 32 are diagnosis and fix.
Common mistakes
- No severity levels. If every incident is treated the same, minor ones wake people up and major ones get no extra help. Define two or three severities.
- Fixing without communicating. Users and support staff need updates. Post on the status page even when the cause is still unknown.
- Closing without follow-up. An incident that ends with "it works again" and no actions will recur.
- Using the alert time as the start time. The incident started when users were affected, which is usually before the alert. Record both.
How Vigil handles an incident
- Vigil opens an incident automatically when a monitor or server rule fails (after a confirmation check), alerts your channels, and closes it when the check recovers.
- Incidents can be acknowledged, updated, escalated if nobody acknowledges them, and resolved; failure evidence (status, headers, timings, the first 4 KB of the body) is attached.
- "Analyse with AI" drafts a summary, likely cause, fix steps and a postmortem from the evidence, and Time Detective shows everything that happened around the start time. Public updates go to your status page.
Incident: frequently asked questions
What is the difference between an incident and an alert?
An alert is a notification that a condition was met. An incident is the event being handled; several alerts can belong to one incident, and many alerts never become incidents.
Who should own an incident?
Usually the on-call engineer who acknowledged it, until they hand over. For large incidents a separate incident commander coordinates while others fix.
Do small incidents need a postmortem?
Not always. Write one for anything that affected users noticeably, broke an SLO, or could have been much worse; skip it for well-understood minor blips.