How to monitor systemd services and timers
Monitor systemd services by alerting when a unit you depend on is failed, inactive or missing, and by watching NRestarts so a crash loop hidden by Restart= does not go unnoticed. Ship the unit's journal for errors. For timers, make the service ping a heartbeat on success. Vigil's agent watches named units and every failed unit.
What can go wrong with systemd services and timers
- Crash loops behind Restart=. With
Restart=on-failurea crashing service comes back in a second and looks active most of the time. Only the restart counter and the journal show it. - start-limit-hit. If a unit fails to start more than
StartLimitBursttimes (5) withinStartLimitIntervalSec(10 s) by default, systemd gives up and leaves it failed withstart-limit-hit. - Not enabled. A unit that was started by hand but never enabled is gone after the next reboot.
- Timers that stop firing. A timer whose service fails, or whose OnCalendar was edited without a daemon-reload, simply stops doing its job; nothing errors.
- Dependencies. A service that starts before its database or network is ready fails, and stays failed unless Restart= is set.
The signals that matter, with alert thresholds
| Signal | Alert when | Why it matters |
|---|---|---|
| Watched unit state | failed, inactive or not-found | The service is down. |
| Any failed unit on the host | count > 0 | Something broke you did not think to watch. |
| NRestarts | increasing | Crash loop. |
| Errors in the unit's journal | above the normal rate | Early warning. |
| Timer's service success ping | late | The scheduled work stopped happening. |
Thresholds are starting points. Tune them after a week of data: an alert nobody acts on should be raised or removed.
How to check systemd services and timers by hand
systemctl --failed
systemctl is-active myapp systemctl show myapp -p ActiveState -p SubState -p NRestarts -p Result
journalctl -u myapp -p err -b
systemctl list-timers --all
systemctl is-enabled myapp
How to watch systemd services and timers with Vigil
- Add the host under Servers and run the one-line installer it shows (
curl -fsSL https://puzaricloud.in/install-server-agent.sh | sh -s -- --token <token>). The agent is one Python file with no dependencies; CPU, memory, disk, inode and load rules start working with the first report. - In the server's Alert rules, list the units under systemd services to watch (e.g.
myapp, nginx, postgresql). The list is pushed to the agent; a unit that is failed, inactive or missing opens an incident. Any other failed unit on the host opens a separate "failed services" incident. - Add the restart counter below with a limit (
myapp_restarts = 5) to catch loops that Restart= hides. - Put the unit under Ship journal units to search its logs in Logs, and add a log alert rule for level
errorabove your normal rate. - For timers, add
ExecStartPost=/usr/bin/curl -fsS -m 10 https://puzaricloud.in/ping/<token>to the oneshot service and create a Heartbeat with the timer's schedule. ExecStartPost only runs when ExecStart succeeded.
{
"commands": {
"myapp_restarts": "systemctl show myapp -p NRestarts --value"
}
}
Every alert goes to the channels you pick: email, Slack, Telegram, Discord, Microsoft Teams, Pushover, webhooks or on-call paging. Failures are confirmed by a second check before anyone is woken up.
systemd services and timers monitoring FAQ
How do I see why a service failed?
systemctl status myapp shows the last lines and the result (exit-code, signal, timeout, start-limit-hit); journalctl -u myapp -b shows the whole story since boot.
Can systemd alert on failure by itself?
Yes, with OnFailure= pointing at a unit that sends a message, e.g. a oneshot that curls /ping/<token>/fail. Vigil's agent covers this without per-unit wiring and also catches units that were never started.
How do I convert a cron job to a systemd timer?
Every page in the cron library shows the equivalent OnCalendar timer and service units, with the heartbeat ping included.