How to monitor Docker containers
Monitor Docker containers by alerting when one exits, dies or keeps restarting, when the kernel OOM-kills it, and when its HEALTHCHECK turns unhealthy, then watch host disk for images and logs. docker ps and docker inspect show each by hand; Vigil's server agent watches every container on the host and opens an incident on the first bad state.
What can go wrong with Docker containers
- Restart loops hide crashes. With
restart: alwaysorunless-stoppeda crashing container comes back every few seconds and looks up most of the time, whileRestartCountclimbs. - OOM kills. A container that exceeds its memory limit is killed by the kernel: exit code 137 and
State.OOMKilled=true. Without a limit, the whole host runs out of memory instead. - Unhealthy but running. Outside Swarm, Docker does not restart a container whose HEALTHCHECK fails; it just marks it
unhealthyand keeps routing to it. - Disk filled by logs and images. The default json-file log driver has no size limit, and old images and build cache pile up under
/var/lib/dockeruntil the disk is full and every container fails. - Daemon down. If dockerd stops, every container on the host stops with it (unless live-restore is on).
The signals that matter, with alert thresholds
| Signal | Alert when | Why it matters |
|---|---|---|
| Container state | exited, dead, restarting or paused | The service is not running. |
| Restart count | increasing | A crash loop behind a restart policy. |
| OOMKilled / exit code 137 | any | The memory limit is too low or the app leaks. |
| Health status | unhealthy | Running but not serving. |
| Container memory vs limit | ≥ 90% | OOM kill is next. |
| Disk used on /var/lib/docker | ≥ 85% | Logs and images need pruning. |
Thresholds are starting points. Tune them after a week of data: an alert nobody acts on should be raised or removed.
How to check Docker containers by hand
docker ps -a --filter status=exited --filter status=dead
docker inspect -f '{{.State.ExitCode}} oom={{.State.OOMKilled}} restarts={{.RestartCount}}' my-container
docker inspect -f '{{json .State.Health}}' my-container
docker stats --no-stream
docker system df
docker events --since 1h --until "$(date +%s)" --filter event=die
How to watch Docker containers with Vigil
- Add the host under Servers and run the one-line installer it shows (
curl -fsSL https://puzaricloud.in/install-server-agent.sh | sh -s -- --token <token>). The agent is one Python file with no dependencies; CPU, memory, disk, inode and load rules start working with the first report. Install it on the host, not in a container; it reads container state through thedockerCLI. - Leave Alert when a Docker container exits, dies or restart-loops on in the server's Alert rules. Each container gets its own incident, which resolves when it is running again. The Services & containers tab shows state, CPU and memory per container.
- Docker only marks unhealthy containers, so add the unhealthy-count command below with the limit
docker_unhealthy = 1. - Keep the disk rule (85%) and the disk-full forecast on; they catch runaway logs. Set
log-optsmax-sizein/etc/docker/daemon.jsonto fix the cause. - Add an Uptime or API check from New monitor on each published port, so you also know when a running container stops answering.
{
"commands": {
"docker_unhealthy": "docker ps --filter health=unhealthy -q | wc -l"
}
}
Every alert goes to the channels you pick: email, Slack, Telegram, Discord, Microsoft Teams, Pushover, webhooks or on-call paging. Failures are confirmed by a second check before anyone is woken up.
Docker containers monitoring FAQ
Will one-off containers trigger alerts?
Yes: any container left in the exited state counts. Run one-off jobs with docker run --rm, or remove them when done, so only real failures alert.
Does it work with Docker Compose and Portainer?
Yes. Compose and Portainer create ordinary containers, and the agent sees them by name like any other.
What about Podman?
The agent reads Docker through the docker command. With podman-docker installed, which provides a docker-compatible command, the same view works; otherwise use the process and service watches.