How to monitor Kubernetes nodes and pods
Monitor Kubernetes on two layers. Cluster health means nodes Ready with no memory, disk or PID pressure, and no pods in CrashLoopBackOff, OOMKilled or Failed. Service health means each ingress answers correctly from outside. Vigil's DaemonSet watches pods and nodes and opens incidents; uptime and API checks cover what users reach.
What can go wrong with Kubernetes nodes and pods
- CrashLoopBackOff. A container exits right after start; Kubernetes restarts it with a growing back-off (up to five minutes). The pod is Running between attempts, so naive checks miss it.
- OOMKilled. The container went over its memory limit and the kernel killed it.
kubectl describe podshowsLast State: Terminated, Reason: OOMKilled. - Node NotReady. The kubelet stopped reporting (crash, network, full disk). After the eviction timeout its pods are rescheduled, if there is room elsewhere.
- Pressure conditions.
MemoryPressure,DiskPressureandPIDPressuremake the kubelet evict pods and refuse new ones. DiskPressure is often container images and logs. - Everything green, users still down. An ingress, certificate or DNS problem leaves every pod healthy while nobody can reach the service.
The signals that matter, with alert thresholds
| Signal | Alert when | Why it matters |
|---|---|---|
| Pods in CrashLoopBackOff | any, for > 5 min | A broken deploy or config. |
| OOMKilled terminations | any | Limits too low or a leak. |
| Pods in Failed phase | any outside finished Jobs | Something exited non-zero. |
| Node Ready condition | False or Unknown | Capacity lost. |
| Memory / Disk / PID pressure | True | Evictions are happening. |
| Ingress HTTP check | non-2xx or p95 > 1 s | What users actually get. |
Thresholds are starting points. Tune them after a week of data: an alert nobody acts on should be raised or removed.
How to check Kubernetes nodes and pods by hand
kubectl get nodes -o wide
kubectl get pods -A | grep -E 'CrashLoopBackOff|OOMKilled|Error|ImagePull|Pending'
kubectl describe pod my-pod -n my-ns | grep -A5 'Last State' kubectl logs my-pod -n my-ns --previous
kubectl describe node my-node | grep -A8 Conditions
kubectl get events -A --sort-by=.lastTimestamp | tail -30
kubectl top nodes kubectl top pods -A --sort-by=memory | head
How to watch Kubernetes nodes and pods with Vigil
- Open Kubernetes, name the cluster and apply the DaemonSet manifest it shows with
kubectl apply -f. One agent per node reads its pods, node conditions, restarts and kubelet CPU and memory. - Incidents open on their own for CrashLoopBackOff and OOMKilled containers, Failed pods, NotReady nodes and Memory/Disk/PID pressure, and resolve when the condition clears.
- From New monitor add an Uptime or API check for each public ingress, with SSL expiry. For internal services, run the checks from a private location inside the cluster network.
- Already running Prometheus? Point
remote_writeat Vigil's Prometheus endpoint, or send Alertmanager to an alert ingestion URL, and route those alerts through the same channels and on-call rota. Traces go to the OTLP endpoint (see OpenTelemetry quick start). - For scheduled workloads, see how to monitor Kubernetes CronJobs: a pod-level view cannot tell you a job never started.
Every alert goes to the channels you pick: email, Slack, Telegram, Discord, Microsoft Teams, Pushover, webhooks or on-call paging. Failures are confirmed by a second check before anyone is woken up.
Kubernetes nodes and pods monitoring FAQ
Do I need Prometheus to use this?
No. The DaemonSet talks to the Kubernetes API and the kubelet directly. If you already run Prometheus you can also send its data and alerts to Vigil, but it is optional.
Does it work on managed clusters (EKS, GKE, AKS)?
Yes. It is an ordinary DaemonSet with read access to pods and nodes, so any conformant cluster works, including k3s and microk8s.
Why alert on OOMKilled when the pod restarts by itself?
Because it will happen again: an OOM kill means the limit is wrong or the app leaks. Each kill drops in-flight requests, and repeated kills turn into CrashLoopBackOff.