How to monitor RabbitMQ

What can go wrong with RabbitMQ

• Resource alarms block publishers. When memory use passes the high watermark or free disk drops below disk_free_limit (50 MB by default), RabbitMQ blocks all publishing connections. Producers hang rather than error.
• No consumers. A consumer deploy failed and a queue has zero consumers; messages pile up until memory or disk alarms trigger.
• Unacked messages pile up. A consumer receives messages and never acknowledges them (stuck handler, large prefetch). They are invisible to other consumers until the channel closes.
• Network partition. In a cluster, a partition splits the nodes; depending on cluster_partition_handling some nodes pause or diverge.
• File descriptors. Each connection needs a file descriptor; running out refuses new connections.

The signals that matter, with alert thresholds

Thresholds are starting points. Tune them after a week of data: an alert nobody acts on should be raised or removed.

How to check RabbitMQ by hand

sudo rabbitmq-diagnostics -q ping
sudo rabbitmq-diagnostics -q check_local_alarms

sudo rabbitmqctl list_queues name messages_ready messages_unacknowledged consumers

sudo rabbitmq-diagnostics cluster_status

curl -s -u monitor:secret http://localhost:15672/api/health/checks/alarms

How to watch RabbitMQ with Vigil

1. In New monitor create an App health monitor on http://rabbit.internal:15672/api/health/checks/alarms with an Authorization: Basic … header for a user with the monitoring tag. It answers {"status":"ok"}, or HTTP 503 with the alarm when one is in effect. Run it from a private location if the port is internal.
2. Add the host under Servers and run the one-line installer it shows (curl -fsSL https://puzaricloud.in/install-server-agent.sh | sh -s -- --token <token>). The agent is one Python file with no dependencies; CPU, memory, disk, inode and load rules start working with the first report. Add rabbitmq-server to the systemd services to watch.
3. Add queue depth and consumer commands for the queues that matter (below), with limits such as rabbit_orders_ready = 5000 and rabbit_orders_no_consumers = 1.
4. With the rabbitmq_prometheus plugin enabled, you can instead add http://127.0.0.1:15692/metrics under Prometheus exporters to scrape and set limits on those series.

{
  "commands": {
    "rabbit_orders_ready": "rabbitmqctl list_queues -q name messages_ready | awk '$1==\"orders\" {print $2}'",
    "rabbit_orders_no_consumers": "rabbitmqctl list_queues -q name consumers | awk '$1==\"orders\" {print ($2==0)}'"
  }
}

Every alert goes to the channels you pick: email, Slack, Telegram, Discord, Microsoft Teams, Pushover, webhooks or on-call paging. Failures are confirmed by a second check before anyone is woken up.

RabbitMQ monitoring FAQ