How to monitor Elasticsearch and OpenSearch

What can go wrong with Elasticsearch and OpenSearch

• Red or yellow cluster. Red means at least one primary shard is unassigned: searches return partial results and writes to that index fail. Yellow means replicas are missing, so one more node loss can turn it red.
• Disk watermarks. At 85% disk used (low watermark) no new shards go to the node, at 90% (high) shards move away, at 95% (flood stage) indices with a shard on that node become read-only and indexing fails with a cluster_block_exception.
• Heap pressure and circuit breakers. When JVM heap stays above ~85%, garbage collection pauses get long and requests trip circuit breakers (CircuitBreakingException). Nodes that pause too long drop out of the cluster.
• A node leaves. A node that stops answering is removed; its shards recover elsewhere, which costs I/O and network for minutes or hours.
• The 200-OK trap. GET _cluster/health answers 200 whether the status is green, yellow or red. An uptime check that only looks at the status code never fires.

The signals that matter, with alert thresholds

Thresholds are starting points. Tune them after a week of data: an alert nobody acts on should be raised or removed.

How to check Elasticsearch and OpenSearch by hand

curl -s 'http://localhost:9200/_cluster/health?pretty'

curl -s 'http://localhost:9200/_cluster/allocation/explain?pretty'

curl -s 'http://localhost:9200/_cat/allocation?v'

curl -s 'http://localhost:9200/_cat/nodes?v&h=name,heap.percent,ram.percent,disk.used_percent,node.role'

curl -s 'http://localhost:9200/_cat/shards?v&h=index,shard,prirep,state,unassigned.reason' | grep -v STARTED

With security on (the default since Elasticsearch 8), add -u elastic or -H 'Authorization: ApiKey <key>' and use https. The same APIs work on OpenSearch.

How to watch Elasticsearch and OpenSearch with Vigil

1. In New monitor pick API check, URL https://es.internal:9200/_cluster/health, header Authorization: ApiKey … (a key with the monitor cluster privilege is enough).
2. Add assertions, one per line: json status == green, json number_of_nodes >= 3, json unassigned_shards == 0. A failed assertion is confirmed by a second check 12 seconds later and the failure evidence keeps the JSON body.
3. If the cluster is not reachable from the internet, run the check from a private location: the same agent pulls the job inside your network and Vigil evaluates the result.
4. On each node, install the server agent and add the disk and heap commands below as custom metrics with limits (es_disk_max_pct = 85, es_heap_max_pct = 85). Add a log_watch for flood stage disk watermark in the cluster log (/var/log/elasticsearch/<cluster.name>.log; one file per entry, no globs).
5. Add elasticsearch (or opensearch) to the systemd services to watch.

{
  "commands": {
    "es_disk_max_pct": "curl -s -u monitor:secret 'http://localhost:9200/_cat/allocation?h=disk.percent' | sort -n | tail -1",
    "es_heap_max_pct": "curl -s -u monitor:secret http://localhost:9200/_nodes/stats/jvm | jq '[.nodes[].jvm.mem.heap_used_percent] | max'"
  },
  "log_watch": [
    {"name": "es_flood_stage", "path": "/var/log/elasticsearch/my-cluster.log", "pattern": "flood stage disk watermark"}
  ]
}

Every alert goes to the channels you pick: email, Slack, Telegram, Discord, Microsoft Teams, Pushover, webhooks or on-call paging. Failures are confirmed by a second check before anyone is woken up.

Elasticsearch and OpenSearch monitoring FAQ