The Monitoring Console

The platform’s own health dashboard — the cluster overview an administrator reads first to see whether the platform itself is running well. It is the Monitoring Console app (the cluster_monitoring app), reached from the app menu, and it answers “is the platform healthy” rather than “is the customer’s data healthy”.

---

Overview

The front door: the platform’s core state in one screen.

• Availability — the ratio of available primary shards to all primary shards in the cluster. Watch unusable shards; under load, also watch shards in migration.
• Machine load — load per core (sum of machine loads over sum of cores); above 100% means the service is running under load overall.
• Disk information — cluster disk capacity, watermarks and per-disk distribution.
• Instance deployment — total instances over total machines, i.e. the average instance density per machine.
• Repository information — repos in use, record count and capacity; the guidance is to prefer a reasonable number of repos with sourcetypes for business differentiation rather than many small repos.
• Queries per minute — cluster query load; sustained high volume warrants attention.
• Hotspot threads — where the service’s work is concentrated, useful for governance and diagnosis. This panel is the engine’s own _nodes/hot_threads output rendered verbatim in a single untitled column, not a parsed table: expect stack traces and percentages as text, and read it the way you would read that endpoint. It is a diagnosis aid, not something to chart or alert on.
• Repo / shard / disk traffic — per-second write traffic broken down by repo, by shard, and by disk.
• Writes per second — the write load.

The rest of the console

Beyond the overview, the console splits into the surfaces that explain why an overview metric moved:

surface	what it shows
Query Overview / Operator Overview	query traffic and the operators that consume it (exec_overview, query_analysis_* for plans, schedules and tracing)
Machine Matrix	per-machine health at a glance
Node Load / Node Disk / Node Threads	per-node load, disk and thread breakdowns
Data Shards	shard state and distribution
Data ILM	index lifecycle state (the retention machinery)
Data License	license consumption
Collect Overview / Collect Metric	the collection pipeline’s own health (agent metrics and logs)
Netflow	netflow collector, data and lag
Alert Config / Alert View	the console’s own alerting
Node Setting	per-node platform settings

Everything is the platform monitoring itself: the agents’ metrics and logs land in the _collector_*/_runner_*-style indexes, the cluster writes to its shards, and this app reads those back.

The node’s own resources

Every node writes its own resource series into the metrics_sondar index every 30 seconds, beside the indexing and merge statistics already there:

series	what it measures
sondar_jvm_heap_used_percent, sondar_jvm_heap_used_bytes, sondar_jvm_heap_max_bytes	the JVM heap
sondar_jvm_gc_collection_total, sondar_jvm_gc_collection_time_millis_total	garbage collection, tagged gc with the collector’s name
sondar_jvm_threads	live threads
sondar_process_cpu_percent, sondar_os_cpu_percent	CPU, for the process and for the machine
sondar_fs_used_percent, sondar_fs_total_bytes, sondar_fs_available_bytes	each data path, tagged path

They are read from the same source as the node view, so a chart built on them agrees with it. Every series carries a host tag, so a query can break them out per node:

mstats avg(sondar_jvm_heap_used_percent) as heap by host

Disk use is measured against available space, not free space: available is what the disk watermarks are judged on, so reserved blocks count as full here exactly as they do when the platform decides whether it can still write.

Rules that ship on

Two of the console’s alert rules are enabled on install, because each guards a condition that stops the platform:

• Disk health — fires as a node’s disk crosses the low and high watermarks, before the flood stage stops ingest.
• Heap health — fires when a node’s heap averages 85% or more over five minutes (P2), and 95% (P1). Heap rises and falls with every garbage collection, so a single high reading is normal; the five-minute average is what makes the rule quiet on a healthy node.

The others — host load, cluster health, license and collector errors — ship disabled; turn them on under Alert Config. A rule added in a later version arrives with the upgrade itself, in the state it ships in. The upgrade offers each shipped rule once: a rule you deleted stays deleted, and a rule you changed is left as you left it.