Middleware monitoring

Monitoring the middleware layer — message brokers, application servers, stream processors, service-discovery and config systems — with the middleware integration apps. Each app installs its collection templates, repos, dashboards and alerts, and the metric/log data lands in the app’s own indexes.

How the middleware apps work

Each middleware app follows the same shape as the database apps:

collect types — the collection templates (each names a reader config and a repo);
repos — the indexes the data lands in (prefixed app_*);
dashboards — prebuilt views, installed with the app;
alert prototypes — recommended alerts (currently only Zookeeper ships them).

Installing the app from the catalog makes its collection templates available under Configure → Data Collection → Collection Templates, grouped by the app; run the one that matches your setup and the metrics/logs begin flowing. Most apps offer a metric collect type and one or more log collect types.

Metrics are pulled on an interval (30s default) and land in the app’s metric repo; logs are tailed by the agent and land in the app’s log repo. The metric names are plain English (not localised keys), so Metric Explorer and the dashboards render friendly names directly.

Message brokers

Kafka

Collects broker metrics and server logs.

collect type	what it collects	reader
`kafka_tel`	broker and JVM metrics via JMX	`jolokia2_agent_tel`
`kafka_log`	Kafka server logs	`fileset` + parsing
`kafka_gc_log`	JVM GC logs	`fileset` + parsing

Metrics need Jolokia. The kafka_tel template reads Kafka’s JMX through the Jolokia agent (jolokia2_agent_tel): install the Jolokia JavaAgent beside the Kafka process, then point the collector at its HTTP endpoint (${URL}, e.g. http://localhost:8778/jolokia). A cluster_name tag (the template defaults to kafka) lets several clusters be collected into one repo and distinguished in queries. The template defines the JVM and broker MBeans explicitly — runtime, memory, GC, plus the broker metrics (under-replica partitions, request times, message counts).

Data lands in app_kafka_tel (metrics) and app_kafka_server_log (logs). Dashboards: Kafka Metrics, Kafka Logs.

The JMX gap. Kafka’s most common failure modes — consumer lag, partition leaders moving, offline replicas — are read from JMX on the broker, not from the consumer. The template’s broker MBeans cover the broker side; the per-consumer-group lag metric is what most operators reach for first and is not in the default metric list. Worth extending if consumer monitoring is a priority.

RabbitMQ

Collects broker metrics and server logs.

collect type	what it collects	reader
`rabbitmq_tel`	broker metrics via the management plugin	`rabbitmq_tel`
`rabbitmq_log`	RabbitMQ server logs	`fileset` + parsing

Metrics need the management plugin. The rabbitmq_tel template reads RabbitMQ’s management HTTP API (default http://localhost:15672, guest/guest). It can collect per-nodes, per-exchanges, or everything; node and exchange lists restrict the poll when the broker is large. Data lands in app_rabbitmq_tel and app_rabbitmq_server_log. Dashboard: RabbitMQ Overview.

ActiveMQ

Collects broker metrics.

collect type	what it collects	reader
`activeMQ_tel`	broker metrics	`activemq_tel`

The activemq_tel template reads ActiveMQ’s JMX (activemq_tel reader). Data lands in app_activemq_tel. Dashboard: ActiveMQ Overview.

Stream processing and log shipping

Logstash

Collects Logstash pipeline metrics.

collect type	what it collects	reader
`logstash_tel`	pipeline/node metrics	`logstash_tel`

The logstash_tel template reads Logstash’s monitoring endpoint. Data lands in app_logstash_tel. Dashboard: Logstash Overview.

Fluentd

Collects Fluentd metrics.

collect type	what it collects	reader
`fluentd_tel`	Fluentd runtime metrics via its monitor plugin	`fluentd_tel`

The fluentd_tel template reads the metrics Fluentd exposes through its monitor plugin. Data lands in app_fluentd_tel. Dashboard: Fluentd Overview.

Application servers and runtimes

Tomcat

Collects server metrics and access/server logs.

collect type	what it collects	reader
`tomcat_tel`	servlet/connector metrics via the Status page	`tomcat_tel`
`tomcat_access_log`	HTTP access logs	`fileset` + parsing
`tomcat_server_log`	Tomcat server logs	`fileset` + parsing

Metrics need the Status page enabled. The tomcat_tel template reads Tomcat’s built-in Status servlet (http://localhost:8080/manager/status, or the ${URL} you configure; HTTP basic auth via username/password). Data lands in app_tomcat_tel, app_tomcat_access_log and app_tomcat_server_log. Dashboards: Tomcat Overview, Tomcat Access Logs.

PHP-FPM

Collects PHP-FPM pool statistics.

collect type	what it collects	reader
`phpfpm_tel`	FPM pool stats via the status page	`phpfpm_tel`

The phpfpm_tel template fetches PHP-FPM’s status page (enable pm.status_path in php-fpm.conf and set the URL, e.g. http://localhost/status?full). Data lands in app_phpfpm_tel. Dashboard: PHP-FPM Overview.

Service discovery and configuration

Nacos

Collects Nacos metrics, access/server logs, and GC logs.

collect type	what it collects	reader
`nacos_prom_tel`	Nacos metrics via its Prometheus endpoint	`prometheus_tel`
`nacos_access_log`	access logs	`fileset` + parsing
`nacos_server_log`	server logs	`fileset` + parsing
`nacos_gc_log`	JVM GC logs	`fileset` + parsing

Metrics need the Prometheus endpoint. The nacos_prom_tel template reads Nacos’ built-in Prometheus metrics (enable the metrics endpoint in Nacos and set ${URL}). Data lands in app_nacos_prom_tel, app_nacos_access_log and app_nacos_server_log. Dashboards: Nacos Metrics, Nacos Logs.

ZooKeeper

Collects metrics, audit logs, and server logs.

collect type	what it collects	reader
`zookeeper_prom`	metrics via ZooKeeper’s Metrics API (3.6+)	`prometheus_tel`
`zookeeper_mntr`	metrics via the `mntr` command (before 3.6)	`zookeeper_tel`
`zookeeper_audit_log`	audit logs	`fileset` + parsing
`zookeeper_server_log`	server logs	`fileset` + parsing

The two metric paths are version-gated. ZooKeeper gained a Prometheus Metrics API in 3.6: metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider with metricsProvider.httpPort=7000 in conf/zoo.cfg; use zookeeper_prom for that. Before 3.6 the API does not exist, so zookeeper_mntr polls the mntr command over the client port (default 2181). The metric list varies by ZooKeeper version — the shipped dashboards cover the common set (latency, connections, znodes, watches, leader election).

Data lands in app_zookeeper_prom / app_zookeeper_mntr, app_zookeeper_audit_log and app_zookeeper_server_log. Dashboards: ZooKeeper Overview, ZooKeeper Logs.

ZooKeeper is the only middleware app with alert prototypes — ten shipped alerts: too many connections, too many open files, too many znodes, too many watches, high avg response time, leader re-election, fsync too slow, snapshot too slow, znode memory too large, JVM high memory usage.

Container and cluster infrastructure

Beyond the classic middleware, the same catalog covers container-adjacent infrastructure:

Docker — container metrics (docker_tel) and container logs (app_docker_log_tel). Dashboard: Docker Overview.
etcd — etcd metrics via its Prometheus endpoint. Dashboard: etcd Monitoring.
CoreDNS — CoreDNS metrics via its Prometheus endpoint. Dashboard: CoreDNS Monitoring.

These use the standard Prometheus scrape pattern (see database-data-sources.md for the shared prometheus_tel reader and how the repos are named).

Setup checklist

Install the app from the catalog.
Enable the source the app reads — this is the step that varies: JMX via Jolokia (Kafka, ActiveMQ), management plugin (RabbitMQ, Tomcat, PHP-FPM), Metrics API (ZooKeeper 3.6+, Nacos), monitor plugin (Fluentd), or a Prometheus endpoint (etcd, CoreDNS, Docker).
Run the collection template for the collect type that matches your setup, setting ${URL} / ${Server} to the real endpoint.
Open the app’s dashboards — the metrics and logs should appear within a collection interval.