Middleware monitoring

Monitoring the middleware layer — message brokers, application servers, stream processors, service-discovery and config systems — with the middleware integration apps. Each app installs its collection templates, repos, dashboards and alerts, and the metric/log data lands in the app’s own indexes.

Start here

The path is the same for all twelve apps, and none of it is hand-written YAML:

Install the app from the catalog.
Enable the interface the app reads. This is the step that varies, and it is the one that fails: JMX via Jolokia (Kafka, ActiveMQ), a management plugin (RabbitMQ, Tomcat, PHP-FPM), the Metrics API (ZooKeeper 3.6+, Nacos), the monitor plugin (Fluentd), a Prometheus endpoint (etcd, CoreDNS), or the Docker daemon socket (Docker).
Run the collection template for the collect type that matches your setup, setting ${URL} / ${Server} / Endpoint to the real address.
Open the app’s dashboards — data should appear within one collection interval.

Step 2 is why the per-component sections below exist: each one names the interface that has to be turned on, and the endpoint the template defaults to. Read yours before creating the task.

How the middleware apps work

Each middleware app follows the same shape as the database apps:

collect types — the collection templates (each names a reader config and a repo);
repos — the indexes the data lands in (prefixed app_*);
dashboards — prebuilt views, installed with the app;
alert prototypes — recommended alerts (currently only Zookeeper ships them).

Installing the app from the catalog makes its collection templates available under Data Collection → Agents ▸ Collection Templates, grouped by the app; run the one that matches your setup and the metrics/logs begin flowing. Most apps offer a metric collect type and one or more log collect types.

Metrics are pulled on an interval (30s default) and land in the app’s metric repo; logs are tailed by the agent and land in the app’s log repo. The metric names are plain English (not localized keys), so Metric Explorer and the dashboards render friendly names directly.

Message brokers

Kafka

Collects broker metrics and server logs.

collect type	what it collects	reader
`kafka_tel`	broker and JVM metrics via JMX	`jolokia2_agent_tel`
`kafka_log`	Kafka server logs	`fileset` + parsing
`kafka_gc_log`	JVM GC logs	`fileset` + parsing

Metrics need Jolokia. The kafka_tel template reads Kafka’s JMX through the Jolokia agent (jolokia2_agent_tel): install the Jolokia JavaAgent beside the Kafka process, then point the collector at its HTTP endpoint (${URL}, e.g. http://localhost:8778/jolokia). A cluster_name tag (the template defaults to kafka) lets several clusters be collected into one repo and distinguished in queries. The template defines the JVM and broker MBeans explicitly — runtime, memory, GC, plus the broker metrics (under-replica partitions, request times, message counts).

Data lands in app_kafka_tel (metrics) and app_kafka_server_log (logs). Dashboards: Kafka Metrics, Kafka Logs.

The JMX gap. Kafka’s most common failure modes — consumer lag, partition leaders moving, offline replicas — are read from JMX on the broker, not from the consumer. The template’s broker MBeans cover the broker side; the per-consumer-group lag metric is what most operators reach for first and is not in the default metric list. Worth extending if consumer monitoring is a priority.

RabbitMQ

Collects broker metrics and server logs.

collect type	what it collects	reader
`rabbitmq_tel`	broker metrics via the management plugin	`rabbitmq_tel`
`rabbitmq_log`	RabbitMQ server logs	`fileset` + parsing

Metrics need the management plugin. The rabbitmq_tel template reads RabbitMQ’s management HTTP API (default http://localhost:15672). Do not leave the credentials at RabbitMQ’s guest/guest default — it is the broker’s own out-of-the-box account, it can read every queue and exchange the template polls, and RabbitMQ only restricts it to loopback. Give the collector its own monitoring user with the monitoring tag before pointing it at a broker anything else uses. It can collect per-nodes, per-exchanges, or everything; node and exchange lists restrict the poll when the broker is large. Data lands in app_rabbitmq_tel and app_rabbitmq_server_log. Dashboard: RabbitMQ Overview.

ActiveMQ

Collects broker metrics.

collect type	what it collects	reader
`activeMQ_tel`	broker metrics	`activemq_tel`

The activemq_tel template reads ActiveMQ’s JMX (activemq_tel reader). Data lands in app_activemq_tel. Dashboard: ActiveMQ Overview.

Stream processing and log shipping

Logstash

Collects Logstash pipeline metrics.

collect type	what it collects	reader
`logstash_tel`	pipeline/node metrics	`logstash_tel`

The logstash_tel template reads Logstash’s monitoring endpoint. Data lands in app_logstash_tel. Dashboard: Logstash Overview.

Fluentd

Collects Fluentd metrics.

collect type	what it collects	reader
`fluentd_tel`	Fluentd runtime metrics via its monitor plugin	`fluentd_tel`

The fluentd_tel template reads the metrics Fluentd exposes through its monitor plugin. Data lands in app_fluentd_tel. Dashboard: Fluentd Overview.

Application servers and runtimes

Tomcat

Collects server metrics and access/server logs.

collect type	what it collects	reader
`tomcat_tel`	servlet/connector metrics via the Status page	`tomcat_tel`
`tomcat_access_log`	HTTP access logs	`fileset` + parsing
`tomcat_server_log`	Tomcat server logs	`fileset` + parsing

Metrics need the Status page enabled. The tomcat_tel template reads Tomcat’s built-in Status servlet (http://localhost:8080/manager/status, or the ${URL} you configure; HTTP basic auth via username/password). Data lands in app_tomcat_tel, app_tomcat_access_log and app_tomcat_server_log. Dashboards: Tomcat Overview, Tomcat Access Logs.

PHP-FPM

Collects PHP-FPM pool statistics.

collect type	what it collects	reader
`phpfpm_tel`	FPM pool stats via the status page	`phpfpm_tel`

The phpfpm_tel template fetches PHP-FPM’s status page (enable pm.status_path in php-fpm.conf and set the URL, e.g. http://localhost/status?full). Data lands in app_phpfpm_tel. Dashboard: PHP-FPM Overview.

Service discovery and configuration

Nacos

Collects Nacos metrics, access/server logs, and GC logs.

collect type	what it collects	reader
`nacos_prom_tel`	Nacos metrics via its Prometheus endpoint	`prometheus_tel`
`nacos_access_log`	access logs	`fileset` + parsing
`nacos_server_log`	server logs	`fileset` + parsing
`nacos_gc_log`	JVM GC logs	`fileset` + parsing

Metrics need the Prometheus endpoint. The nacos_prom_tel template reads Nacos’ built-in Prometheus metrics (enable the metrics endpoint in Nacos and set ${URL}). Data lands in app_nacos_prom_tel, app_nacos_access_log and app_nacos_server_log. Dashboards: Nacos Metrics, Nacos Logs.

ZooKeeper

Collects metrics, audit logs, and server logs.

collect type	what it collects	reader
`zookeeper_prom`	metrics via ZooKeeper’s Metrics API (3.6+)	`prometheus_tel`
`zookeeper_mntr`	metrics via the `mntr` command (before 3.6)	`zookeeper_tel`
`zookeeper_audit_log`	audit logs	`fileset` + parsing
`zookeeper_server_log`	server logs	`fileset` + parsing

The two metric paths are version-gated. ZooKeeper gained a Prometheus Metrics API in 3.6: metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider with metricsProvider.httpPort=7000 in conf/zoo.cfg; use zookeeper_prom for that. Before 3.6 the API does not exist, so zookeeper_mntr polls the mntr command over the client port (default 2181). The metric list varies by ZooKeeper version — the shipped dashboards cover the common set (latency, connections, znodes, watches, leader election).

Data lands in app_zookeeper_prom / app_zookeeper_mntr, app_zookeeper_audit_log and app_zookeeper_server_log. Dashboards: ZooKeeper Overview, ZooKeeper Logs.

ZooKeeper is the only middleware app with alert prototypes — ten shipped alerts: too many connections, too many open files, too many znodes, too many watches, high avg response time, leader re-election, fsync too slow, snapshot too slow, znode memory too large, JVM high memory usage.

Container and cluster infrastructure

Beyond the classic middleware, the same catalog covers container-adjacent infrastructure:

Docker — container metrics (docker_tel) and container logs (app_docker_log_tel), both into app_docker_tel / app_docker_log_tel. One variable, Endpoint, defaulting to unix:///var/run/docker.sock — so the agent must run on the Docker host with access to that socket. This is not a Prometheus scrape. Dashboard: Docker Overview.
etcd — etcd_prom_tel scrapes etcd’s Prometheus endpoint on its client port, 2379 (default http://localhost:2379/metrics; https and a client certificate on a TLS cluster). Nothing to enable on etcd’s side. Into app_etcd_prom_tel. Dashboard: etcd Monitoring.
CoreDNS — coredns_prom_tel scrapes CoreDNS’s Prometheus endpoint on 9153, which requires the prometheus plugin in the Corefile. The default is the in-cluster address http://kube-dns.kube-system:9153/metrics, so on Kubernetes it usually works unchanged. Into app_coredns_prom_tel. Dashboard: CoreDNS Monitoring.

Product note. The two templates disagree on the field’s name: etcd calls it URL, CoreDNS calls it URl. Both default values are correct — etcd 2379, CoreDNS 9153.

For container logs at cluster scale, prefer the dedicated container readers in Log collection sources — container_stdout with autodiscover handles pod churn, which a single daemon endpoint does not.