Alerts from other tools

Running Sondar beside the monitoring you already have is the usual way in: keep Datadog, Dynatrace, New Relic or Elastic alerting while you move logs, metrics and traces over, and see every alert in one place. Each of the four has an app that does this:

App	Receives	Index
Datadog Alerts (datadog_event)	Datadog monitor notifications (webhook integration)	app_datadog_event
Dynatrace Alerts (dynatrace_event)	Dynatrace problem notifications (custom integration)	app_dynatrace_event
New Relic Alerts (newrelic_event)	New Relic workflow notifications (webhook destination)	app_newrelic_event
Elastic Alerts (elk_event)	Kibana alerting rules (Webhook connector); Watcher webhooks too	app_elk_event

Installing one gives you three things:

• An index holding one event per notification, normalized to the same fields whatever the vendor: status, target, metric, description, the event time, and the vendor’s own fields under attribute.*. The original payload is the raw event. Retention is 30 days by default.
• An alert rule, <Vendor> alerts, enabled on install. Every minute it takes the latest status of each vendor alert from the last 7 days and raises the open ones in Alerts → Alert Events: CRITICAL as P0, WARNING as P1. The Sondar alert resolves within about three minutes of the vendor reporting recovery, once more than one run has passed without it. It has no trigger actions, so installing the app never pages anyone; add one to the rule when you want to be notified.
• A Documentation page in the app with the exact webhook body for that vendor, which is the part you paste into the vendor’s UI.

How an alert gets in

vendor webhook --POST--> agent event listener (:10007) --> Sondar index --> alert rule --> Alert Events

1. The agent listens. Alerts arrive at an agent, not at the Sondar server. Turn its listener on in Data Collection → Agents: the agent’s ⋯ menu → Configuration → Custom Configuration → Event listener, or for every agent at once in Global Configuration. The port is 10007 by default.

2. A collection task receives them. In Data Collection → Data Sources, the Events tab lists the alert apps you have installed. Choose one, give it a Source name, and distribute it to the agent. The form shows the webhook URL for that name.

3. The vendor posts to the agent:

   http://<agent-host>:10007/event?event_source=<source name>&event_sourcetype=<app>

   with the body from the app’s Documentation page. The agent converts it and sends it to the app’s index with the agent’s collection token.

The listener answers 200 {} when a task accepted the alert, 400 unknown eventSource … when no running task has that source name, and 400 when event_sourcetype does not end in _event. Allow a minute after turning the listener on: tasks that were already on the agent subscribe on their next retry, and until then their alerts get that 400.

How vendor states map

Vendor sends	Sondar status	Sondar alert
Datadog: Recovered, or alert type success	OK	resolves
Datadog: Warn, No Data, or priority P3	WARNING	P1
Datadog: trigger with priority P1/P2, or with no priority and alert type error	CRITICAL	P0
Dynatrace: problem RESOLVED, CLOSED or MERGED	OK	resolves
Dynatrace: open AVAILABILITY or ERROR problem	CRITICAL	P0
Dynatrace: open PERFORMANCE, RESOURCE_CONTENTION or CUSTOM_ALERT problem	WARNING	P1
New Relic: issue CLOSED	OK	resolves
New Relic: priority CRITICAL	CRITICAL	P0
New Relic: priority HIGH or MEDIUM	WARNING	P1
Elastic: action group recovered	OK	resolves
Elastic: severity critical	CRITICAL	P0
Elastic: severity warning, or none	WARNING	P1

A vendor alert’s target and metric are the same on the notification that opens it and the one that closes it — Datadog’s monitor and group, Dynatrace’s problem, New Relic’s issue, Elastic’s rule and group — so a recovery resolves the Sondar alert it belongs to, and each vendor alert is one Sondar alert.

Searching them

Each app’s index is an ordinary index:

search2 repo="app_datadog_event" status="CRITICAL"
| stats count() as alerts by target

The shipped rule is an ordinary alert rule too. Copy it to change the window, add a trigger action, or alert only on some targets.

Security

By default the agent’s event listener accepts any request that reaches the port. Set an Event token to require a shared secret: Data Collection → Agents → ⋯ → Configuration → Custom settings → Event listener (or Global Configuration for every agent). Once set, each webhook must send it in one of two headers; anything else is refused with 401:

Authorization: Bearer <token>
X-Sondar-Event-Token: <token>

Datadog, Dynatrace, New Relic and Kibana can all add a custom header to their webhooks, and Alertmanager sends Authorization from its http_config. The token is not accepted as a query parameter, so it does not end up in proxy or access logs.

The listener also refuses a body larger than 10 MiB (413). It speaks plain HTTP, so keep the port on a private network or put a reverse proxy that terminates TLS in front of it.

Other event sources

The agent’s listener also converts Prometheus Alertmanager (prometheus_event), Zabbix (zabbix_event) and payloads already in the event shape (restful_event). These have no app yet, so you set up the three pieces an app would have given you. Using Alertmanager as the example:

1. Create the index first, in Configure → Data Management → Index → Create, for example alerts_prometheus. This step is not optional: the agent still answers Alertmanager 200 {} when the index is missing, then drops the alert with repo [alerts_prometheus] not found in its own log. It retries for a while, so some alerts arrive once you create the index, but not all of them.

2. Create the task in YAML. In Data Collection → Data Sources, open any template, switch the editor from Interactive to YAML (top right), choose the agents, then replace the configuration with:

   readers:
     - prometheus_event:
         event_source: alertmanager
   senders:
     - sondar:
         repo: alerts_prometheus
         sourcetype: json
         token: "${sondar_token}"

   json is a built-in source type. A sourcetype that does not exist fails the same way a missing index does.

3. Point Alertmanager’s webhook at http://<agent-host>:10007/event?event_source=alertmanager&event_sourcetype=prometheus_event.

Each alert in the notification becomes one event with the same normalized fields as the apps. The target is the alert’s labels (without alertname), the metric is alertname, and the severity label sets the status: critical, page or error is CRITICAL, info or none is INFO, anything else (including no severity label) is WARNING, and a resolved alert is OK. To raise them in Alert Events, copy one of the four shipped rules and change repo in its query to your index; the rest of the query works unchanged because the fields are the same.

Anything that can post JSON can also skip the agent and write to an index directly through Splunk HEC.

What changed on September 27, 2026

Before this date the four apps sent alerts to an incident-response service that Sondar does not ship, so none arrived. The Event Access page that configured them is gone, and /eventSource now opens Data Sources → Events. The agent’s sondar_event sender, which posted to that service, is no longer used by any app.