Event collection

Event collection

Turning alerts and changes from other monitoring systems into Sondar events — the feed that drives Incident Response. An external system’s alert is received, converted to the standard event format, and sent to the event endpoint, where it becomes an event that can trigger incident logic.

---

How an event gets in

Two steps, and they are separate products:

1. The agent listens. The agent runs an HTTP event server on port 10007 (/event and /change). A monitoring system POSTs its alert to that listener.
2. A collection task consumes it. A task with an event reader subscribes to a source type. The reader converts each incoming alert to the standard event format and hands it to the senders, which forward it to the server’s event API (sondar_event sender → /api/v1/neuralert-dc/_sondar_event, or _sondar_change for change events).

So the pieces of a working event flow are: an event source configured in the product (Event → Event Access), an agent with its event server enabled, a collection task with an event reader and an event sender, and the external system told to POST to the agent’s listener URL.

The listener URL

Each event source has one. Its shape is fixed by the agent:

http://<agent>:10007/event?event_source=<source name>&event_sourcetype=<type>

For change events the path is /change instead of /event. The agent answers 200 {} when a subscriber accepted the payload, 400 when the body is empty or the event_sourcetype does not end in the expected suffix, and 400 unknown eventSource ... when nothing subscribed. The source name and type must match a running collection task or the agent refuses the POST — the external system gets a 400 rather than a silent drop.

The event server must be enabled

The agent’s event_server block defaults to disabled (enabled: false, port 10007, read/write timeouts 1m). An event collection task cannot subscribe until it is on:

event_server:
  enabled: true
  port: 10007

With it off, an event reader fails to start with agent event server is disabled.

---

The standard event format

Every source is converted to the same shape before it leaves the agent — the RawEvent struct below, which also carries the alert-signature fields the incident engine uses. Required: status, timestamp, target.

{
  "status": "CRITICAL",
  "target": "my-computer",
  "targetType": "host",
  "description": "CPU usage above 90%",
  "timestamp": 1723500000000,
  "host": "db-1",
  "metric": "cpu_usage",
  "metricKey": "system.cpu.usage",
  "normalizedTags": {},
  "customTags": {},
  "customAiTags": {},
  "attribute": {},
  "raw": "<original payload>",
  "eventSource": "datadog",
  "eventSourcetype": "datadog_event"
}

• status — one of NONE, WARNING, CRITICAL, OK, RESOLVED, MANUAL_RESOLVED, INFO.
• target — the alerting/monitored object’s id or name; the unique key.
• timestamp — 13-digit Unix milliseconds.
• metric — the monitored item; target + metric is the unique alert signature — the pair the incident engine de-duplicates and groups on.
• metricKey — the metric type identity.
• normalizedTags / customTags / customAiTags / attribute — structured metadata, the last used by the incident algorithms.
• raw — the original payload, kept for drill-down.
• eventSource / eventSourcetype — which source produced it.

Change events are a different shape: changeId, status, target (an array), description, planStartTime/planEndTime, startTime/ endTime, affectedConfig. The valid status values differ too: CANCELED, PLANNING, INSTRUMENTING, FINISHED.

---

The sources

Thirteen converters are registered in the agent. Four ship as installable apps (they appear in the app catalog and add a template to Event → Event Access); the rest are built in and configured directly.

Installable apps (the shipping four)

These are the coexistence story: Sondar consumes alerts from the monitoring platform a prospect already runs, so the two can operate side by side rather than a rip-and-replace.

app	event source type	what it converts
datadog_event	datadog_event	Datadog alert events
dynatrace_event	dynatrace_event	Dynatrace alert events
elk_event	elk_event	Elastic/ELK alert events
newrelic_event	newrelic_event	New Relic alert events

Installing the app adds its collection template to Event Access. Each template’s form asks for a data source name and offers a listen address button that fills in the exact listener URL for that source type — for Datadog, …/event?event_source=<name>&event_sourcetype=datadog_event. Point the vendor’s webhook at that URL.

The other four event apps — aliyun_event, cloudwise_event, emcc_event, huawei_event (Alibaba Cloud, Cloudwise, Huawei, and the EMCC China-market integrations) — are in the hidden list and do not ship. Their converters still compile into the agent but there is no app, template, or UI to reach them.

Built in

These are not apps; a collection task names the reader directly.

reader	what it converts
restful_event	a generic REST payload that is already in the standard format
restful_change	generic change events (the /change endpoint)
prometheus_event	Prometheus alertmanager webhooks (the alertname label becomes the metric)
zabbix_event	Zabbix triggers and resolves
sondar_event	Sondar’s own alerts (see Alerts)

A minimal restful event task:

event_server:
  enabled: true

runners:
  - name: event_ingest
    readers:
      - restful_event:
          event_source: my-source
    senders:
      - sondar_event:
          event_source: my-source
          event_sourcetype: restful_event

The event_source on the reader must match the event_source query param the external system sends, and the sondar_event sender stamps the same event_source/event_sourcetype on the request to the server’s event API. A reader and sender pair that disagree on these values produces events that arrive mislabeled.

The sondar_event sender

Modes select the server endpoint: event (default) → /_sondar_event, change → /_sondar_change, history → /ace/training/data (historical event training for incident grouping). Defaults: timeout 30s, max_idle_conns_per_host 2. If url is unset it uses the agent’s configured server URL and appends the mode’s path.

---

Product surface: Event Access

Event → Event Access (/eventSource) is the management UI. It lists configured data sources with name, type, status, latest data time; supports create, edit, enable/disable, delete, and import configuration file.

Creating a source is a four-step wizard:

1. Select data source — pick a template from the app’s event/change-event collection types (this is where installing datadog_event etc. shows up).
2. Fill in the data source information — name, description, the listen URL, and two time controls: alert auto-off time and data maximum receiving time (how stale a “no data” verdict is allowed to be).
3. Parsing transformation data — preview live sample data and define the mapping from the vendor payload to standard fields. The field table lets you set each field’s value via: direct mapping, field value mapping, regex extraction, field composition, JSONPath mapping, or custom SonQL mapping, and shows per-field detection results (Exist/Invalid/Legitimate/Does Not Exist).
4. Identify and add data sources — confirm and save.

The wizard validates the reserved semantics: timestamp must be 13 digits, status must be one of the standard values (change events get their own allowed set), and affected-config fields must be string arrays.