Skip to content
Sondar

Alerts and notifications

Alerts and notifications

Section titled “Alerts and notifications”

How Sondar evaluates alert rules, and how it tells someone.

The shape of an alert

Section titled “The shape of an alert”

An alert is a query, a schedule, a condition on the result, and one or more notifications.

condition: search2 repo="app_logs" ERROR | stats count() as error_count
           error_count > 100
schedule:  every 5 minutes
notify:    email → the on-call group

The query runs on its schedule; the condition is applied to its result; when the state changes the alert fires and an event lands in _alert_event.

Name the engine in the query. A condition that begins with neither search nor search2 is evaluated by whichever engine the global Default Search Engine setting names, and the two engines do not agree on every command — the current example is the transaction added-field naming (_event_count/_duration on search2, eventcount/duration_second on search1; both engines run a partitioned transaction since 15 Aug 2026); chart, mvcombine and mvexpand agree since 14 Aug 2026. An administrator changing that dropdown can therefore change what your alerts do. Of the seven alert conditions that ship with the product, one is a bare query; the rest are pinned or use command families the setting does not affect.

Rule types

Section titled “Rule types”

Seven are documented. Four work.

type state
Static threshold works — the common case, and what the example above is
Month-on-year comparison works
Log pattern recognition works, and does not depend on the ML runtime
Custom works
Anomaly detection cannot be built
Predictive cannot be built
Outlier anomaly detection cannot be built

The last three rest on the SonQL commands anomalies, forecast and outliers, and both engines reject all three — search2 cannot parse them, search1 answers do not support spl. This is not a missing Python runtime that could be assembled back in; the commands never reach the stage where a runtime would matter. Until that is restored or the rule types are withdrawn, treat the product as having four.

Notifications

Section titled “Notifications”

A notice is a named, reusable configuration — a channel plus its credentials — stored under Alert → Notification Methods (/alert/notice/way). A rule references notices by id. Notices are scoped per app and can be created for either alert category (event or incident).

Changed 13 Aug 2026 — the picker now offers the named channels US evaluations expect. Measured against the shipping code, not the 5 Aug version of this page. The per-user types are still Email, WeChat and DingTalk (the latter two are Chinese messaging platforms, delivered as User or Robot notices), but beside them the picker now lists, each with its own setup form and server-side method:

channel wire value notes
Slack slack incoming-webhook URL
PagerDuty pagerDuty Events API v2; dedup_key correlation (#102)
Microsoft Teams teams Workflows Adaptive Card
Opsgenie opsgenie alias correlation
ServiceNow serviceNow correlation_id; configurable resolvedState
Jira jira v3 ADF; recovery comments rather than transitioning
Discord discord answers 204 No Content
Google Chat googleChat webhook URL
Mattermost mattermost webhook URL
Amazon SNS sns SigV4; correlationKey
Kafka kafka inherited
Webhook / Script / Custom — generic destinations; payload is the customer’s to template

The named channels were shipped 11 Aug 2026: Slack and PagerDuty landed with #102 on 9 Aug but were a listing without a form — selectable, required server-side, with nowhere to type the URL. The missing form and the rest of the set followed. Each shipped channel is configurable in the UI, has a platform-shaped payload, the platform’s real success contract checked, and a recovery that closes rather than re-opens. Jira is the deliberate exception: it comments instead of transitioning because transition ids are per-project and not portable.

Credentials are write-only (#139). A notice’s secrets read back as a sentinel; an unchanged edit preserves the stored value, and an absent value deletes. Only the notice’s LIST permission is needed to read them, so this is what stops a read permission becoming credential disclosure.

Changed 13 Aug 2026. Two channels that used to ship, CiticWebHook and WasuWebHook — integrations with CITIC Bank’s and Wasu Media’s internal webhook platforms — were removed 6 Aug 2026 and are no longer offered to any customer. The DeployEnv.Citic / DeployEnv.Wasu deployment flags that gated an SMS channel are gone with them; the deploy enum is now only public and private. This page said “specified and scoped but not yet built (#102)” for Slack/PagerDuty through 11 Aug; that row was already wrong when written and is corrected here.

Still unverified by anyone: a single real delivery. Every payload matches the vendor’s published contract and is pinned by tests (47, including the published AWS SigV4 key-derivation vector); not one has been accepted by a live vendor endpoint. Closing that needs a live account per channel.

Aggregation

Section titled “Aggregation”

Aggregation rules group related alert events so that one incident does not produce fifty notifications. Rules are created, managed, enabled and disabled independently of the alerts they group, and the configuration can be exported.

This is the mechanism to reach for before turning an alert’s threshold up to quiet it: raising a threshold loses signal, grouping does not.

What was verified, and what was not

Section titled “What was verified, and what was not”

Verified running: creating an alert; forcing a real threshold crossing (normal → abnormal, with events landing in _alert_event); listing and inspecting the stored condition of all seven shipped alerts; the engine exposure of each; and that anomalies/forecast/outliers are rejected on both engines.

Verified in code, 13 Aug 2026: the notification picker’s channel list is server-driven (GET .../notice/types) and now names Slack, PagerDuty, Teams, Opsgenie, Discord, Google Chat, Mattermost, ServiceNow, Jira, Amazon SNS, Kafka, Webhook, Script and Custom; each named channel has a backend NoticeMethod and a UI form branch.

Test each channel before you rely on it. That an alert fires is independent of whether its message arrives: delivery depends on the credentials you supply for the target service. Send a test notification on every channel you configure.

Event review

Section titled “Event review”

An alert firing produces events in _alert_event. Alert → Events (/alert/events) is where they are worked.

Two views, switchable with a toggle at the top of the page:

  • Raw (/alert/events/raw) — one row per event.
  • Aggregate (/alert/events/aggregate, the default) — one row per group of related events, with a merge count column; the aggregate row expands into a drawer showing the events in the group. Alert → Overview offers the same two views read-only as raw/aggregate tabs.

An event has a status, and the status gates what can be done to it:

status meaning open to
created fired, not yet touched Confirm / Assign
assigned someone owns it Confirm / Assign / Process / Close
processing being handled Close
resolved recovered (or recovery confirmation recorded) — (closed)
closed done — (closed)

The list filter/search covers event id and name, target, app, level, status, business system and suppression state, and the assignee view can be switched between All and Own — the “own” view plus the created/assigned status filter is the on-call inbox. Column visibility is per-user through the gear.

Row operations and batch operations (same set, both available from the table):

  • Confirm — acknowledge the event without assigning it.
  • Assign — hand it to a user; the modal offers the same delivery methods as notices (email / WeChat / DingTalk) plus a template, and the assigned user is notified on those channels.
  • Process — mark it as being worked.
  • Close — finish it.

Selecting the event name opens a detail drawer (or a full page at /alert/events/raw/detail/:eventId) showing:

  • the alert preview — the rule’s condition view with the real value that triggered it, the query that produced it, and the alert range;
  • the event detail — the triggering records with per-metric normal/ abnormal status;
  • the timeline — trigger, notice, suppression, noticeError, assigned, confirm, closed, resolved, suggestion and notifyUpgrade entries with timestamps; the detail page inlines a recovery (“Resolve manually”) action that writes a resolved timeline entry;
  • the notices that fired, with any per-channel delivery errors, and a link back to each notice’s configuration;
  • drill-down querySpl links that open the triggering logs in Search.

Aggregate event groups

Section titled “Aggregate event groups”

Alert → Aggregate Rules (/alert/aggregate_rules) groups related events so that one incident does not produce fifty notifications — reach for this before raising an alert’s threshold to quiet it (raising a threshold loses signal, grouping does not).

An aggregate rule defines:

  • Group fields (groupFields) — the fields whose values identify the subject (host, service, …). A group’s subject is rendered as field=value, ….
  • A filter condition (filterCondition) — an AND/OR tree of judge groups (IJudgeGroup = OR of ANDs; each leaf is field + type + threshold/ value with gt/gte/lt/lte/eq/neq/match/mismatch/regex) that selects which events join a group.
  • An interval (value + h/m) — the aggregation window.
  • Trigger condition and the notices to fire when a group forms.

The Aggregate event list shows, per group, the merge count, the subject, the filter condition, the group’s time range and its level. Opening a group’s row runs the stored drill-down querySpl (start/end/queryString/fragment) to list the merged events, and the raw events behind a group can be worked individually.

Per-type walkthroughs

Section titled “Per-type walkthroughs”

Rules are created under Alert → Rules → Create, as a two-step wizard (/alert/rules/create/step1, step2). The type picker (the sceneType) determines what the condition form asks for. The picker offers eight types; the “seven documented, four work” table above counts seven because multi-rule was the extra one:

  • Log pattern recognition (pattern) — pick a repo/query and an event pattern; does not depend on the ML runtime. This is the log-relevant type this handbook is mostly about.
  • Metric alert (static) — a metric value compared against a threshold (the common case; the example at the top of this page).
  • Multi-rule alert (multiple) — combines other rules with AND/OR into one event (subRules). Not among the measured seven; the form supports it.
  • Ratio alert (ratio) — month-on-year comparison of a metric.
  • Outliers / Anomaly / Forecast (outliers/anomalies/forecast) — present in the picker but cannot be built: the commands both engines reject, as described under Rule types above. Do not promise one.
  • Custom (custom) — the query author writes the SonQL (data source spl or a saved dataset) and the condition by hand; the engine note above applies in full.

Every type shares the same supporting blocks: frequency (a cron or a cron-date schedule), effective duration (daily / working day / non-working day / trading day / non-trading day), the alert range (relative start/end), levels with per-level judge groups, event name/description templates (rich text or plain), no-data handling, recovery (by condition or after a period, optional), suppression (levels, window, event-threshold and notice-threshold, optionally reset on recovery), and the notices + optional escalations to fire.

Permission control

Section titled “Permission control”

Alert surfaces are gated by role functions:

function grants
alert_event_view Alert → Overview (read-only raw/aggregate review)
alert_event_own Alert → Events — the working surface (confirm / assign / process / close), all routes under /alert/events
alert_rules Alert → Rules
aggregation_rules Alert → Aggregate Rules
alert_shield Alert → Shield (mask notifications for an app or rule for a period)
alert_prototype Alert → Prototype (rule templates)
alert_notices Alert → Notification Methods (/way), with per-object operations
alert_notice_upgrade escalations
alert_templates / alert_level / alert_user_group / alert_notice_user the supporting notice configuration pages

Row-level actions additionally check the object’s own operations (e.g. only an owner or an admin can assign a created/assigned event). Route guards live in modules/Alert/index.tsx; the function ids are in @sondar/app-sdk/apis/permission/role/model.ts. The full model — functions, roles, operations and the limits a role can carry — is in roles-and-permissions.md.