External data and lookups

Enrich search results with data from outside the index — a reference CSV, an external database, a key-value store, a geospatial dataset, or a script. A lookup joins an external source onto search results on a shared key, the way a SQL LEFT JOIN enriches a row.

This is the platform’s “external data” surface, reached from three Configure entries: External Connection, Lookup Tables and External Database.

Lookup tables (files)

A lookup table is an uploaded CSV — the reference data you want to join. Under Look Up File, upload the CSV, name the table, set the field separator, and declare its fields. The table is then available to lookup rules.

Lookup rules

A lookup rule maps search-result fields onto table fields and copies selected columns back into the results. Rules are of several types:

File — join against an uploaded CSV table.
External — join against an external database connection (see below).
Kvstore — join against an app’s key-value store.
Geospatial — enrich with location data.
Command — run a Python script (placed under the app’s bins/) to compute the enrichment.

Two behaviors make a rule “automatic” or time-aware:

Automatic lookup — apply the rule automatically to a sourcetype: declare the input fields (in the search result), the output fields (from the table), and whether to rewrite (overwrite an existing field’s value with the table’s).
Time-based lookup — a time-series lookup: match on a time field, with a time format and a shift window, for correlating events against a timeline.

In queries, the lookup SonQL command and the lookup_sql / lookup_sondardb transformers perform the same join at other stages — see SonQL and Transformers.

Which engine applies what

lookup in a query runs on both engines. On search2 (the default) it takes the first matching table row per event; search1 returns every match.
Automatic lookups run on both engines, and only when the search names exactly one sourcetype (sourcetype="nginx"). A search over several sourcetypes, or none, is not enriched. A rule that cannot be applied (its table deleted, a field renamed) is skipped rather than failing the search.
An automatic lookup needs a sourcetype and at least one input field. The event field may be left blank; it then defaults to the table field’s name. A rule missing either is refused when saved.
Settings that are off stay off. With Advanced settings disabled, the minimum and maximum match counts are ignored; with Time-based lookup disabled, so are the time shifts.

Measured September 2026: a table mapping one host to a team enriched 81 of 81 events by lookup "<rule>", and the same rule applied automatically to the 47 events of its sourcetype and to none of the 34 of another — the same counts on search1 and search2. Before that month, a rule saved with Advanced off and a maximum of 0 matched nothing, by name or automatically.

External databases

Configure → External Database connects Sondar to a relational database so its tables can be queried and joined:

Connection — name a connection, pick the type and driver, set the database’s own host and port (MySQL 3306, PostgreSQL 5432, and so on), credentials, and an optional row limit. These are the database’s coordinates, not Sondar’s — Sondar’s own gRPC port, 9500, is not involved in a database connection and does not belong in this form.
Driver — the installed JDBC drivers available for connections.
Query — run SQL over a connection to see its data and build lookups.

An external connection is what an External lookup rule and the lookup_sql transformer point at.

HTTPS connections are verified. A Prometheus, Elasticsearch or HTTP connection checks the server’s certificate against the Java trust store and checks that it names the host you entered — for its queries and for the connection test alike. For a server with a self-signed certificate, turn on Skip TLS verification on that connection only. Anyone on the network path to that server could then read what is sent to it, including the connection’s credentials. The server logs a warning the first time a connection skips the check.

External connections

The External Connect surface is the same connection mechanism for services other than databases — a named endpoint the platform can call out to.

A saved credential never comes back out. Passwords and keys are stored encrypted, the API returns them as __sondar_secret_set__, and the edit form shows an empty box marked Saved. Leave blank to keep it. — leave it empty to keep the saved value, or type a new one. Changing the address keeps nothing: if you change a connection’s endpoint, URL or brokers, enter its credentials again, so a saved password cannot be pointed at a different server. The same goes for turning on Skip TLS verification: the server it reaches is then no longer checked, so the password must be typed again.

Saved passwords were re-encrypted in October 2026. The first start of that release moves every saved connection password to a new key, so a copy of an encrypted password taken from an older version no longer works. Saved connections keep working. A browser tab left open across the upgrade still holds the old key: reload the page before saving a password. A credential that was stored by an older build in plain text no longer works; enter it again and save.

Transformers — lookup_sql / lookup_sondardb do the same join in the collection pipeline.
SonQL — the lookup command at search time.