Exporting log data

How to get log data out of Sondar and into somewhere else — object storage, another index, a message bus, a database, or an HTTP endpoint.

---

The shape of an export

An export is a search with | export on the end. The search selects the data; export says where it goes and in what shape.

search2 repo="app_logs" ERROR
| export type="json" sink="s3" bucketName="acme-log-archive" region="us-west-2"
    accessKey="..." secretKey="..." prefix="app_logs/errors"

Two parameters matter everywhere:

• sink — the destination. One of s3, es, kafka, mysql, http, repo.
• type — the serialization format. One of json, csv, raw.

raw writes the original log line and nothing else, which is what you want for archival: it is the smallest, and it is what a future reader can parse without knowing Sondar’s schema. csv is what a spreadsheet or a BI tool wants. json is the default choice for another machine.

What each row looks like:

• json — one JSON object per event, one per line (NDJSON). A multi-value field is a JSON array holding every value.
• csv — one RFC 4180 row per event, with no header row: a streaming export writes the first event before it has seen the rest, so it cannot know every column up front. Columns follow the search’s field order, so put | fields host, status, duration before export to fix them and to give every row the same columns. A cell holding a comma, a quote or a line break is quoted; a missing value is an empty cell; a multi-value field is one cell with its values on separate lines.

Export covers one hour unless you say otherwise. export runs over the search’s time range, and a saved export task with no explicit range gets the last hour. Set it on the time picker, or in the query: start="-3h" end="-2h" repo="app_logs" | export …

---

Destinations

s3 — archive to object storage

The one to reach for when the question is “keep these logs cheaply for a year”. Works with Amazon S3 and any S3-compatible store — MinIO, Ceph, R2 — via endpoint.

parameter	what it is
bucketName	target bucket — required
region	required, e.g. us-west-2
accessKey, secretKey	credentials — required unless a named client is used
endpoint	for S3-compatible stores; omit for AWS
prefix	required — key prefix, so exports land under a path rather than the bucket root
timeSuffix	appends a time-based suffix to the key, which is what makes the objects partition sensibly by date
maxLines, maxSize, maxPartitions	roll to a new object at a line count, a byte size, or across partitions
compress, compressType	compress before upload
splitter	record separator

bucketName, region, accessKey, secretKey and prefix are all validated before the export runs; omitting any of them fails the search with Export to s3 error: <name> is null or empty. prefix is required even though it reads like a convenience — an export with no prefix is rejected.

search2 repo="app_logs"
| export type="raw" sink="s3"
    bucketName="acme-log-archive" region="us-west-2"
    prefix="app_logs/" timeSuffix="true"
    compress="true" maxSize="268435456"

repo — copy into another Sondar index

Index-to-index. This is how you build a downsampled rollup, or a sanitized copy of a sensitive index that a wider audience may read.

parameter	what it is
repo	destination index
sourcetype	source type to write the copies under
indexed	whether extracted fields are materialized at write time — see the field-acceleration note in indexes-and-field-extraction.md

kafka — publish to a topic

parameter	what it is
host	broker list
topic	destination topic
client	named client configuration

es — write to Elasticsearch

parameter	what it is
host	cluster address
version	target major version; the document shape differs across them
index	destination index
indextype	mapping type, for versions that still have one

A rejected document fails the export. Elasticsearch’s _bulk answers 200 even when it rejects documents; the export reads the per-document results and fails with the count rejected and the first reason, rather than reporting success.

mysql — insert into a table

parameter	what it is
host	a JDBC URL, jdbc:mysql://<server>:3306/<database> — not a bare hostname
userName, password	the database login; password must be given, and may be ""
table	destination table, which must already exist
client	named client configuration, used instead of the three above

Rows are matched to the table by column name, so the table’s column order does not matter; a table column the rows do not carry is written as NULL.

http — POST to an endpoint

The escape hatch. Anything that accepts an HTTP POST — a webhook, a queue gateway, your own service.

parameter	what it is
url	endpoint
header	additional request headers

Large exports are sent in batches; if any batch fails, the export fails and names the first failure.

HTTPS is verified. For http and es, the receiver’s certificate must be trusted by the server’s Java trust store and must name the host in the URL. For a self-signed receiver, see TLS for alert delivery: the same trust store and the same node-wide opt-out apply to exports.

---

Scheduled exports

An export can run on a schedule instead of on demand: Settings → Analysis & Tools → Export.

• Frequency by preset (minute, hour, day, week, month, quarter, year) with a specific send time, or a crontab expression for anything else.
• Tasks can be edited, started, stopped and deleted, and their run history shows start time, duration and success or failure.
• Configuration imports and exports as JSON, so an export task can live in version control.

You can also build one from a search you already have: run the query, then Save as → Export task, and complete the export clause in the dialog.

“Instant” means run once, not run now. A task set to run instantly executes a single time on its schedule. It is not a “run this immediately” button, and reading it that way has cost time before.

---

What was verified, and what was not

Verified: the destination and format registries, and every parameter name above, read from ExportFactoryRegister, each *ExportConfig, and each *Factory.

One destination is now verified end to end: repo. On 3 Sep 2026, three rows were exported into a repo and then read back out of it —

makeresults count=3 | eval p43="marker9f3a"
| export sink="repo" repo="default" indexed=true

— and search2 repo="default" | where p43 == "marker9f3a" returned exactly those three. So the export pipeline itself works: rows leave a query, reach a sink, and are queryable afterwards. Two details from that run worth having: sink selects the destination (it defaults to repo) while type selects the serialization, which is easy to mix up — passing type="repo" fails with “no such serialization: repo”. And the exported rows were found by a field query; a bare full-text search for the same value did not match them.

A second destination is verified end to end, and this one leaves the product: http. On 4 Sep 2026 three rows were exported to a local receiver —

makeresults count=3 | eval p43="marker4sep"
| export sink="http" type="json" url="http://<your-host>:8097/sink"

— and the receiver logged one POST /sink, 63 bytes, carrying exactly the three rows as newline-delimited JSON. So “the task ran” and “the bytes arrived” are now both established for http, not just for repo.

Two details from that run that will cost you time otherwise:

• type is required for http. sink="http" without it fails with “export: type is missing”. repo does not need it, so copying the repo example and swapping the sink does not work.
• The request arrives as text/plain; charset=UTF-8, even with type="json", and the body is newline-delimited JSON objects rather than a JSON array. A receiver that requires application/json will reject it.

A third destination is verified end to end: mysql. On 26 Sep 2026 three rows were exported to a MySQL 8.0 server —

makeresults count=3 | eval marker="p43mysql26sep", host="h-probe", n=43
| fields host, n, marker
| export sink="mysql" host="jdbc:mysql://<server>:3306/probe"
    userName="<user>" password="<password>" table="zz_export_p43"

— the search returned total: 3, and SELECT on the table returned exactly those three rows. Two failure cases were run as well: a table that does not exist, and a server that cannot be reached. Both wrote nothing and reported why, in the job’s failed field (“Table ‘probe.zz_no_such_table’ doesn’t exist”, “No route to host”). So a bad destination does not pass for a successful export.

Run the one you depend on before you depend on it. Delivery to s3, kafka and es still depends on credentials and network reach that only your environment has, and none of those three has been confirmed against a live endpoint. “The task ran” and “the bytes arrived in your bucket” are different claims; for those three, confirm the second one yourself.