Exporting log data

Exporting log data

How to get log data out of Sondar and into somewhere else — object storage, another index, a message bus, a database, or an HTTP endpoint.

---

The shape of an export

An export is a search with | export on the end. The search selects the data; export says where it goes and in what shape.

search2 repo="app_logs" ERROR
| export type="json" sink="s3" bucketName="acme-log-archive" region="us-west-2"
    accessKey="..." secretKey="..." prefix="app_logs/errors"

Two parameters matter everywhere:

• sink — the destination. One of s3, es, kafka, mysql, http, repo.
• type — the serialisation format. One of json, csv, raw.

raw writes the original log line and nothing else, which is what you want for archival: it is the smallest, and it is what a future reader can parse without knowing Sondar’s schema. csv is what a spreadsheet or a BI tool wants. json is the default choice for another machine.

Export covers one hour unless you say otherwise. export runs over the search’s time range, and a saved export task with no explicit range gets the last hour. Set it on the time picker, or in the query: start="-3h" end="-2h" repo="app_logs" | export …

---

Destinations

s3 — archive to object storage

The one to reach for when the question is “keep these logs cheaply for a year”. Works with Amazon S3 and any S3-compatible store — MinIO, Ceph, R2 — via endpoint.

parameter	what it is
bucketName	target bucket — required
region	required, e.g. us-west-2
accessKey, secretKey	credentials — required unless a named client is used
endpoint	for S3-compatible stores; omit for AWS
prefix	required — key prefix, so exports land under a path rather than the bucket root
timeSuffix	appends a time-based suffix to the key, which is what makes the objects partition sensibly by date
maxLines, maxSize, maxPartitions	roll to a new object at a line count, a byte size, or across partitions
compress, compressType	compress before upload
splitter	record separator

bucketName, region, accessKey, secretKey and prefix are all validated before the export runs; omitting any of them fails the search with Export to s3 error: <name> is null or empty. prefix is required even though it reads like a convenience — an export with no prefix is rejected.

search2 repo="app_logs"
| export type="raw" sink="s3"
    bucketName="acme-log-archive" region="us-west-2"
    prefix="app_logs/" timeSuffix="true"
    compress="true" maxSize="268435456"

repo — copy into another Sondar index

Index-to-index. This is how you build a downsampled rollup, or a sanitised copy of a sensitive index that a wider audience may read.

parameter	what it is
repo	destination index
sourcetype	source type to write the copies under
indexed	whether extracted fields are materialised at write time — see the field-acceleration note in indexes-and-field-extraction.md

kafka — publish to a topic

parameter	what it is
host	broker list
topic	destination topic
client	named client configuration

es — write to Elasticsearch

parameter	what it is
host	cluster address
version	target major version; the document shape differs across them
index	destination index
indextype	mapping type, for versions that still have one

mysql — insert into a table

parameter	what it is
host, userName, password	connection
table	destination table
client	named client configuration

http — POST to an endpoint

The escape hatch. Anything that accepts an HTTP POST — a webhook, a queue gateway, your own service.

parameter	what it is
url	endpoint
header	additional request headers

---

Scheduled exports

An export can run on a schedule instead of on demand: Settings → Analysis & Tools → Export.

• Frequency by preset (minute, hour, day, week, month, quarter, year) with a specific send time, or a crontab expression for anything else.
• Tasks can be edited, started, stopped and deleted, and their run history shows start time, duration and success or failure.
• Configuration imports and exports as JSON, so an export task can live in version control.

You can also build one from a search you already have: run the query, then Save as → Export task, and complete the export clause in the dialog.

“Instant” means run once, not run now. A task set to run instantly executes a single time on its schedule. It is not a “run this immediately” button, and reading it that way has cost time before.

---

What was verified, and what was not

Verified: the destination and format registries, and every parameter name above, read from ExportFactoryRegister, each *ExportConfig, and each *Factory.

Run one export end to end before you depend on it. The parameters below are the ones the product parses, and the examples are syntactically correct — but delivery to s3, kafka, mysql and es depends on credentials and network reach that only your environment has. “The task ran” and “the bytes arrived in your bucket” are different claims; confirm the second one yourself, per destination.