Indexes, source types and field extraction

How Sondar stores log data and how raw lines become queryable fields.

Indexes

An index is where log events live. Internally, and in some API payloads, it is called a repo — the two words mean the same thing.

Every event carries repo (its index), sourcetype (its declared format), host, origin, _time, _indexTime and _raw. Everything else has to be parsed out.

Naming

Lowercase letters, digits and underscores. An index name is permanent — plan it before you send data, because moving events between indexes means re-ingesting them.

Name by retention and access, not by service. app_logs_30d and audit_logs_7y are useful names because they encode the two things that actually differ; checkout_service is not, because you will end up wanting the same retention for twenty services and a different one for the auditor.

Lifecycle

Each index has a retention policy — by age, by size, or both. When an index passes its limit the oldest data is removed. Retention is the main lever on storage cost, and it is worth setting deliberately at creation: a default that keeps everything forever is expensive, and one that keeps a week will lose the incident you are asked about a month later.

Cold data can be archived to S3-compatible storage and still be queried, at higher latency. Configure the archive destination first, then point the index at it.

Retention counts from event time

Retention removes an index’s data in whole parts — the part being written rolls over into a new one as it grows — and judges each rolled-over part by the newest event time in it, as Splunk ages its buckets. Two things follow:

Expired events can outlive their retention while they share a part with newer ones.
Old events imported into an index with a short retention are deleted once the part holding them rolls over. Logs from 2005 sent to an index that keeps 15 days are older than its retention the moment they arrive.

So that an import is not deleted before anyone notices, releases built from 18 Oct 2026 never remove a part by retention until 24 hours after it rolled over — after it stopped taking writes (the cluster setting sondar.ilm.delete.min_index_age; 0s removes the wait). And when ingest accepts events older than their index keeps, the server log says so, once per index every ten minutes, and so does the batch’s collect audit record (its warning field):

accepted events as old as 2005-11-09T12:00:00Z are older than repo [history]'s retention (15 days); ...

To import historical logs, create an index whose retention covers the oldest event, or that keeps everything, and send the import there. Deleting data when a disk fills (below) is separate and does not wait.

When the disk fills

Retention is a plan, and a disk can fill faster than the plan assumed. So each index also says what it gives up when a node’s disk passes the high watermark (90% used by default). Set it in the index’s settings, under When the disk fills:

Delete oldest data — the default for every index except the audit index. Sondar deletes this index’s oldest data first, one rolled-over part of the index at a time, and stops as soon as the node is back under the high watermark. It never touches the part the index is writing to. It checks once a minute, and each deletion is written to the server log and to the audit trail.
Stop and alert — keeps every record. If the disk is still over the high watermark once nothing else can be deleted, the server logs a warning naming the indexes that hold the data; at the flood stage (95%) writes to that node stop until space is freed. The audit log’s index, _audit, and _internal stop and alert by default, and _audit cannot be switched to delete.

Deleting to stay up is the right trade for logs and metrics, and the wrong one for an audit trail — which is why it is chosen per index. To change the behavior for the whole cluster, set sondar.disk.pressure.age_out.mode in sondar.yml (or as a container environment variable of the same name) and restart: dry_run logs what would be deleted and deletes nothing; off never deletes, so ingest stops at the flood stage as it did before this setting existed; delete is the default.

After upgrading from a release without this setting, every existing index has no choice recorded, so each one except _internal takes the default and deletes its oldest data under disk pressure. The server says so once when it starts, in a log line beginning disk-pressure: that names those indexes. To keep an index’s data instead, choose Stop and alert on it; to review first, start with dry_run.

When a write is confirmed

When Sondar answers a sender with HTTP 200, the sender treats those lines as delivered and moves on: the Sondar agent, like an HEC client, does not send a confirmed line again. So each index says when that answer is given. Set it in the index’s settings, under Write durability:

Durable — the default, and the only choice for the audit index, _audit. The answer comes only once the lines are on disk on every copy of the index, so a crashed or killed server loses nothing that was confirmed.
Fast — the answer comes first, and the lines reach disk within 5 seconds. A crash can lose the last few seconds of lines the sender was told were saved. In exchange, a server running at its ingest limit takes up to about 10% more data — measured on a 16 vCPU server, from no difference with 500-line requests to 7–15% with 100-line ones; below that limit there is no measurable difference. Use it only for data you can send again, such as a one-off import of old logs.

A change takes effect at once on the part of the index being written, with no rollover, and every later part keeps it.

After upgrading from an earlier release, every index is Durable. Earlier releases confirmed every write before saving it — the Fast behavior — without offering a choice; nothing needs configuring to get the safer default.

A line can be stored twice; a confirmed line is not lost. Delivery is at least once, as with any ingest whose senders retry:

A sender’s retry. A sender that gets an error, or no answer at all, sends the whole batch again. If the first attempt was written in full or in part before that — a node dying in the middle of the request, say — those lines are stored twice. When Sondar knows part of a batch was written, its answer says how much: 503 Server is busy: … (1673 of 5000 events were written).
A server killed while writing. When a killed server restarts, or a copy of an index is rebuilt after a node failure, a few lines that were being written at that moment can be stored twice on that copy — at most about one per indexing thread. The copies of an index can then differ by those lines, so the same count can vary by a few depending on which copy answers.

On a Durable index neither loses anything that was confirmed. Measured in 9 runs on three nodes, killing one or two of them during ingest: a sender’s retry stored up to one batch (5,000 lines) a second time, and a rebuilt copy held at most 1 extra line.

Full-text indexing and word breakers

By default every event is full-text indexed, so a bare term in a query matches against the raw line:

search2 index="app_logs" "connection refused"

Word breakers decide what a “term” is, and this is the setting most worth understanding before you ingest.

The default breaker list is:

,'";=()[]{}?@&<>/:.

Whitespace always breaks (tab, newline, carriage return, space and the ASCII separator controls). Anything in the list above breaks as well; everything else is part of a token.

. was added to that list in August 2026, and it matters more than it looks. Without it, java.lang.NullPointerException is a single token, so searching NullPointerException returns nothing — no error, just an empty result. Logs are built out of dotted identifiers: class names, module paths, hostnames, FQDNs. If you are running an older build, or an index created before the change, add . to that index’s breaker list.

Two settings sit alongside it:

Case sensitivity — off by default, so ERROR, Error and error all match each other. Leave it off unless you have a specific reason.
CJK segmentation — off by default. Turn it on only for indexes that actually carry Chinese text; it is not needed for English logs.

Changing the breakers later

Tokenizing happens at write time. A change therefore applies to newly written data only, and old and new events in the same index are each searched by whichever rule was in force when they were written. There is no reindex.

Practically: get this right when you create the index. If you must change it afterwards, expect a period where the same query behaves differently either side of the change, and consider a new index instead.

Source types

A source type declares the format of the events in it — json, syslog, nginx_access, or one you define. It is the hook that field extraction hangs off: extraction rules are bound to a source type, not to an index, so the same parsing applies wherever that format arrives.

A source type carries:

Line handling — single-line, or multi-line joined by a head pattern.
Timestamp — where the event time comes from, and its format. Getting this wrong is the most common cause of “my logs are in the wrong place on the timeline”.
Field discovery — whether to attempt automatic key/value discovery.
Extractions — the rules below.

Field extraction

This is the part most likely to cost you data if misread, so read it closely.

There are two ways a raw line becomes fields. Both work on both engines; the difference is when the work happens, and what it costs.

1. Field acceleration (index-time)

An extraction marked indexed runs when data is written. The fields are stored in the index alongside the event.

Works on both search engines.
Costs a little write throughput and some storage.
Applies to new data only — turning it on does not reach back.
What happens to events already stored differs by engine, and this is the part worth reading before you enable it on an index with history.
- On search1, the field still resolves. The engine answers per row: a row that has no accelerated column is parsed from _raw at read time, so older events keep the field and a backfill never has to be complete.
- On search2, it does not. Once the column exists on some rows, the engine reads the column and does not fall back to _raw for the rows that lack one, so events written before the flip lose that field. Measured: a field with no column anywhere recovers every row; a field whose column exists on 6 of 15 rows recovers 6. It is not silent — the scan counts the events it read with no value for each accelerated field the query names and reports the gap, so the answer says how incomplete it is. After a backfill the count reaches zero and the warning stops by itself.
- Either way, _raw must still be retained. Where a source type is set to discard it, nothing can recover the field at read time on either engine.

2. Search-time extraction

An extraction left not indexed is meant to run at query time, against _raw. Nothing is stored, so a rule can be changed and applied to data already ingested — which is why this is the normal mode.

Both engines apply it, with one limit on search2:

rule type	`search1`	`search2`
`regex`	yes	yes, when the pattern names a capture group for the field
`json`	yes	yes
`csv`, `kv`, `xml`	yes	no — the field comes back empty, and the query says so

search2 applies a rule to the rows of the source type it is defined for, and only those, so one index holding several source types cannot have one type’s pattern applied to another’s events.

A search-time field works in the search itself. repo="web" user="alice" filters on a search-time user exactly as | where user=="alice" does, on both engines, including OR and NOT. It cannot narrow the scan by index, so over a long time range it is slower than the same term on an accelerated field. (Before September 2026, search2 returned no rows for this form.)

A new or changed rule applies from the next search. Saving a rule takes effect at once on the server it was saved on; a cluster’s other nodes pick it up within a minute. Measured September 2026: a query grouping by a field that no rule yet extracted returned no rows; the rule was saved; the same query 3 seconds later grouped all nine sample events by it.

A key-value rule takes the pairs it finds. In GET /api/cart user=alice latency_ms=100, with a space between fields and = between key and value, it extracts user and latency_ms and skips GET and the path. A line with no pairs at all fails the preview.

The query tells you which case you are in. A search whose fields are read from _raw carries a notice: the results are complete, but the work happens per row, so a filter on such a field cannot use an index and a large time range will be slower. A search using a rule search2 cannot apply carries a warning instead — that one means the number is wrong, not slow, because the field is empty for every event whose value was not written at index time.

Earlier builds behaved differently. Before Aug 2026, search2 performed no search-time extraction at all and returned null with no error and no warning — a plausible-looking number computed over nothing. If you are reading notes written against an older build, that is what they describe.

Which engine runs a query with no leading search/search2 token is decided by the global Default Search Engine setting. A fresh install persists search2 at first boot; a setting left blank falls back to search1.

Choosing

you want	use
fields on data already ingested	search-time
a rule you expect to change	search-time
the lowest write cost	search-time
`csv`, `kv` or `xml` fields on `search2`	acceleration
a stable, high-volume format	acceleration
the lowest query cost	acceleration

If your data arrives through a collection agent, prefer parsing there — see below — and treat both of these as the fallback for data you cannot parse at the edge.

Field aliases

A field alias renames a field at read time — a rule maps an existing field to a friendlier name, so user_id can be presented and queried as User ID. It is configured under the Fields surface (Field Aliases), alongside the extraction rules, and a rule is scoped to either an index or a sourcetype:

Name — the alias rule’s name.
Scope — index or sourcetype; which fields the alias applies to.
Field aliases — the field → alias pairs.
Overwrite — if the alias field already exists in the data, whether to overwrite its value with the current field’s.

Both scopes work on both engines, for stored and search-time fields alike, in a group-by, a where, and the search itself. An index-scoped alias applies only to events of the indexes it names; a sourcetype-scoped one only to events of that sourcetype.

One difference: Overwrite on search2. If an event already carries a real field with the alias’s name, search2 keeps that field’s own value whatever Overwrite says; search1 follows the setting.

Measured September 2026: service aliased as svc grouped 4 / 4 by either scope on both engines; the search-time user aliased as who grouped 3 / 3 / 3 by either scope on both, who="carol" in the search counted 3, and an index-scoped alias gave no values to events of another index. (Before September 2026, search2 ignored sourcetype-scoped aliases and index-scoped aliases of search-time fields — both returned no rows.)

Parsing at the agent

The collection agent parses with a transformers stage in the collection config, before the event is ever sent. This is usually the best place: the fields arrive already structured, both engines see them, and there is no per-query cost.

Four transformers parse:

transformer	for
`grok named_patterns="…"`	syslog and other named text formats
`rex field="raw" "…"`	anything you can write a regex for
`kv field="raw"`	`key=value` text, such as auditd
`json`	JSON lines

eval, fields, rename and the add_metadata_* family reshape events that are already parsed; they do not parse.

The agent registers 44 transformer types. Seven are in no English document at all, and two of those are ones a customer would want on day one:

transformer	what it does	why it matters
`filter`	drop or keep events at the agent	the primary cost lever. Noise dropped at the edge is never shipped, never indexed and never billed. Every competitor leads with this
`ua`	parse a User-Agent string into browser, OS and device	standard for web access logs; Datadog and Elastic both ship it
`add_metadata`	enrich with agent, host, environment and tag metadata	four variants: `agent`, `agenttags`, `environment`, `system`
`fieldconcat`	join several fields into one
`rename_batch`	rename many fields in one step
`event` (`event_parse`)	parse an event payload
`totrace`	convert events to trace spans

The other 28 are documented somewhere in the handbook. These seven are not, which means filter — the one transformer that directly reduces a customer’s bill — is invisible unless they read the agent source.

Multi-line events are joined by the reader, not a transformer — set head_pattern on the reader to a regex matching the first line of a record, and everything up to the next match becomes one event. A Java stack trace collected this way arrives as a single event with real newlines.

Limits worth knowing

limit	value	what happens
Line length, agent	1 MB (`max_line_len`)	the line is dropped; the agent logs a warning
Line length, server	1 MB (`sondar.text_parser.max_line_bytes`)	the line is truncated; the server logs a rate-limited WARN naming the bytes discarded and the running total

Both thresholds are 1 MB, so a line the agent accepts is one the server accepts. The server limit is measured in bytes, not characters, and is rounded down so a multi-byte character is never split; it is a dynamic setting, so raising it does not need a node restart. A line above 1 MB never arrives at all — the agent drops it. If you collect anything that can be large — thread dumps, serialized payloads, base64 blobs — either raise the server limit or split the record at the source.

Older notes describe a 100,000-character server limit that truncated silently. That was sondar.text_parser.max_line_chars, and it cut ordinary stack traces and large JSON payloads while reporting success: 1, failure: 0. The limit is now 1 MB, measured in bytes, and truncation is logged.

Getting it right the first time

The settings that are expensive to change later, in the order you meet them:

Index name and retention — permanent, and the main cost lever.
Word breakers — write-time, so a change splits your data’s behavior. Confirm . is present.
Timestamp handling on the source type — wrong here means every downstream query and dashboard is wrong.
Where you parse — at the agent if you can.

Everything else can be adjusted as you learn the data.