Indexes, source types and field extraction

How Sondar stores log data and how raw lines become queryable fields.

Indexes

An index is where log events live. Internally, and in some API payloads, it is called a repo — the two words mean the same thing.

Every event carries repo (its index), sourcetype (its declared format), host, origin, _time, _indexTime and _raw. Everything else has to be parsed out.

Naming

Lowercase letters, digits and underscores. An index name is permanent — plan it before you send data, because moving events between indexes means re-ingesting them.

Name by retention and access, not by service. app_logs_30d and audit_logs_7y are useful names because they encode the two things that actually differ; checkout_service is not, because you will end up wanting the same retention for twenty services and a different one for the auditor.

Lifecycle

Each index has a retention policy — by age, by size, or both. When an index passes its limit the oldest data is removed. Retention is the main lever on storage cost, and it is worth setting deliberately at creation: a default that keeps everything forever is expensive, and one that keeps a week will lose the incident you are asked about a month later.

Cold data can be archived to S3-compatible storage and still be queried, at higher latency. Configure the archive destination first, then point the index at it.

Full-text indexing and word breakers

By default every event is full-text indexed, so a bare term in a query matches against the raw line:

search2 index="app_logs" "connection refused"

Word breakers decide what a “term” is, and this is the setting most worth understanding before you ingest.

The default breaker list is:

,'";=()[]{}?@&<>/:.

Whitespace always breaks (tab, newline, carriage return, space and the ASCII separator controls). Anything in the list above breaks as well; everything else is part of a token.

. was added to that list in August 2026, and it matters more than it looks. Without it, java.lang.NullPointerException is a single token, so searching NullPointerException returns nothing — no error, just an empty result. Logs are built out of dotted identifiers: class names, module paths, hostnames, FQDNs. If you are running an older build, or an index created before the change, add . to that index’s breaker list.

Two settings sit alongside it:

Case sensitivity — off by default, so ERROR, Error and error all match each other. Leave it off unless you have a specific reason.
CJK segmentation — off by default. Turn it on only for indexes that actually carry Chinese text; it is not needed for English logs.

Changing the breakers later

Tokenizing happens at write time. A change therefore applies to newly written data only, and old and new events in the same index are each searched by whichever rule was in force when they were written. There is no reindex.

Practically: get this right when you create the index. If you must change it afterwards, expect a period where the same query behaves differently either side of the change, and consider a new index instead.

Source types

A source type declares the format of the events in it — json, syslog, nginx_access, or one you define. It is the hook that field extraction hangs off: extraction rules are bound to a source type, not to an index, so the same parsing applies wherever that format arrives.

A source type carries:

Line handling — single-line, or multi-line joined by a head pattern.
Timestamp — where the event time comes from, and its format. Getting this wrong is the most common cause of “my logs are in the wrong place on the timeline”.
Field discovery — whether to attempt automatic key/value discovery.
Extractions — the rules below.

Field extraction

This is the part most likely to cost you data if misread, so read it closely.

There are two ways a raw line becomes fields. Both work on both engines; the difference is when the work happens, and what it costs.

1. Field acceleration (index-time)

An extraction marked indexed runs when data is written. The fields are stored in the index alongside the event.

Works on both search engines.
Costs a little write throughput and some storage.
Applies to new data only — turning it on does not reach back.
Turning it on removes the search-time fallback for that rule, so events already stored without accelerated fields stop resolving it. If you enable acceleration on an index with history, expect older events to lose that field.

2. Search-time extraction

An extraction left not indexed is meant to run at query time, against _raw. Nothing is stored, so a rule can be changed and applied to data already ingested — which is why this is the normal mode.

Both engines apply it, with one limit on search2:

rule type	`search1`	`search2`
`regex`	yes	yes, when the pattern names a capture group for the field
`json`	yes	yes
`csv`, `kv`, `xml`	yes	no — the field comes back empty, and the query says so

search2 applies a rule to the rows of the source type it is defined for, and only those, so one index holding several source types cannot have one type’s pattern applied to another’s events.

The query tells you which case you are in. A search whose fields are read from _raw carries a notice: the results are complete, but the work happens per row, so a filter on such a field cannot use an index and a large time range will be slower. A search using a rule search2 cannot apply carries a warning instead — that one means the number is wrong, not slow, because the field is empty for every event whose value was not written at index time.

Earlier builds behaved differently. Before Aug 2026, search2 performed no search-time extraction at all and returned null with no error and no warning — a plausible-looking number computed over nothing. If you are reading notes written against an older build, that is what they describe.

Which engine runs a query with no leading search/search2 token is decided by the global Default Search Engine setting. A fresh install persists search2 at first boot; a setting left blank falls back to search1.

Choosing

you want	use
fields on data already ingested	search-time
a rule you expect to change	search-time
the lowest write cost	search-time
`csv`, `kv` or `xml` fields on `search2`	acceleration
a stable, high-volume format	acceleration
the lowest query cost	acceleration

If your data arrives through a collection agent, prefer parsing there — see below — and treat both of these as the fallback for data you cannot parse at the edge.

Field aliases

A field alias renames a field at read time — a rule maps an existing field to a friendlier name, so user_id can be presented and queried as User ID. It is configured under the Fields surface (Field Aliases), alongside the extraction rules, and a rule is scoped to either an index or a sourcetype:

Name — the alias rule’s name.
Scope — index or sourcetype; which fields the alias applies to.
Field aliases — the field → alias pairs.
Overwrite — if the alias field already exists in the data, whether to overwrite its value with the current field’s.

search2 constraint. search2 supports only per-index aliases, and the alias field must be an accelerated (index-time) field. On search1 both scopes work regardless of acceleration.

Parsing at the agent

The collection agent parses with a transformers stage in the collection config, before the event is ever sent. This is usually the best place: the fields arrive already structured, both engines see them, and there is no per-query cost.

Four transformers parse:

transformer	for
`grok named_patterns="…"`	syslog and other named text formats
`rex field="raw" "…"`	anything you can write a regex for
`kv field="raw"`	`key=value` text, such as auditd
`json`	JSON lines

eval, fields, rename and the add_metadata_* family reshape events that are already parsed; they do not parse.

Thirty-five transformers ship. Seven are in no English document at all, and two of those are ones a customer would want on day one:

transformer	what it does	why it matters
`filter`	drop or keep events at the agent	the primary cost lever. Noise dropped at the edge is never shipped, never indexed and never billed. Every competitor leads with this
`user_agent`	parse a User-Agent string into browser, OS and device	standard for web access logs; Datadog and Elastic both ship it
`add_metadata`	enrich with agent, host, environment and tag metadata	four variants: `agent`, `agenttags`, `environment`, `system`
`fieldconcat`	join several fields into one
`rename_batch`	rename many fields in one step
`event` (`event_parse`)	parse an event payload
`totrace`	convert events to trace spans

The other 28 are documented somewhere in the handbook. These seven are not, which means filter — the one transformer that directly reduces a customer’s bill — is invisible unless they read the agent source.

Multi-line events are joined by the reader, not a transformer — set head_pattern on the reader to a regex matching the first line of a record, and everything up to the next match becomes one event. A Java stack trace collected this way arrives as a single event with real newlines.

Limits worth knowing

limit	value	what happens
Line length, agent	1 MB (`max_line_len`)	the line is dropped; the agent logs a warning
Line length, server	1 MB (`sondar.text_parser.max_line_bytes`)	the line is truncated; the server logs a rate-limited WARN naming the bytes discarded and the running total

Both thresholds are 1 MB, so a line the agent accepts is one the server accepts. The server limit is measured in bytes, not characters, and is rounded down so a multi-byte character is never split; it is a dynamic setting, so raising it does not need a node restart. A line above 1 MB never arrives at all — the agent drops it. If you collect anything that can be large — thread dumps, serialized payloads, base64 blobs — either raise the server limit or split the record at the source.

Older notes describe a 100,000-character server limit that truncated silently. That was sondar.text_parser.max_line_chars, and it cut ordinary stack traces and large JSON payloads while reporting success: 1, failure: 0. The limit is now 1 MB, measured in bytes, and truncation is logged.

Getting it right the first time

The settings that are expensive to change later, in the order you meet them:

Index name and retention — permanent, and the main cost lever.
Word breakers — write-time, so a change splits your data’s behavior. Confirm . is present.
Timestamp handling on the source type — wrong here means every downstream query and dashboard is wrong.
Where you parse — at the agent if you can.

Everything else can be adjusted as you learn the data.