Parsing and transformation

The middle stage of a collection config — readers → transformers → senders. Transformers parse, clean, enrich and aggregate the raw data before it is sent: a regex to break a log line into fields, an eval to derive a timestamp, a stats window to aggregate, a lookup to join an external source.

This is also the stage the product calls parsing transformation: in interactive collection-task creation it is step three, and every collection template’s transformer section can be written either by hand or through the transformer-template mechanism (see agent-machines.md).

How a transformer config reads

Transformers run in order, each seeing the output of the last. A transformer names its operation and carries that operation’s keys. The simplest and most useful first example — parse an Apache error log, derive a timestamp, drop the raw line:

transformers:
  - rex field="raw" "\[(?P<time>[^\]]*)\]\s+\[(?P<module>[^:]*):(?P<level>[^\]]*)[^[]*\[(?P<pid>[^\]]*)\]"
  - eval timestamp=strptime(time, "Mon Jan 2 15:04:05 2006")
  - fields -time

The same work is available interactively: the collection-task wizard’s “Parse and Transform Data” step offers the same operators through the transformer templates.

The shipped inventory

29 transformers are registered. They fall into five families:

Parsing — break raw text into fields

transformer	what it does
`rex`	regex named-capture groups become fields (the Apache example above)
`grok`	grok patterns (built-in pattern directory plus an `external` dir)
`kv`	`key=value` pairs
`json`	parse a JSON body into fields
`jsonpath`	extract by JSONPath
`jsonflatten`	flatten nested JSON
`xmlpath`	extract by XMLPath
`csv`	split a CSV line by separator
`split`	split one field into many by a delimiter
`syslog`	parse a syslog line
`user_agent`	parse a User-Agent string

Field processing — clean and rename

transformer	what it does
`eval`	compute new fields from expressions (the full grammar below)
`fields`	keep or drop fields (`fields +a -b`, `fields -time`)
`rename`	rename a field
`replace`	replace a substring by regex
`fieldconcat`	concatenate fields into one
`datamask`	mask sensitive field values
`mvexpand`	expand a multivalued field into rows
`mvcombine`	group by every other field and collect a field’s values into one multivalued field (the inverse of `mvexpand`)
`filter` / `where`	keep or drop events by a predicate
`tometric`	shape fields into metric values (fields/tags/timestamp)

Metadata enrichment

transformer	what it does
`add_metadata_agent`	the agent’s own info (version, id, tags)
`add_metadata_host`	host metadata
`add_metadata_net` / `add_metadata_disk` / `add_metadata_memory` / `add_metadata_cpu`	the asset metadata (see host-monitoring.md)
`add_metadata_env` / `add_metadata_process`	environment and process metadata
`k8stag`	Kubernetes pod labels as tags
`userinfo`	look up a uid in the local passwd database

Aggregation

transformer	what it does
`stats`	windowed aggregation with grouping (below)

Association (lookup)

transformer	what it does
`lookup_sql`	join an external relational database
`lookup_sondardb`	join another Sondar index

eval — the expression grammar

eval <field>=<expression>[, <field>=<expression>]... computes a new field (overwriting if it exists).

Operators: arithmetic + - * / ( ) %, comparison < <= = > >=, logical and or not, string concatenation.
Scalar functions: len(), tostring(), toint(), tolong(), todouble(), tobool(), upper()/lower(), substr(), strptime() (parse a time string — the Apache example), strftime() (format), plus date/timestamp() helpers.
Multivalued functions — the mv family: mvappend, mvcount, mvdedup, mvindex(X, start, end), mvjoin(X, sep), mvrange(start, end, step), mvzip(X, Y, sep), mvsort, mvfind(X, regex), mvfilter(X, cond), mvmap(X, expr).
Conditional functions: case(cond1, v1, cond2, v2, ...), if(cond, a, b), and coalesce-style null handling.
Math: abs, round, ceil, floor, sqrt, pow, log, min(a,b)/max(a,b).

- eval:
    exprList:
      - type: binary
        left:
          type: lit
          val: 10
        right:
          type: lit
          val: 1
        op: '-'

field processing — the operators

fields — fields +keep1 +keep2 -drop1; fields -time drops time. Without a +/-, fields a,b keeps only those.
rex / grok — named groups become top-level fields; rex field="raw" parses the raw field specifically.
kv — split on a separator; nested key=value pairs become fields.
replace — regex replacement, e.g. replace field=msg regex="\\d+" replacement="N".
datamask — mask sensitive values (credit-card-like patterns and freeform), the mechanism behind the data-masking surface. It runs at collection time, so the mask is written into the stored event and is irreversible — unlike the search-time masking rules it complements, which leave the stored data untouched. See Data masking.
filter / where — keep an event when a predicate holds; the collection-time equivalent of the search-time filter.

stats — windowed aggregation

stats [<func>(field) [as <name>]]... [by <field>...] [window=<size>] [grace_period=<lag>].

Aggregation functions (shipped AggCount…): count, sum, avg, min, max, stddev, s2 (variance), diff (change from the previous value), interval (time since the previous value), rate (diff/interval).

window — time-bucket size, default 5 minutes. Timestamps are bucketed from zero: a timestamp of 12 with window 5 lands in [10,15).
grace_period — how far out-of-order a timestamp may be, default 1 minute. The watermark is max timestamp received − grace_period; when a bucket’s right edge falls below the watermark the bucket closes and emits. An event older than the watermark is dropped, not buffered.

Config keys: agg_funcs (function/field/as), group_fields, window, grace_period. This is a stateful transformer — it holds partial aggregations in memory until a window closes, so it only makes sense for streaming aggregate pipelines, not one-shot file reads.

association — lookup

Two join transformers, both loading an external source once and joining on a key with an in-memory cache:

lookup_sql — join a relational database:

transformers:
  - lookup_sql:
      src: user_id            # field in the event (the "left table" key)
      ref: id                 # the column in the external source
      output_fields: "name,email"
      sql_type: mysql         # mysql | postgres | oracle
      sql_dsn: "<user>:<password>@tcp(localhost:3306)/?parseTime=true"
      sql_query: "select * from users"   # or sql_table: users

src (required) — the field in the data to join on.
ref (required) — the corresponding column in the external source.
output_fields — which external columns to copy in (empty = all).
cache_ttl — how long the loaded source is cached, default 1 hour.

lookup_sondardb — the same shape but the external source is a Sondar index (a SonQL query over a repo), so events can be enriched from an index Sondar already holds — e.g. join app_logs against a reference index.

userinfo — a local lookup: resolve a uid field to the system user (username, name, primary group id gid, home_dir; all_fields picks the four vs two, prefix nests the result).