Log collection sources

The collection sources the Sondar agent ships with, and what each one is for.

The collection chapter documents eight sources — File, Script, JVM, HTTP Fetch, HTTP Listen, Prometheus remote r/w, Agent Metric/Log, Server Log. This page covers the rest: container, Kubernetes, Docker, Filebeat and Lumberjack readers are all built and shipping, and they are the sources a US enterprise reaches for first.

Start here

Two configs cover the common cases. Take one, then come back for the inventory only when you need a source that is not here.

Plain log files on a host — fileset watches a path and tails what it finds:

readers:
  - fileset:
      path: /var/log/myapp/*.log
      read_from: newest         # skip what is already there.
                                # `oldest` is the default and reads the backlog

senders:
  - sondar:
      repo: default
      sourcetype: my_app
      url: ["https://<your-sondar-host>:9200"]
      token: "<collect token>"

Container logs on Kubernetes — container_stdout, with the agent running as a DaemonSet. That one is worked through in full at Kubernetes — a worked example at the bottom of this page.

Both write into an index you have created (default exists on a fresh install), and the collect token comes from Configure → Access Control → Collect Tokens. Parsing is optional at this stage — a source type can extract fields at search time instead; see Indexes, source types and field extraction.

The rest of this page is the complete inventory: every reader that ships, and the keys each accepts.

Container and Kubernetes

The largest gap in the older documentation. Six distinct readers ship for container logs alone.

reader	what it collects
`files/container_stdout`	stdout/stderr of running containers, the usual entry point for Docker and Kubernetes
`files/docker_log`	Docker’s own JSON log files
`files/container_tail`	tails a single container log file
`files/container_tailx`	tails many container log files by pattern
`files/container_dir`	watches a directory of container logs
`files/container_dirx`	as above, with pattern matching
`files/container_fileset`	multi-file container collection with an open-file ceiling
`autodiscover`	discovers containers and attaches collection as they start and stop

autodiscover is the one that makes Kubernetes practical: without it, every pod restart would need its log path re-registered by hand.

See Configuration — file readers below for the keys these accept; the container readers share the same option set.

Files

reader	what it collects
`files/fsnotify`	filesystem-event-driven collection, no polling
`files/aws_s3`	log objects from an S3 bucket

The general file modes — fileset, dir, dirx, tail, tailx — share the same option set, documented under Configuration below.

Agent-to-agent and shipper protocols

reader	what it collects
`beats/filebeat/v1`, `beats/filebeat/v2`	accepts data from Elastic Filebeat
`beats/heartbeat`	Elastic Heartbeat
`beats/packetbeat`	Elastic Packetbeat
`lumberjack_listener`	the Lumberjack protocol — what Filebeat’s Logstash output sends (below)

These matter for migration: a team already running Filebeat can point it at a Sondar agent without replacing its fleet — see Lumberjack — from Filebeat.

The Filebeat collector family — twelve shipped, none documented

Separate from the beats/filebeat reader (which accepts data from a customer’s existing Filebeat), the integration catalog ships twelve Filebeat-based collectors as installable apps. All twelve are on the ship list — none are hidden — and none appear in any English document.

app	collects
`journald_fbt`	the systemd journal
`winlog_fbt`	Windows Event Log
`syslog_fbt`	syslog over the network
`tcp_fbt`, `udp_fbt`	arbitrary line streams over TCP or UDP
`unix_fbt`	a Unix domain socket
`kafka_fbt`	a Kafka topic
`filestream_fbt`, `log_fbt`	files, the modern and legacy Filebeat inputs
`container_fbt`, `docker_fbt`	container stdout/stderr
`stdin_fbt`	standard input

Two of these answer questions that come up in every enterprise evaluation. “Can you collect Windows Event Logs?” and “can you read the systemd journal?” are near-universal, the answer is yes for both, and nothing a prospect or a sales engineer can read says so. syslog_fbt, tcp_fbt and udp_fbt matter almost as much: network devices and appliances that cannot run an agent are a standard part of a log estate.

Verified rather than assumed: tools/scope/hidden-apps.json lists all twelve in ship (337 apps) and none in hidden (30).

Standards and platforms

reader	what it collects
`opentelemetry`	OTLP ingest
`vsphere/event`	VMware vSphere events
`script`	output of an arbitrary script
`metrics`, `telegraf`	agent and Telegraf metrics
`event`	internal agent event stream

Pulling from a system that already stores the data

The consumer family reads from an existing store rather than tailing a file — 31 readers, listed below.

Every SQL source comes in two modes, which is the distinction the handbook calls incremental and full:

append — incremental. Poll for rows added since the last checkpoint.
snapshot — full. Read the whole table each cycle.

source	readers
MySQL	`sql_consumer/{append,snapshot}/mysql`, plus `binlog_consumer/mysql_binlog_consumer` for change-data-capture
PostgreSQL	`sql_consumer/{append,snapshot}/postgres`
Oracle	`sql_consumer/{append,snapshot}/oracle`
SQL Server	`sql_consumer/{append,snapshot}/mssql`
Dameng	`sql_consumer/{append,snapshot}/dm`
Trino	`sql_consumer/{append,snapshot}/trino`
Druid	`sql_consumer/{append,snapshot}/druid`
MongoDB	`sql_consumer/mongodb_consumer/{append,snapshot}`
Elasticsearch	`es_consumer/{append,snapshot}` — two client implementations, `go-elasticsearch` and `olivere`
HBase	`hbase_consumer/{append,snapshot}`
Redis	`redis_consumer/{goredis,redigo}`
Kafka	`kafka_consumer/{sarama,segmentio}`
Sondar itself	`sondardb_consumer/{append,snapshot}` — read one index to write another
Zabbix	`zabbix_consumer/zabbix_mysql_binlog_consumer`
S3	`files/aws_s3` (in the file family, but the same job)

Which of these matter for logs. kafka_consumer is the common one — a great many organizations already put logs on a topic, and this reads them without changing that pipeline. es_consumer matters for migration: it reads an existing Elasticsearch cluster, so logs can move to Sondar without a re-instrumentation project. files/aws_s3 picks up logs already landed in object storage. sondardb_consumer is how you build a rolled-up index from a raw one.

Two client libraries, for Redis and Kafka. Nothing in the product says which to prefer, and the choice is not obviously reversible once a checkpoint exists. That is a documentation gap worth closing before a customer picks by coin-flip.

Event sources

Separate from log collection, the agent’s event listener converts alerts from thirteen sources into Sondar events. Four ship as apps with an index, an alert rule and setup documentation — Datadog, Dynatrace, New Relic and Elastic; see Alerts from other tools. The rest are used from a collection task written in YAML.

event source	registered as
Datadog	`datadog_event`
Dynatrace	`dynatrace_event`
New Relic	`newrelic_event`
ELK	`elk_event`
Prometheus	`prometheus_event`
Zabbix	`zabbix_event`
RESTful (generic)	`restful_event`
RESTful change events	`restful_change`
Sondar	`sondar_event`
Alibaba Cloud	`aliyun_event`
Huawei Cloud	`huawei_event`
Cloudwise	`cloudwise_event`
EMCC	`emcc_event`

Datadog, Dynatrace, New Relic and ELK are the interesting four. They mean Sondar can consume events from the monitoring platform a prospect already runs, which is a coexistence story rather than a rip-and-replace one — and it is currently invisible to anyone reading the documentation. Whether they stay is a scope question: aliyun_event, huawei_event, cloudwise_event and emcc_event are China-market integrations in the same family as the Alibaba Cloud log reader, and belong to the same decision.

Most of that decision is made. The China-market apps do not ship: the four event apps above were deleted in Aug 2026, the aliyun collector left the shipping catalog on 3 Sep 2026, and the gitee forge integration was deleted on 28 Sep 2026. On the agent side the aliyun_sls reader is gone too (5 Aug 2026, with its Go SDKs).

What is still there is narrower than “the agent”: the four event converters (aliyun, cloudwise, emcc, huawei under eventutils) remain compiled into every agent binary, with no app, template or UI able to reach them. They are dead code rather than reachable surface.

All thirteen are registered in eventutils/all/all.go.

Out of scope for the US product

reader note

aliyun_sls Alibaba Cloud Log Service. Removed 5 Aug 2026 — the plugin, its registration and the Alibaba Cloud Go SDKs all came out of the agent together. An earlier revision of this row said it was “compiled into every agent” and recommended removal; that had been true and stopped being true, and the row outlived the fix. Verified 3 Sep 2026: no reader package, and go mod why reports the main module does not need alibaba-cloud-sdk-go.

reader	note
`aliyun_sls`	Alibaba Cloud Log Service. Removed 5 Aug 2026 — the plugin, its registration and the Alibaba Cloud Go SDKs all came out of the agent together. An earlier revision of this row said it was “compiled into every agent” and recommended removal; that had been true and stopped being true, and the row outlived the fix. Verified 3 Sep 2026: no reader package, and `go mod why` reports the main module does not need `alibaba-cloud-sdk-go`.

Configuration — file readers

Every file-based reader (fileset, dir, dirx, tail, tailx, and the container_* variants) takes the keys below. Read from the reader source, not from any document: plugins/reader/files/fileset/types.go for the reader’s own keys and plugins/reader/files/inner/ for the shared ones, with defaults taken from the constructors (New() and NewOptions()) rather than from comments.

Selecting files

key	type	default	notes
`path`	string	—	Required. Glob patterns are allowed.
`valid_path_pattern`	list	all	Glob patterns to include.
`ignore_path_pattern`	list	none	Glob patterns to skip.
`max_open_files`	int	`10`	Ceiling on concurrently open files. Validated 1–10000.
`stat_interval`	duration	`10s`	How often the directory is rescanned.

Reading

key	type	default	notes
`read_from`	string	`oldest`	`oldest` or `newest`. `newest` skips everything already in the file — the right choice for a busy log you do not want to backfill.
`encoding`	string	`UTF-8`
`delimiter`	string	`\n`	Line separator.
`head_pattern`	regex	none	Multiline. A line matching this starts a new event; lines that do not are appended to the previous one. This is how a Java stack trace stays one event.
`max_line_len`	bytes	`1 MB`	Validated at a minimum of 100.
`runner_max_line_len`	bytes	`1 MB`	Per-runner ceiling.
`reader_buf_size`	bytes	`4096`
`read_limit`	MB	`-1`	Read rate cap; `-1` is unlimited.
`read_line_limit`	int	`-1`	Lines-per-interval cap; `-1` is unlimited.
`ignore_file_not_exist`	bool	`false`	Do not error when `path` matches nothing yet.
`channel_size`	int	`100`	Internal buffer between reader and pipeline.
`delete_after_read`	bool	`false`	Destructive. Removes the file once consumed. Only for a spool directory you own.

A worked example

Collecting a multiline application log, skipping the backlog, into a named index:

readers:
  - fileset:
      path: /var/log/myapp/*.log
      ignore_path_pattern: ["*.gz", "*.1"]
      read_from: newest
      # An event starts at an ISO-8601 timestamp; continuation lines -- the
      # stack trace -- are folded into the event above.
      head_pattern: '^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}'

transformers:
  - grok field="raw" pattern="%{TIMESTAMP_ISO8601:ts} %{LOGLEVEL:level} %{GREEDYDATA:msg}"

senders:
  - sondar:
      repo: myapp_logs
      sourcetype: myapp
      token: "${sondar_token}"

sondar:, not keta:. The shipped configs still use the old sender name. The agent accepts both — keta is kept as a deprecated alias precisely so those configs keep working — but write new configs against sondar.

Parsing happens here, in the agent. There is no server-side parser: events sent straight to the ingest API arrive with only the built-in fields. The transformers stage is where structure is created.

Configuration — non-file readers

The readers that do not tail files, with their verified config keys.

Script

script runs a command and collects its output. The command is a script file or a shell command; it must emit data on stdout.

readers:
  - script:
      script_cmd: bash            # required: a script file or command
      script_args: /opt/collect.sh   # arguments to the command
      script_timeout: 10s         # per-run timeout (default 10s)
      app: myapp                  # optional app label

Lumberjack — from Filebeat

lumberjack_listener accepts the Lumberjack protocol, which is what Filebeat’s Logstash output speaks. An existing Filebeat ships to the agent by pointing that output at it; nothing else in its configuration changes:

output.logstash:
  hosts: ["sondar-agent.internal:5000"]

Filebeat’s message becomes the event’s raw text, log.file.path its origin and host.name its host; the rest of Filebeat’s fields (agent, input, log.offset, …) arrive as event fields. Measured with Filebeat 8.19.22 and 9.5.4 at their default compression: every event delivered and found by a keyword search. Logstash’s own lumberjack output has not been tested against it.

The listener runs on :5000 by default, both protocol versions enabled.

readers:
  - lumberjack_listener:
      address: ":5000"            # required; listen address
      enable_v1: true             # Lumberjack v1 frames (default on)
      enable_v2: true             # Lumberjack v2 frames (default on)
      keep_alive: 3s
      timeout: 30s
      encoding: utf-8
      tls_ca: /path/to/ca.pem     # optional server TLS

S3

s3 reads log objects from an S3-compatible bucket, downloading them to a local spool and tailing them. This is the log-archive ingestion path.

readers:
  - s3:
      endpoint: https://s3.example.com   # required
      region: us-east-1                  # required
      bucket: logs                        # required
      access_key_id: AKIA...              # required
      secret_access_key: "..."            # required
      prefix: prod/                       # object key prefix
      pattern: "*.log.gz"                 # filename match
      s3_delimiter: "/"
      hostname_immutable: true            # see below — set this for MinIO, Ceph, Wasabi
      # plus the file-reader keys (head_pattern, encoding, max_line_len, …)

hostname_immutable is the one to know about if endpoint is not AWS. By default the AWS SDK rewrites an endpoint into virtual-host style — it moves the bucket into the hostname, so https://s3.example.com becomes https://logs.s3.example.com. Against a real AWS endpoint that is correct. Against MinIO, Ceph, Wasabi or any other S3-compatible store it usually resolves to nothing, and the reader fails with a DNS or connection error that says nothing about buckets. Setting hostname_immutable: true tells the SDK to use the endpoint exactly as written.

OpenTelemetry (OTLP)

otlp_otel accepts OTLP traces, metrics and logs — this is the APM ingest path (see APM). skywalking_otel accepts the SkyWalking protocol. gRPC defaults to 0.0.0.0:4317, HTTP to 0.0.0.0:4318.

readers:
  - otlp_otel:
      receiver:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
            # tls: { ca_file: …, cert_file: …, key_file: … }
            # auth: { authenticator: basicauth }   # see extensions below
          http:
            endpoint: 0.0.0.0:4318
      extensions:
        basicauth:
          htpasswd:
            inline: "user:password"
      processors:
        traces:
          - batch:
      timeout: 5s

The Filebeat collector family

The twelve *_fbt readers (filestream_fbt, log_fbt, journald_fbt, winlog_fbt, syslog_fbt, tcp_fbt, udp_fbt, unix_fbt, kafka_fbt, container_fbt, docker_fbt, stdin_fbt) are Filebeat inputs running inside the agent, so they take Filebeat’s own input configuration — the same paths/multiline/processors keys a standalone Filebeat config uses. The integration apps ship ready-made templates for these; the reader names above are the ones the agent registers.

Kubernetes — a worked example

Collecting pod logs on Kubernetes. The agent runs as a DaemonSet — one pod per node — with the node’s runtime and host paths mounted in. Install it with the Helm chart (Agent and machines); the chart takes SONDARD_RUNTIME (containerd, docker, or crio) and mounts /hostfs/… for the host’s logs and runtime socket.

Then a collection task reads container stdout with container_stdout, which tails the container runtime’s log files and emits one event per line:

readers:
  - container_stdout:
      interval: 30s
      codec: containerd           # docker, containerd, or cri
      read_from: newest           # skip the existing backlog
      head_pattern: '^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}'   # multiline: a new
                                                              # event starts at a timestamp

transformers:
  - grok field="raw" pattern="%{TIMESTAMP_ISO8601:ts} %{LOGLEVEL:level} %{GREEDYDATA:msg}"

senders:
  - sondar:
      repo: k8s_pod_logs
      sourcetype: container_stdout
      token: "${sondar_token}"

The autodiscover reader discovers containers as they start and stop and attaches collection automatically, so pod restarts do not need re-registering — the mechanism that makes this work at cluster scale without hand-editing paths. Its config is interfaces (the container runtimes to watch) and probe_timeout.

The same pod-log data, without the agent, can be pulled by the container_fbt/docker_fbt Filebeat collectors or pushed via the otlp_otel reader. The k8s metrics side (cluster, node, pod, event) is the infrastructure_monitoring app’s job.