Data archiving
Data archiving
Section titled “Data archiving”Data archiving moves older data out of the hot index into cheaper, durable storage — object storage (S3-compatible) or a shared file system — where it is still recoverable but no longer costs hot-storage retention. It is the counterpart to index retention: retention deletes, archiving keeps.
The three archive methods
Section titled “The three archive methods”| method | how | when to use |
|---|---|---|
| Snapshot archive | a point-in-time snapshot of Lucene shards | hot/warm/cold tiering; fast recovery, good integrity, no compression |
| Real-time archive | streams through Kafka before ingest, compressed | capture everything as it arrives; object storage only |
| Historical archive | an export task on a schedule, for data already ingested | regularly offload large volumes of history; object storage only |
Storage targets: object storage (S3-compatible) for all three, and a shared file system (mounted share or cloud drive) for snapshot archives.
Destinations
Section titled “Destinations”An archive destination is a named storage target: its type (object storage or shared file system), the path/bucket, the maximum file size, and whether data is encrypted. Create destinations first; archive rules and tasks point at them.
Snapshot and file archives
Section titled “Snapshot and file archives”Configure → Data Archive manages two archive kinds — Snapshot (shard snapshots) and File (file archives) — each as a list of archive definitions. A definition names the index, the data range (before / between / after a date), the archive type, and the destination. A retention setting controls how long the archive is kept before it is automatically deleted.
The Restore surface recovers an archived index back into a queryable index. Monitors watch archive jobs; tasks are the scheduled archive runs (their frequency, progress, and history).
Related
Section titled “Related”Archiving is one of several ways to manage data life:
- Index retention — the hot-storage policy that archiving extends.
- Export — sending search results out of Sondar, distinct from archiving Sondar’s own data.
- The agent’s archive senders — write-once copies at collection time, a third, agent-side mechanism.