Core Guarantees
topics is a durability-focused engine, so its guarantees are the product. Two of them are the headline, and they answer the two questions that matter most about any log:
- How durable is a write? Durability is a dial you turn per topic —
ephemeral(resident-only),memory(best-effort),disk(survives a crash), orfsync(survives power loss). Weak where you want speed, strong where an ack must be a promise. See Durability Classes. - What happens when data is lost? Loss is always explicit, never silent — anything you didn’t ask to lose, you are told about in band, with the exact range. That is the load-bearing invariant stated next.
This page states the rule the whole data model is built to keep, then summarizes each guarantee and links to the page that specifies it in full.
The load-bearing invariant
One rule sits underneath everything:
Involuntary, capacity-driven loss you didn’t ask for — cap eviction, TTL expiry —
always produces a tombstone with the exact
[gap_from, gap_to] range, in-band at HTTP 200. Voluntary removal you did ask for — a
permanent delete, your own node’s events filtered by
loop-prevention — is silently skipped, never a
tombstone.
Mixing those two would make the gap alarm useless. If a delete you issued looked identical to data that fell off a retention window, you could never trust a tombstone to mean “something was lost that I wanted.” So topics keeps the two structurally separate.
The mechanism is a dual watermark per topic. earliest_seq is the first currently-live seq
— it advances on eviction, TTL expiry, and deletion. evict_floor advances only on
involuntary cap/TTL loss, and it is the sole tombstone trigger. The invariant
evict_floor <= earliest_seq always holds, which has a precise consequence:
- A cursor below
earliest_seqbut at or aboveevict_floorfell into a purely-deleted gap (voluntary) — the read is silent,tombstone: null, and the cursor advances past the deleted seqs. - A cursor below
evict_floorlost live records to cap or TTL (involuntary) — the read emits a tombstone before resuming fromearliest_seq.
A consumer that wants to detect deletions or node-filtered drops can do so with
head_seq/earliest_seq arithmetic — but it never confuses them with data loss, because a
delete advances earliest_seq and never evict_floor.
The seven safety invariants
The full contract, distilled:
seqis strictly increasing and gap-free at assignment. Visibility holes (deleted, expired, node-filtered, evicted) are normal and are categorized for the consumer — see Ordering & cursors.- No silent capacity loss. Any cap eviction or TTL expiry of live records crossing a
cursor (
from_seq + 1 < evict_floor) yields an in-band tombstone with an authoritative[gap_from, gap_to]and a best-effortreason. Identical indiffand SSE. - Voluntary removal is silent — and, for deletion, permanent. Permanent deletes and own-node filtering drop records without a tombstone. Deletion is permanent, effective immediately, asynchronously reclaimed, and point-in-time (never a standing filter).
- The dual watermark separates the two.
earliest_seqis the single source of truth for “what can I still read”;evict_floor(involuntary only,evict_floor <= earliest_seq) is the single tombstone trigger. Cap arithmetic is never authoritative — lazy, segment-granular eviction may exceed cap transiently. $nodeis immutable and preserved through routers, making N-way multi-master fan-out safe by construction — see Node loop-prevention.- Routers are at-least-once, per-source FIFO, DAG-by-default; a
dstenforces its own retention anddiscard:"reject"applies backpressure rather than silently dropping. Pair that with idempotent consumers. - Durability is per-topic and explicit. Only data a topic class does not promise to keep
is lost on crash:
ephemeralresident records,memorybest-effort records, or the un-fsynced tail ofdisk. Such loss appears to consumers as ordinary eviction-style gaps. See Durability.
Where each guarantee lives
The four commit classes — ephemeral, memory, disk, fsync — where each lands, when
the ack fires, and what a crash costs.
Explicit loss: the dual watermark, when a tombstone is emitted, its shape and reason
values, and why deleted or node-filtered gaps stay silent.
Permanent, immediate, asynchronously reclaimed, silent, point-in-time removal by seq
range and/or tag match.
The monotonic seq is the cursor. Cursor-advance is ack-all; caught_up, not
records.length, is the “no more” signal.
$node is byte-exact filtered on read so symmetric nodes mirror over routers without
echo — the foundation of multi-master.
Per-node dedupe windows, and why consumers must be idempotent under at-least-once routing and queue delivery.
Idempotency