Skip to Content
Core GuaranteesOverview

Core Guarantees

topics is a durability-focused engine, so its guarantees are the product. Two of them are the headline, and they answer the two questions that matter most about any log:

  • How durable is a write? Durability is a dial you turn per topicephemeral (resident-only), memory (best-effort), disk (survives a crash), or fsync (survives power loss). Weak where you want speed, strong where an ack must be a promise. See Durability Classes.
  • What happens when data is lost? Loss is always explicit, never silent — anything you didn’t ask to lose, you are told about in band, with the exact range. That is the load-bearing invariant stated next.

This page states the rule the whole data model is built to keep, then summarizes each guarantee and links to the page that specifies it in full.

The load-bearing invariant

One rule sits underneath everything:

Involuntary, capacity-driven loss you didn’t ask for — cap eviction, TTL expiry — always produces a tombstone with the exact [gap_from, gap_to] range, in-band at HTTP 200. Voluntary removal you did ask for — a permanent delete, your own node’s events filtered by loop-preventionis silently skipped, never a tombstone.

Mixing those two would make the gap alarm useless. If a delete you issued looked identical to data that fell off a retention window, you could never trust a tombstone to mean “something was lost that I wanted.” So topics keeps the two structurally separate.

The mechanism is a dual watermark per topic. earliest_seq is the first currently-live seq — it advances on eviction, TTL expiry, and deletion. evict_floor advances only on involuntary cap/TTL loss, and it is the sole tombstone trigger. The invariant evict_floor <= earliest_seq always holds, which has a precise consequence:

  • A cursor below earliest_seq but at or above evict_floor fell into a purely-deleted gap (voluntary) — the read is silent, tombstone: null, and the cursor advances past the deleted seqs.
  • A cursor below evict_floor lost live records to cap or TTL (involuntary) — the read emits a tombstone before resuming from earliest_seq.

A consumer that wants to detect deletions or node-filtered drops can do so with head_seq/earliest_seq arithmetic — but it never confuses them with data loss, because a delete advances earliest_seq and never evict_floor.

The seven safety invariants

The full contract, distilled:

  1. seq is strictly increasing and gap-free at assignment. Visibility holes (deleted, expired, node-filtered, evicted) are normal and are categorized for the consumer — see Ordering & cursors.
  2. No silent capacity loss. Any cap eviction or TTL expiry of live records crossing a cursor (from_seq + 1 < evict_floor) yields an in-band tombstone with an authoritative [gap_from, gap_to] and a best-effort reason. Identical in diff and SSE.
  3. Voluntary removal is silent — and, for deletion, permanent. Permanent deletes and own-node filtering drop records without a tombstone. Deletion is permanent, effective immediately, asynchronously reclaimed, and point-in-time (never a standing filter).
  4. The dual watermark separates the two. earliest_seq is the single source of truth for “what can I still read”; evict_floor (involuntary only, evict_floor <= earliest_seq) is the single tombstone trigger. Cap arithmetic is never authoritative — lazy, segment-granular eviction may exceed cap transiently.
  5. $node is immutable and preserved through routers, making N-way multi-master fan-out safe by construction — see Node loop-prevention.
  6. Routers are at-least-once, per-source FIFO, DAG-by-default; a dst enforces its own retention and discard:"reject" applies backpressure rather than silently dropping. Pair that with idempotent consumers.
  7. Durability is per-topic and explicit. Only data a topic class does not promise to keep is lost on crash: ephemeral resident records, memory best-effort records, or the un-fsynced tail of disk. Such loss appears to consumers as ordinary eviction-style gaps. See Durability.

Where each guarantee lives

Last updated on