Pub/Sub

Pub/sub on topics is a small capped, TTL’d feed topic with routers fanning every record out to one topic per subscriber, and subscribers tailing over SSE with tail: true. Memory stays bounded because the feed and the subscriber topics have a small cap_records and a short ttl_ms; a subscriber that falls behind the retention window doesn’t break — it gets an explicit tombstone frame naming the exact gap, then resumes. This is the weak-guarantee, low-latency pattern: fast fan-out, bounded resources, gaps tolerated and signalled, never silent.

It is deliberately the opposite of the durable streams recipe. There you make eviction impossible; here you embrace it for bounded memory, and rely on the gap signal to keep slow subscribers honest.

Use the disk durability class (the default, durable: false) for pub/sub. A group-committed WAL keeps latency near the ~1 ms target without paying a per-write fsync, and a crash only loses the un-fsynced tail — acceptable for a feed whose whole point is recency. Use durability: "memory" for the best-effort class — the same group-committed WAL path as disk but with no durability guarantee (after a restart its records may survive or be lost). See Durability classes.

The shape


                       router →web         ┌──────────────────────┐ SSE tail  subscriber web
   publisher ──►   ┌───────────────┐ ─────►│   notifications:web   │ ────────► (watch …:web)
   POST            │ notifications │       └──────────────────────┘
   /notifications  │ cap+ttl       │ router →mobile ┌──────────────────────┐ SSE tail subscriber mobile
                   └───────────────┘ ──────────────►│ notifications:mobile  │ ───────► (watch …:mobile)
                                                    └──────────────────────┘

A publisher writes once to notifications. Routers copy each record into notifications:web, notifications:mobile, … — one topic per subscriber. Each subscriber watches its own topic over SSE. Giving each subscriber its own topic (rather than sharing one) means each keeps an independent cursor, an independent retention window, and an independent slow-consumer fate.

Build it

Create the feed with a small cap + TTL

A bounded feed: keep at most cap_records records, expire anything older than ttl_ms. discard: "old" (the default) evicts the oldest when full — that is the pub/sub-friendly policy.

Examples use $TOPICS as the base URL — see the Quickstart.


curl -X PUT $TOPICS/v0/topics/notifications \
  -H 'content-type: application/json' \
  -d '{ "ttl_ms": 5000, "cap_records": 10000, "discard": "old", "durable": false }'


# →
{ "topic": "notifications", "created": true,
  "config": { "ttl_ms": 5000, "cap_records": 10000, "cap_bytes": 0, "discard": "old",
              "durable": false, "durability": "disk", "auto_create": true,
              "idempotency_window_ms": 120000, "dedupe_node": true },
  "performance": { "server_total_ms": 0.21 } }

Fan out with one router per subscriber

A router copies every record appended to notifications into a per-subscriber topic, at-least-once and per-source FIFO. The subscriber topics auto-create (lazily) on first forward, inheriting defaults — give them their own small cap+TTL up front if you want a different retention than the feed.


curl -X PUT $TOPICS/v0/topics/notifications:web -d '{ "ttl_ms": 5000, "cap_records": 10000 }'
curl -X PUT $TOPICS/v0/topics/notifications:mobile -d '{ "ttl_ms": 5000, "cap_records": 10000 }'
 
curl -X PUT $TOPICS/v0/routers/notifications-to-web -d '{ "source": "notifications", "dest": "notifications:web" }'
curl -X PUT $TOPICS/v0/routers/notifications-to-mobile -d '{ "source": "notifications", "dest": "notifications:mobile" }'


# → (for each router)
{ "router": "notifications-to-web", "created": true, "source": "notifications", "dest": "notifications:web",
  "preserve_node": true, "preserve_tag": true, "filter": null, "allow_cycle": false,
  "performance": { "server_total_ms": 0.20 } }

A forward obeys the destination topic’s config: with discard: "old", a full notifications:web evicts its oldest and the forward succeeds (a lagging web subscriber loses the oldest, gets a tombstone, and mobile is unaffected). Routers can also carry a filter (the same tuple language as deletes) to forward only matching records — e.g. ["tag","Glob","public:*"] for a per-subscriber topic filter. See Routers.

Publish

The publisher writes to notifications; the routers do the fan-out. Tag records if subscribers will filter or you’ll want to cancel by tag.


curl -X POST $TOPICS/v0/topics/notifications \
  -H 'content-type: application/json' \
  -d '{ "records": [
          { "data": { "type": "comment", "post": "p-123", "actor": "user-1042" }, "tag": "user-1042:msg-5" }
        ] }'


# →
{ "topic": "notifications", "first_seq": 1, "last_seq": 1, "seqs": [1],
  "head_seq": 1, "count": 1, "created": false, "deduped": false,
  "performance": { "server_total_ms": 0.18 } }

Each subscriber watches its own topic with tail: true, which starts at the topic’s current head — only records published after subscribe (the SSE analog of Redis XREAD $). Create the session, then open the stream.


# 1. Create the watch session.
curl -X POST $TOPICS/v0/watch \
  -H 'content-type: application/json' \
  -d '{ "topics": { "notifications:web": { "tail": true } } }'
# → { "wid": "wid_BuRguGorNdVFWNQULz-rrw", "stream_url": "/v0/watch/wid_BuRguGorNdVFWNQULz-rrw", ... }
 
# 2. Open the stream.
curl -N $TOPICS/v0/watch/wid_BuRguGorNdVFWNQULz-rrw


retry: 2000

id: eyJzdWItYSI6MX0
event: record
data: {"topic":"notifications:web","records":[{"$seq":1,"$ts":1748470000123,"data":{"type":"comment","post":"p-123","actor":"user-1042"}}],"from_seq":0,"to_seq":1,"head_seq":1}

: hb

A subscriber can watch many topics over the one SSE connection (up to 256), each with its own cursor — so a single client tailing several topics needs only one stream. The id: on each data frame encodes the whole per-topic cursor map; on reconnect the client passes it as Last-Event-ID to resume exactly.

Tolerating gaps — the tombstone

This is the load-bearing part of weak-guarantee pub/sub. Because the subscriber topics are small and TTL’d, a slow subscriber will eventually fall behind the retention window. When that happens topics does not silently drop the missed records — it emits a tombstone frame naming the exact range, then resumes from the new floor:


id: eyJzdWItYSI6ODMwMDB9
event: tombstone
data: {"topic":"notifications:web","reason":"cap","gap_from":80000,"gap_to":83000,"earliest_seq":83001,"head_seq":88130}

The subscriber handles record and tombstone uniformly: process records, and on a tombstone log/account for the missed [gap_from, gap_to] range (and re-fetch state from your source of truth if you need to). reason is cap (evicted for capacity), ttl (expired), mixed (both), or from_seq_too_old (the requested cursor was already below the floor at connect time). The frame’s id: already advances the cursor past the gap, so a resume after it is correct.

The tombstone is the contract that makes “tolerate gaps” safe: involuntary cap/TTL loss is always signalled in-band at HTTP 200, never a silent skip. A subscriber that ignores tombstones is choosing to drop the missed range; a subscriber that handles them can fall arbitrarily far behind and still know precisely what it missed. See Tombstones.

Liveness and slow consumers

Heartbeats keep the connection alive when idle. After heartbeat_ms (default 15000, clamped [1000, 60000]) with no data, the server sends a bare : hb comment — it carries no payload and no id:, so it never perturbs your resume cursor. Suppressed whenever real data went out in the window. Arm a client watchdog at roughly 2 × heartbeat_ms and reconnect on timeout.
Slow consumers have a bounded outbound queue; while it can’t drain, records coalesce into larger, fewer batches. If a subscriber stays stuck, a lossy (capped/TTL’d) topic degrades to a tombstone on resume — exactly the gap signal above — while the rest of the stream keeps flowing. The session survives the disconnect for session_ttl_ms (default 300000) so the client resumes with no loss beyond the topic’s own cap/TTL.

Notes & variations

Lightweight tailing. Set include_data: false on the watch to receive only $seq/$ts/$tag/$node metadata per record — handy for a “something changed, go fetch it” notification feed.
One shared topic instead of per-subscriber topics. You can point every subscriber at notifications directly and skip the routers — but then all subscribers share one retention window, and a single slow subscriber’s lag is measured against the same cap as everyone else. Per-subscriber topics isolate each subscriber’s fate; that isolation is the reason to use routers here.
Bounded memory across the whole topology. Each topic trims itself via cap+TTL. To put a hard ceiling on total retained bytes server-wide, set TOPICS_MAX_TOTAL_BYTES — a write past it returns 429 throttled. See Configuration.