Skip to Content
API ReferenceConventions

Conventions

These are the rules every /v0 endpoint shares, specified once so the per-area reference pages can stay short. Read this page once and the rest of the API is predictable: a JSON body in, a JSON body out, a stable error.code on failure, a performance block on every response, and the monotonic $seq as the cursor.

Base URL & versioning

http://{host}:{port}/v0/...

The API version is the first path segment (/v0). The default bind is 127.0.0.1:4000, so a local server is http://localhost:4000/v0/.... There is no region or tenant in the host — topics is single-machine. Examples across these docs use $TOPICS for the base URL — set it with export TOPICS=http://localhost:4000 (or your server’s address).

Breaking changes would ship as a new prefix (/v1) that may run concurrently. Within a version, all changes are additive: new optional request fields, new response fields, new endpoints. Clients must ignore unknown response fields — that is the forward-compat contract.

Auth

Authorization: Bearer <API_KEY>

A plain bearer token. Keys are supplied at startup via TOPICS_API_KEYS (comma-separated). A missing or unknown key returns 401 unauthorized. If the server starts with no keys configured, auth is disabled (single-tenant dev mode) and the header is ignored — logged loudly at boot. The default loopback bind keeps that dev mode safe; a non-loopback bind with no keys refuses to start unless you set TOPICS_ALLOW_INSECURE_NO_AUTH=1.

Keys are hashed at rest (SHA-256) — only the digest is kept in memory, never the plaintext, which is zeroized after parse and never logged. The comparison is constant-time, so it leaks neither which key nor how many leading bytes matched. A key may additionally carry a scope set (read / write / delete / admin) and a topic-name prefix allowlist; a request that authenticates but lacks the required scope, or names a topic outside its allowlist, returns 403 forbidden.

Scopes, the prefix allowlist, key hashing, and the per-route requirements are documented in full on Security. Each endpoint page also notes the scope it requires.

topics speaks plain HTTP and does not terminate TLS. Bearer keys are secrets: a token sent over plain HTTP, or in a URL query string, can be observed in transit or in logs. For any non-loopback exposure, run behind a TLS-terminating reverse proxy. See Security.

Content types & encoding

  • Request bodies require Content-Type: application/json; charset=utf-8 on every POST/PUT with a body. A wrong or missing type returns 415 unsupported_media_type.
  • Response bodies are application/json, except the SSE stream, which is text/event-stream.
  • data payloads are arbitrary JSON (object, array, string, number, bool, null), stored and returned verbatim — never parsed, indexed, or validated for shape. For binary, put a base64 string in data and a hint in meta.
  • Compression is honored both ways (Accept-Encoding: gzip / Content-Encoding: gzip) but off by default — on a local-NVMe single-machine deployment the bottleneck is CPU, not bandwidth. Enable it only for large bulk writes over a slow link.
  • Timestamps and durations are integer milliseconds ($ts, *_ms). There are no string dates anywhere.
  • $seq is a u64 rendered as a JSON number. It fits in an IEEE-754 double well beyond any single topic’s lifetime; clients SHOULD still parse as a 64-bit integer.

The $-prefixed metadata convention

Server-computed, per-record metadata is returned under $-prefixed keys so it can never collide with your content. data and meta are your namespaces (pure passthrough, same key in both directions). On write, you set node/tag as plain top-level keys; on read, the server echoes them as $node/$tag to signal they are now server-canonical and immutable.

FieldTypeMeaning
$sequ64Server-assigned sequence id.
$tsi64 (ms)Server commit time.
$nodestringOrigin node id supplied by the writer (loop prevention). Omitted when absent.
$tagstringTag supplied by the writer (deletion-match key). Omitted when absent.
_(SSE distinguishes payload kinds by the event nameevent: record, event: tombstone,
event: caught-up, … — not by a $type field. Record payloads carry the keys above plus
data/meta; there is no $type key on the wire.)_

$node, $tag, and meta are omitted from a response object when absent — absence, not null. data is always present (and may be JSON null).

The canonical error body

Every non-2xx response (except in-stream SSE errors) carries this exact shape:

{ "error": { "code": "topic_not_found", "message": "topic \"orders\" does not exist", "detail": { "topic": "orders" } } }
FieldMeaning
error.codeStable, machine-readable snake_case string. Branch on this.
error.messageHuman-readable; may change between versions; never parse it.
error.detailOptional structured context. May be absent.

Success responses carry bare data — there is no {"status":"ok"} envelope. The presence of a top-level error key is the only success/failure discriminator.

Tombstones and gaps are NOT errors. Cap eviction and TTL expiry surface as in-band 200 payload signals — a tombstone object in a diff, an event: tombstone frame in SSE. Data loss is always explicit but never an HTTP error. See Tombstones.

Status codes

CodeMeaningerror.code examples
200OK (read, idempotent write / create / delete)
201Created (topic / router created on this call)
400Malformed request (bad JSON, bad type, value out of range)invalid_request, batch_too_large, record_too_large
401Missing / invalid bearer tokenunauthorized
403Authenticated but lacks the required scope, or name is outside the prefix allowlistforbidden
404Topic / router does not exist (and was not auto-created)topic_not_found, router_not_found
405Wrong method for pathmethod_not_allowed
406Accept is not text/event-stream on an SSE GETnot_acceptable
409Conflict: router cycle, config conflict, queue op on a non-queue topicrouter_cycle, topic_exists_incompatible, topic_not_empty, not_a_queue
413Body exceeds the server hard limit (pre-parse)payload_too_large
415Wrong / missing Content-Typeunsupported_media_type
422Semantically invalid (write to a full discard:"reject" topic)topic_full
429Elastic throttle under CPU pressure, or a resource cap reachedthrottled
500Internal error (a bug)internal
503Not ready (WAL replay on boot) / shutting downnot_ready, shutting_down

429 throttled carries a Retry-After header. A CPU-pressure throttle adds error.detail.retry_after_ms; a resource-cap rejection adds error.detail.limit. 503 also carries Retry-After. The full list lives on Errors.

Bulk writers that prefer to push through CPU-pressure backpressure may set "disable_backpressure": true in the write body (a trusted-loader opt-out): the server then admits the write but may queue it, trading latency for not failing. See Writing.

Pagination & cursors

Three cursor styles for three read shapes — do not mix them up:

  1. Topic reads are seq-cursored, not opaque. The cursor is from_seq. POST .../diff returns next_from_seq; you pass it back. When fully caught up, next_from_seq == head_seq and "caught_up": true. The server always returns next_from_seq; caught_up is the “done for now” flag — for a live log, “done for now” is not “done forever”.
  2. List endpoints use opaque cursors. GET /v0/topics and GET /v0/routers return next_cursor (opaque base64), present only when more pages exist — its absence means the last page. Pass it back as ?cursor=. page_size defaults to 100, max 1000.
  3. SSE uses a composite cursor. The multiplex watch encodes the per-topic (topic → seq) positions as a base64url-encoded JSON map, usable as both Last-Event-ID and the request cursor field. See Watch.

On a diff read, caught_up — not records.length — is the reliable “no more right now” signal. Node-filtered, deleted, and TTL-expired records advance next_from_seq without appearing in records, so a response can carry zero records while the cursor moved forward. See Ordering & Cursors.

Idempotency

Two complementary mechanisms; there is no global idempotency-token service.

  1. The control plane is idempotent by construction. Create / configure is PUT and upsertable: an identical PUT is a no-op 200; a changed PUT applies the diff. A DELETE of an absent resource returns 200 with "deleted": false.
  2. Writes use an optional idempotency_key. If supplied, the server remembers (topic, idempotency_key) → assigned seqs for idempotency_window_ms (default 120000, per topic). A retried write with the same key in-window returns the original seqs with "deduped": true and does not append again. The key may also be sent as the Idempotency-Key: header (the body field wins).

A log has no “old value,” so a dedup key — not a compare-and-set condition — is the right primitive for safe append retries. See Idempotency.

The performance block

Every JSON response (and most errors) includes a performance object, so latency observability lives in the response, not a side channel:

"performance": { "server_total_ms": 0.41, "wal_append_ms": 0.12, "fsync_ms": 0.0, "records_scanned": 128, "throttle_wait_ms": 0.0 }
FieldWhen presentMeaning
server_total_msalwaysTotal server-side handling time in ms.
wal_append_mswrite pathsTime to enqueue the WAL frame. A memory topic still enters the WAL write path like disk, so this is not always 0 for memory. (Only the pure in-memory test engine, with no WAL, reports 0.)
fsync_msdurable write pathsTime parked on the group fsync; 0 for memory/disk classes (neither is fsync-gated).
records_scannedread pathsRecords examined (including filtered ones).
throttle_wait_msunder pressureTime parked behind the elastic scheduler.
cold_segments_readcold readsCold-tier segments touched to satisfy the read.

Fields are best-effort and additive; clients must tolerate any subset. See Performance for how these numbers are produced.

The Topic config object

This object appears in topic-create requests and topic-state responses. All fields are optional on create; an omitted field takes the documented default. An empty {} creates an all-default topic.

{ "type": "log", "ttl_ms": 0, "cap_records": 0, "cap_bytes": 0, "discard": "old", "durable": false, "durability": "disk", "priority": null, "auto_priority": true, "auto_create": true, "idempotency_window_ms": 120000, "dedupe_node": true, "lease_ms": 30000, "claim_jitter_ms": 0, "max_deliveries": 0, "dead_letter": null, "leases_durable": false }
FieldTypeDefaultMeaning
type"log" | "queue""log"Topic kind. "log" is the plain append-only log. "queue" additionally enables the claim / ack / nack / extend / work endpoints. The five queue-tuning fields below are only meaningful when type:"queue" (accepted but inert on a log). type is immutable — a PUT changing it returns 409 topic_exists_incompatible.
ttl_msu640 (off)Records older than this (by $ts) are not delivered. 0 = no TTL. Crossing a consumer’s cursor yields a tombstone.
cap_recordsu640 (off)Max retained record count. On overflow the topic evicts per discard. 0 = unbounded.
cap_bytesu640 (off)Max retained payload bytes. 0 = unbounded. Whichever of cap_records/cap_bytes is hit first triggers eviction.
discard"old" | "reject""old"Full-topic policy. old = evict oldest (pub/sub friendly). reject = refuse the write with 422 topic_full so durable queues fail loudly rather than drop unconsumed work.
durableboolfalseShorthand alias for durability. On create: true ⇒ class fsync, false ⇒ class disk (consulted only when durability is absent). On every response it is reported as durable == (durability == "fsync"). Prefer durability when selecting ephemeral or memory.
durability"ephemeral" | "memory" | "disk" | "fsync"resolved from durableThe commit class. ephemeral: resident-only records; topic config persists, but appends/deletes skip the record WAL and HOT segment path, are queryable while the process is running, and are intentionally empty after restart. memory: “disk-like but best-effort” — takes the same group-committed WAL write + recovery path as disk and is fully queryable, but with no durability guarantee (records MAY survive OR be lost on restart; config always persists; never fsync-gated, so fsync_ms is 0). disk: WAL + group commit, acked on frame enqueue (not fsync-gated), survives a crash minus the un-fsynced tail. fsync: ack is fsync-gated, survives any crash. An explicit durability always wins; otherwise derived from durable. ephemeral and memory are reachable only by setting durability explicitly.
priorityi32 | nullnullManual delivery priority (higher served first under pressure), clamped [-1000, 1000]. null ⇒ use auto-priority.
auto_prioritybooltrueWhen priority is null, derive effective priority from recency of the last read / SSE / state call on this topic. A manual priority always overrides.
auto_createbooltrueWhether a write to this topic name may lazily create it. A per-write create flag can override downward.
idempotency_window_msu64120000How long (topic, idempotency_key) dedupe state is retained.
dedupe_nodebooltrueWhether node loop-prevention filtering is enabled on reads of this topic. Exposed so a topic can opt out of node filtering entirely.
lease_msu6430000Queue only. Default lease (visibility-timeout) duration for a claim, clamped [100, 86400000]. A per-claim lease_ms overrides it.
claim_jitter_msu640 (greedy)Queue only. Coalescing-window width, clamped [0, 5000]. 0 = serve each claim immediately. >0 = gather the cohort of claimers in the window and divide jobs evenly.
max_deliveriesu640 (off)Queue only. Dead-letter a job after this many deliveries without an ack. 0 = unlimited redelivery.
dead_letterstring | nullnullQueue only. Topic to move a job to when it exceeds max_deliveries. null = no dead-letter topic. Must differ from this topic.
leases_durableboolfalseQueue only. Durability of the leases log. Defaults false because losing leases on a crash is self-healing — in-flight jobs simply become claimable again. The jobs log durability is governed by durable; ack durability == topic durability.

Never compute a topic’s retained floor from head_seq - cap_records. Eviction is segment-granular and lazy (it may transiently exceed the cap), and deletes advance earliest_seq independently. Read earliest_seq from the server. See Topics and Tombstones.

See also

  • API Reference overview — the full endpoint index.
  • Topics — the topic lifecycle endpoints that use this config object.
  • Errors — the complete error.code → HTTP-status catalog.
  • Security — auth, scopes, and the prefix allowlist in full.
  • Durability — the ephemeral/memory/disk/fsync commit classes.
Last updated on