Queues

A queue is a topic created with type:"queue". It adds lease-based, at-least-once job delivery on top of the same persistent log machinery — the WAL, crash recovery, SSE, priority, tombstones, and node filtering all still apply. A queue is a topic, so everything in the rest of the API (writing, reading, deletion, watch) works on it read-only; the endpoints on this page add the lease lifecycle on top.

The model: two logs

Internally a queue is two append-only logs:

• The jobs log — the topic itself. You POST records into it (Writing); each record is a job, identified by its $seq. This is the source of truth for what work exists. Its durability follows the topic’s durable/durability config.
• The leases log — an append-only log of lifecycle events (claimed, released, extended, acked) describing who holds what. The live who-holds-what state is the materialized projection of this log, held in memory and rebuilt on restart. The leases log is event-sourced, not a separately-mutated table.

You produce jobs with a normal append. A worker claims up to N jobs; the server leases them to that worker’s node and returns each job’s data plus a lease_id and a deadline. The worker acks to complete (ack is the permanent delete), nacks to release for immediate or delayed reclaim, or extends the lease to keep working. If a lease’s deadline passes with no ack or extend, the job becomes claimable again — the visibility timeout. After too many redeliveries a job can be dead-lettered.

At-least-once with idempotent consumers

The contract is at-least-once. A slow-but-alive worker whose lease expires while it is still processing will have its job reclaimed and possibly delivered elsewhere; if the slow worker later acks past its deadline, that ack is skipped — but the job was already processed twice. Duplicates are inherent and documented. Consumers MUST be idempotent (dedupe on $seq or a job-level key in data/meta).

Lease durability is best-effort by design

The jobs log is durable per the topic config — work is never lost. The leases log defaults non-durable (leases_durable:false) because losing leases on a crash is self-healing: on restart, a job with no replayed active lease is simply claimable again, which is exactly correct visibility-timeout behavior. A transient WAL error on a lease append degrades to the baseline at-least-once (reclaim on timeout) rather than losing or duplicating work beyond what at-least-once already permits. Ack durability == the topic’s durable: an acked+deleted job stays gone across a crash iff its delete was durable.

Creating a queue

Set type:"queue" on a topic-create. The queue tuning fields go in the same config object (see the topic config table). type is immutable once set — a PUT that changes it returns 409 topic_exists_incompatible.

# A durable queue that fails loudly when full, with dead-lettering after 5 deliveries.
curl -X PUT $TOPICS/v0/topics/transcode \
  -H 'content-type: application/json' \
  -d '{ "type": "queue", "durable": true, "discard": "reject",
        "lease_ms": 30000, "claim_jitter_ms": 0,
        "max_deliveries": 5, "dead_letter": "transcode.dlq" }'

Produce jobs with a normal write — the queue is just a topic:

curl -X POST $TOPICS/v0/topics/transcode \
  -H 'content-type: application/json' \
  -d '{ "records": [ { "data": { "type": "transcode", "src": "s3://uploads/clip-8800.mov", "preset": "h264-1080p" },
                       "tag": "clip-8800" } ] }'

POST /v0/topics/:q/claim — lease jobs to a worker

Lease up to max claimable jobs to a worker node. A job is claimable iff it is not acked (still in the jobs log) and not currently leased (no active lease, or its lease has expired). Each returned job records a claimed event and increments that job’s delivery counter.

Request body

{ "node": "api-fra-1", "max": 16, "lease_ms": 30000 }

Response (200)

{
  "topic": "transcode",
  "claimed": [
    { "$seq": 480101, "lease_id": "lease_7f3a9c", "deadline": 1748450039000,
      "$ts": 1748450001000, "$tag": "clip-8800", "deliveries": 1,
      "data": { "type": "transcode", "src": "s3://uploads/clip-8800.mov", "preset": "h264-1080p" }, "meta": { "trace": "z9" } },
    { "$seq": 480104, "lease_id": "lease_7f3a9d", "deadline": 1748450039000,
      "$ts": 1748450001050, "deliveries": 2,
      "data": { "type": "thumbnail", "src": "s3://uploads/clip-8804.mov", "at_sec": 5 } }
  ],
  "count": 2,
  "ready": 840,
  "performance": { "server_total_ms": 0.42, "throttle_wait_ms": 0.0 }
}

Errors — 400 invalid_request (missing node, bad max/lease_ms type); 404 topic_not_found; 409 not_a_queue; 429 throttled.

POST /v0/topics/:q/ack — complete jobs (ack is the delete)

Complete jobs. The ack is the delete: the server records an acked event and removes each seq from the jobs log via the permanent delete path. Ack durability equals the topic’s durable — an acked+deleted job stays gone iff its delete was durable.

Request body

{ "node": "api-fra-1", "seqs": [480101, 480104], "lease_ids": ["lease_1a2b", "lease_3c4d"] }

Only seqs currently leased to node are acked. A seq that is not leased to node (never claimed, already acked, or its lease expired and was leased to someone else) is silently skipped and reported in skipped — ack is idempotent and safe to retry. This is the at-least-once seam: a worker acking past its deadline may find another worker now holds the lease, so its ack is skipped and the job may run twice.

Response (200)

{ "topic": "transcode", "acked": 2, "skipped": [],
  "ready": 840, "in_flight": 286,
  "performance": { "server_total_ms": 0.30, "fsync_ms": 0.21 } }

fsync_ms > 0 only on a durable queue (the delete is fsynced before the ack returns).

Errors — 400 invalid_request (missing node/seqs, bad seq type, seqs longer than 1000 ⇒ batch_too_large); 404 topic_not_found; 409 not_a_queue.

POST /v0/topics/:q/nack — release leased jobs

Release leased jobs back to the queue for immediate or delayed reclaim, without an ack. Records a released event. Semantically identical to letting the lease expire, just sooner (or after delay_ms).

Request body

{ "node": "api-fra-1", "seqs": [480104], "delay_ms": 0, "lease_ids": ["lease_3c4d"] }

A nack drops the active lease and makes the seq claimable again at now + delay_ms, incrementing the delivery counter on its next claim (and subject to dead-lettering).

Response (200)

{ "topic": "transcode", "nacked": 1, "skipped": [],
  "ready": 841, "in_flight": 285,
  "performance": { "server_total_ms": 0.18 } }

Errors — 400 invalid_request; 404 topic_not_found; 409 not_a_queue.

POST /v0/topics/:q/extend — push out a lease deadline

The heartbeat for long jobs. Push out the deadline of held leases. Records an extended event. Extend sets, not adds — the worker asserts “I need this much more time from now.”

Request body

{ "node": "api-fra-1", "seqs": [480101], "lease_ms": 30000, "lease_ids": ["lease_1a2b"] }

A seq whose lease has already expired (and was reclaimed) cannot be extended — it is skipped; the worker should re-claim. Extending does not change the delivery counter.

Response (200)

{ "topic": "transcode", "extended": 1, "skipped": [],
  "deadlines": { "480101": 1748450069000 },
  "performance": { "server_total_ms": 0.12 } }

Errors — 400 invalid_request (missing lease_ms); 404 topic_not_found; 409 not_a_queue.

Visibility timeout: lazy reclaim, self-healing

A lease carries an absolute deadline. Once now > deadline the lease is expired and the seq becomes claimable again — this is the visibility timeout. There are no per-job timers: expiry is lazy. Expired-lease seqs are collected onto a reclaim freelist that the next claim pass drains first, before handing out fresh jobs, so a reclaimed job jumps the queue ahead of never-delivered ones. Each reclaim increments the job’s delivery counter on its next claim and is subject to dead-lettering.

Because the live lease state is a purely-derived projection of a (default non-durable) leases log, losing the leases log on a crash is self-healing: on restart the projection is rebuilt from whatever lease events survived (possibly none), so any job without a replayed active lease is immediately claimable — identical to every lease having expired. No job is lost (the jobs log is durable per config); at worst an in-flight job is redelivered, which at-least-once already permits.

Coalescing window & even division

claim_jitter_ms is a fairness/coalescing window, not a backoff.

• claim_jitter_ms = 0 (default, greedy). A claim is served immediately, lowest latency. First-arrival drains the head of the available set.
• claim_jitter_ms > 0 (coalescing). A claim waits up to that window. The server gathers every claimer that arrived during the window into a cohort, then in one batched coordinator pass (a single critical section) divides the available jobs evenly across the whole cohort — round-robin, proportional to each claimer’s max. This is not first-arrival-drains-the-head: ten workers each asking for max:10 against 50 available jobs get ~5 each, not 10 / 10 / 10 / 10 / 10 / 0 / 0 / 0 / 0 / 0.

The available set for a pass is, in order: (1) the reclaim freelist (expired-lease and elapsed-delay_ms seqs) drained first, then (2) fresh jobs handed out by a monotonic claim cursor over the jobs log. The single-pass cohort design is also a scalability win — one coordinator pass under one lock instead of N independent per-claim atomic races over the head. The /work SSE streams participate in the same cohort, so polling claimers and pushed workers are balanced together.

Dead-lettering

Each job carries a delivery counter, incremented on every claim (including reclaims of expired leases and re-claims after a nack). When a job is about to be delivered for the (max_deliveries + 1)-th time and the topic has a non-null dead_letter, it is not re-delivered. Instead the server moves it: appends the job’s record to the dead_letter topic (preserving $tag/meta/data) and permanently deletes it from the jobs log (the same delete path as an ack). The moved record is stamped with provenance meta:

With max_deliveries = 0 (default) or dead_letter = null, a job is reclaimed forever and never dead-lettered. Each move increments the dead_lettered counter. The dead-letter topic is an ordinary topic (log or queue), so a poison job can be inspected via diff / watch, or re-driven by reading it and re-producing into the source queue. Forwarded / dead-lettered copies honor the destination topic’s durability class — a memory dead-letter topic retains a best-effort copy (it may survive or be lost on restart, same as disk minus the durability guarantee).

Queue observability

GET /v0/topics/:q on a queue returns type:"queue" and a queue sub-object beside the normal topic state:

{
  "topic": "transcode", "type": "queue",
  "head_seq": 480231, "earliest_seq": 479101, "count": 1130,
  "config": { "type": "queue", "lease_ms": 30000 },
  "queue": { "ready": 842, "in_flight": 288, "dead_lettered": 4 },
  "performance": { "server_total_ms": 0.06 }
}

ready + in_flight equals the live job count, modulo jobs whose nack delay_ms has not yet elapsed (counted in neither until claimable). The jobs log stays fully readable via normal diff and watch for monitoring — those paths observe the jobs log and never claim, ack, or mutate leases.

GET /v0/topics/:q/work — auto-claim over SSE (PUSH mode)

A streaming alternative to the claim → process → claim poll loop: the server keeps up to max jobs leased-and-pushed to this one connection, claiming more as the worker acks, and applying backpressure at max in-flight. The stream is one-way (SSE, read+write scope) — the worker still acks (and may nack/extend) via the separate POST endpoints above; the stream only delivers.

curl -N "$TOPICS/v0/topics/transcode/work?node=api-fra-1&max=8" \
  -H 'accept: text/event-stream'

Response headers are the SSE set (Content-Type: text/event-stream, Cache-Control: no-store, X-Accel-Buffering: no). A retry: 2000 and an initial heartbeat are sent at open. The same coalescing-window logic feeds connected /work streams fairly alongside polling claimers.

event: job — one leased job (the streaming analog of one claimed[] entry). id: is the job $seq (an opaque resume hint; the authoritative lease state lives server-side).

id: 480101
event: job
data: {"topic":"transcode","$seq":480101,"lease_id":"lease_7f3a9c","deadline":1748450039000,"$ts":1748450001000,"$tag":"clip-8800","deliveries":1,"data":{"type":"transcode","src":"s3://uploads/clip-8800.mov","preset":"h264-1080p"}}

: hb

The worker processes the job and acks it via POST /v0/topics/:q/ack; on each ack the server claims and pushes replacement jobs to keep the stream at max in-flight. To reject, the worker nacks; to keep a long job alive, it extends. Heartbeats are bare : hb comments on the standard cadence, suppressed when a real job frame went out in the window.

A per-stream event: error carries an HTTP-aligned code (same shape as the watch error frame), e.g. code: 429 advisory pacing under pressure, or code: 409 if the topic ceased to be a queue (deleted + recreated as a log). Terminal errors close the stream; the worker reconnects.

Errors at establishment — 200 (stream opened); 400 invalid_request (missing node, bad max); 404 topic_not_found; 406 not_acceptable (Accept not text/event-stream); 409 not_a_queue; 429 throttled.

See also

• Job Queue guide — end-to-end worker recipe with claim/ack/retry.
• Writing Records — how you produce jobs into a queue.
• Idempotency — why at-least-once needs idempotent consumers.
• Durability classes — what durable/durability buys for jobs and acks.