Skip to main content

Leases, acks, and delivery counts

The lease model​

Consume returns one or more messages plus a single lease_id. That lease_id grants exclusive in-flight ownership of every returned version until one of these happens:

  • The consumer sends Ack { version, lease_id }. Version is now terminal-Acked.
  • The consumer sends Nack { version, lease_id, delay_ms }. Version returns to Available, optionally with a redelivery delay.
  • The consumer sends Extend { version, lease_id, additional_visibility_ms }. Deadline pushes out by additional_visibility_ms, capped at the queue's visibility_timeout_ms.
  • The deadline (server_timestamp_at_lease + visibility_timeout_ms) passes without any of the above.

When the deadline passes, no sweeper or background thread reaps. The next plan_consume sees the lease is expired and re-folds the version back into Available. Expiry as a fold rule. There is no in-memory timer to lose on failover.

delivery_count is durable​

Every version carries a delivery_count derived from the durable log:

delivery_count(v) = number of DISTINCT lease_ids that have ever covered v
  • Fresh Lease event: +1.
  • Extend reuses the same lease_id: no bump.
  • Nack does not bump on its own. The next fresh Lease does.

This matters. Failover-safe attempt counts are the difference between "park after 5 real failures" and "park after 5 reconnects." Kurrent's persistent subscriptions reset their retry counter on leader change (source). Redis and River strand poison messages entirely when they drop from the in-memory delivery list. Celeriant Queue counts from the log, so the count survives anything a fsync survives.

When delivery_count > max_delivery_attempts, the next consume routes the message per the queue's dlq_strategy.

Visibility timeout​

visibility_timeout_ms is per-queue. Set it long enough for the slowest legitimate processing, plus margin. Too short and leases expire mid-process, bumping delivery_count on a healthy message. Too long and a crashed consumer's work waits N seconds before another consumer can pick it up.

Recipe: pick the p99 of your real processing latency, multiply by 3.

If you can't predict it, use Extend from inside the consumer when you're still alive but not done. Same lease_id, no delivery_count bump. Safe to call as a heartbeat.

Ack batching​

AckRequest carries Vec<AckHandle>. The handler batches the entire request into one durable ControlEvent::AckBatch write. N acks per request, one fsync. Inside-request amortisation. NackRequest gets the identical treatment: one durable NackBatch event per request, mirroring AckBatch.

There is no cross-request coalescing window today. Each Ack RPC pays one fsync. The window is a small future optimisation that would fold acks from independent RPCs landing within a few ms into one event.

Idempotency on produce​

Produce carries one client_id per request and a client_seq per message. The storage layer enforces strict monotonicity: each new message's client_seq must be greater than everything previously committed for that client_id.

A replay of an already-durable client_seq (e.g., after a TCP reset where the client retried the batch) is not an error. The entry is skipped, and its slot in ProduceResponse.assigned_versions comes back None; new entries in the same batch still commit and come back Some(version). Same seq = same message, so a blind retry of the whole batch self-heals. Kafka-style idempotent producer. Never a silent duplicate write, never a spurious error on retry. The integration test produce_idempotency_replays_same_versions_across_reconnect is the canonical demonstration.

If you genuinely lost the in-flight response and don't know whether the write committed, just retry the batch and read assigned_versions. None means it was already on disk.

Ack-hole policy​

A queue's max_ack_holes config caps the number of coalesced range gaps in the ack set. When the count exceeds the cap, Consume returns AckHoleCapExceeded. Block-on-overflow.

The cap is a back-pressure signal, not silent data loss. Drain the holes (ack the missing versions, or wait for their leases to expire and re-deliver) and Consume resumes. Pulsar-style. Redis, River, and SQS-shaped queues accumulate holes until memory blows up.