Skip to content

ADR-009: CQS Re-enqueue Streaming API

Status: Accepted Date: 2026-06

Context

A CQS consumer can dequeue a notification whose unit of work runs into the minutes. The pathological case is a design.v1 change in a multi-store tenant: ~500 stores × ~10 000 ESLs each = ~5 000 000 label re-renders, all implied by a single notification. Two bad options exist without a re-enqueue capability:

  1. Process inline under one ack lease. Any mid-job failure forces the whole batch to be redone after lease expiry. The lease itself has to be artificially long, weakening redelivery guarantees.
  2. Process inline anyway. While the consumer is "under water" on this low-SLA work, urgent high-SLA notifications pile up behind it for the entire duration.

DtoChangeNotification.urn already supports a parent_id form intended for "re-evaluate the children of this parent". What was missing is the RPC that produces such tiered notifications while atomically acking the heavy original.

Two further constraints shape the API:

  • Arrays do not scale. A unary RPC with a repeated SplitItem items field cannot carry millions of items without violating the gRPC max-message size and buffering the full list on both ends.
  • The give-back is asymmetric. Re-enqueuing items without acking the original is recoverable (dedup + lease expiry). Acking the original without enqueuing the replacements would lose work.

Decision

A new client-streaming RPC ReEnqueue(stream ReEnqueueRequest) returns (ReEnqueueResponse) on DtoChangeQueueService, with the following shape and rules:

  • Stream framing. ReEnqueueRequest is a oneof of three payloads: exactly one Header{queue_name, ack_token} first, zero or more SplitItem{dto_type, oneof(dto_id|parent_id), alias_name}, and exactly one Footer{} last. Footer is the commit signal — the server only acks the original after observing it.
  • Atomicity. Items already flushed survive client disconnect; the original returns to the queue via the normal lease-expiry path. Ack-without-items cannot occur because the ack runs only after the final batch of items is durably enqueued. Consumer retries are idempotent via existing (dto_type, dto_id, alias_name) dedup.
  • Inheritance. sla_timestamp and tracking_id on each enqueued item are inherited from the original. ack_token is minted at the eventual re-dequeue. Items do not carry these fields.
  • Lease extension. While the stream is open, the server re-stamps the original's lease every 10 s of wall clock or every 10 000 items, whichever comes first. Hard-coded for v1.
  • Routing. Split-items always land in the originating queue. Cross-queue routing is not supported.
  • Zero-item form. A stream of Header + Footer with no items is valid and observably equivalent to a single-token BatchAcknowledge. No separate Nack RPC.
  • Field forms on a SplitItem. dto_id, dto_id + alias_name, parent_id, parent_id + alias_name. The only field-level rejection is "neither dto_id nor parent_id set".

Consequences

  • Easier: Heavy fan-out notifications can be split into priority-tiered work; urgent traffic interleaves between the pieces of a large job. Mid-stream failures no longer redo all preceding work — the per-store / per-link items that were enqueued are present when the original is re-delivered, and the consumer skips ahead.
  • Harder: CQS gains a dedicated server thread pool for re-enqueue streams (one thread per call, modelled on the existing dequeue pool) and a per-session lease-extension cadence. Consumers must implement the streaming protocol correctly — Footer is the commit signal, not onCompleted().
  • Trade-off: PubSub publishers cannot emit parent_id directly. parent_id remains a CQS-internal concept, produced only by ReEnqueue and visible only on dequeue output. This keeps the public publisher contract unchanged.

Alternatives considered

  • Unary RPC with repeated SplitItem. Rejected: violates the bounded-memory constraint and breaks at the gRPC max-message-size limit for the worst case.
  • Bidirectional streaming with server-issued credits. Deferred: HTTP/2 per-stream flow control already exerts backpressure, and the only adopting service is co-located. Bidi can be added additively if credit shaping later proves valuable.
  • Implicit commit on onCompleted() (no explicit Footer). Rejected: makes "incomplete stream" indistinguishable from "intentional empty re-enqueue (nack)". An explicit Footer makes the commit a positive client signal.
  • Separate Nack RPC for the zero-item case. Rejected: adds API surface for no benefit. The zero-item form of ReEnqueue covers it.
  • Carry sla_timestamp / tracking_id per SplitItem. Rejected: opens the door to inconsistent items and an ambiguous priority story. The tiered-SLA effect ("finish one branch before splitting the next") falls out naturally because the queue ranks longer ids ahead of shorter ones at equal SLA.