ADR-010 — Operator Trigger APIs for Force Re-processing¶

Status: Accepted
Date: 2026-06
Tickets: PLT-2486

Context¶

Operators (devops, support) occasionally need to force a service to re-process records without a real data change having occurred. Canonical cases:

Force re-render after a bug fix is deployed: the input DTOs are unchanged, but the service's output was wrong and must be regenerated.
Force re-export after a hardware reset: the output DTOs are correct, but hardware missed the delivery and needs it resent.
Model refresh: an ECC model's rendering logic has changed in ways the platform cannot detect (e.g. an external rendering library was upgraded). All links rendered with that model must be re-rendered.

The driving example for this ADR is PLT-2486: "Re-render all links with eccmodel X".

Options Considered¶

Option B — Generic CQS side-door (`InjectNotification`)¶

A new CQS RPC that lets operators enqueue synthetic notifications directly onto any queue, bypassing the normal DTOflow write-triggered flow.

Rejected. Notifications injected this way are indistinguishable from real DTO-write-driven ones. Services reason about their notification stream as a faithful projection of the DTO store's write history; fabricated entries break that invariant. ADR-009 blocks parent_id injection by external callers for exactly this reason — the same logic applies here. Additionally, operators would need CQS-internal knowledge (queue names, DTO type routing) to use such an API.

Option C — DTOflow touch (`TouchDto`)¶

A new TouchDto(dto_type, dto_id[]) RPC on DTOflow Server re-writes the current DTO value (same payload, bumped timestamp), causing CQS to deliver a normal notification to all subscribers.

Not sufficient on its own. A touch travels exactly one hop. In the model-refresh example:

Touch eccmodel.v1 → CQS notifies ecc-image-render-service
The service's subscription to eccmodel.v1 performs only a Caffeine cache eviction — it does not re-render
Re-rendering only happens when an ecclink.v1 notification arrives for the affected links

To force re-rendering you would also need to touch every ecclink.v1 record referencing that model. There is no mechanism to propagate "this is a forced trigger, keep passing it on" through the pipeline — each hop would need explicit handling. TouchDto remains useful for simple single-hop cases where the subscriber's logic already does the desired work.

Option A — Per-service trigger API with `SelfEnqueue` ✓ (chosen)¶

Each service exposes a named, typed gRPC operation for each supported trigger case. The handler calls a new CQS SelfEnqueue RPC to inject a small number of parent_id notifications onto the service's own queue. The normal dequeue loop then expands those via ReEnqueue — the same 2-stage rocket used for real heavy changes.

Decision¶

Use per-service trigger APIs backed by CQS.SelfEnqueue.

Why not resolve scope in the API handler¶

An obvious implementation of TriggerModelRefresh would query all affected ecclink IDs inside the handler and call ReEnqueue directly. This works for eccmodel (store-scoped, bounded), but breaks for tenant-scoped triggers: TriggerDesignRefresh could imply millions of ESLs across hundreds of stores. Resolving that synchronously in the handler is O(millions), blocks the call, and floods the queue at once — exactly what the 2-stage rocket was designed to avoid.

Instead, the handler injects 1–N parent_id notifications (one per store, or one per tenant) and lets the existing ReEnqueue machinery handle expansion. The API call is always O(1) regardless of affected record count.

Why `SelfEnqueue` is not Option B¶

The key distinction from a generic CQS side-door:

Restricted target. SelfEnqueue only allows a service to inject into its own registered queue. CQS authenticates the caller and rejects any call where queue_name does not match the calling service's identity. A service cannot inject into another service's queue.
Validated at the trigger API. The named trigger RPC enforces scope, parameters, and business rules before calling SelfEnqueue. The injected notifications are not arbitrary — they are the exact parent_id shapes the service already handles on real changes.
parent_id is already synthetic. ReEnqueue produces parent_id notifications today; they do not correspond to a specific DTO write. SelfEnqueue is a constrained extension: same notification shape, same handling code, service-identity gated.
No new handling code. The service processes SelfEnqueue-injected notifications through the same dequeue handler as real ones. No "is this a force trigger?" branching is needed anywhere.

`SelfEnqueue` RPC¶

rpc SelfEnqueue(SelfEnqueueRequest) returns (SelfEnqueueResponse);

message SelfEnqueueRequest {
  string queue_name  = 1;         // must match caller's registered service identity (enforced by CQS)
  repeated SplitItem items = 2;   // reuses SplitItem from ReEnqueue
  string reason      = 3;         // required; written to CQS audit log
  string tracking_id = 4;         // required; provided by the trigger API caller for log correlation
  google.protobuf.Timestamp sla_timestamp = 5;  // optional; defaults to now
}

message SelfEnqueueResponse {
  int32 items_enqueued = 1;
}

SplitItem is the existing message from ReEnqueue:

message SplitItem {
  string dto_type = 1;
  oneof urn {
    string dto_id    = 2;
    string parent_id = 3;
  }
  string alias_name = 4;
}

The tracking_id is propagated to all injected DtoChangeNotifications, so the operator's original request can be traced end-to-end through the processing logs.

Consequences and guidelines¶

Engineering cost¶

Each trigger case requires a named RPC on the relevant service. There is no generic solution that works across all services for free. The initial set will be small and grows organically.

Naming¶

Name the operation, not the mechanism:

TriggerModelRefresh      ✓
InjectEcclinkNotifications  ✗

`reason`, `dry_run`, `tracking_id`¶

All trigger RPCs should include: - reason — free-text, forwarded to SelfEnqueue and the service's own logs - dry_run — validate scope and return a count without enqueuing anything - tracking_id — caller-provided, forwarded to SelfEnqueue for end-to-end tracing

These are cheap to add at definition time and painful to retrofit.

Scope must match the name¶

A TriggerModelRefresh for model X must not silently re-render records from model Y, even if they share a code path.

Example: PLT-2486 — Re-render all links with eccmodel X¶

ecc-image-render-service exposes:

rpc TriggerModelRefresh(TriggerModelRefreshRequest) returns (TriggerModelRefreshResponse);

message TriggerModelRefreshRequest {
  string tenant_id   = 1;
  string store_id    = 2;
  string model_id    = 3;
  string reason      = 4;
  string tracking_id = 5;
  bool   dry_run     = 6;
}

Handler sketch:

Validate inputs; if dry_run, query the by_eccmodel alias index for the count and return without enqueuing.

Call CQS.SelfEnqueue:

items = [{
  dto_type   = "ecclink.v1",
  parent_id  = "t/{tenant_id}/s/{store_id}/eccmodels/{model_id}",
  alias_name = "by_eccmodel",
}]
reason      = request.reason
tracking_id = request.tracking_id

Return items_enqueued = 1.

Dequeue handler (existing + new branch):

When the service dequeues a parent_id notification with alias_name = by_eccmodel, it calls ReEnqueue to split into individual ecclink.v1 dto_id notifications — identical to how it would handle a real parent notification for that alias. The normal render path executes for each. The output (ecceslimage.v1) is byte-for-byte what a real model change would have produced.

ecclink.v1 already carries the by_eccmodel alias — no proto change is required.

Example: Studio renderer — Tenant-scoped design refresh (why the 2-stage rocket matters)¶

A design is tenant-scoped: one TriggerDesignRefresh could affect millions of ESLs across hundreds of stores. Resolving scope in the API handler would be O(millions) and would flood the queue in one shot. With SelfEnqueue:

Handler injects 1 parent_id notification: {dto_type: "studiolink.v1", parent_id: "t/{tenant_id}/designs/{design_id}"}.
Dequeue: renderer receives it → ReEnqueue splits into 500 per-store parent_id notifications.
Dequeue (×500): each per-store notification → ReEnqueue splits into 10k per-ESL dto_id notifications.
Rendering proceeds normally; urgent single-ESL work interleaves by SLA priority throughout.

This is identical to how a real design change fans out today — no new handling code required.