DTOflow Server¶

Overview¶

The DTOflow Server (dtoflow-spanner) is the central storage layer for all DTOs in the Pricer EVO platform. It exposes a Reader API, a Writer API, and a supplementary History API. The storage backend is an implementation detail — the TestServer Docker image provides the same complete gRPC surface for local development and integration testing.

DTO identity and scoping¶

The ID pattern¶

Every DTO has a stable string ID that encodes both its type scope and its position in the tenant/store hierarchy. ID segments are separated by /. The general pattern is:

t/{tenantId}[/s/{storeId}]/{collection}/{identifier}

The leading t/{tenantId} segment scopes all data to a single tenant; the optional s/{storeId} further scopes to a store within that tenant. There are no cross-tenant IDs.

Scope levels¶

Scope	Pattern	Example DTOs
Global (tenant-wide)	`t/{tenantId}/{collection}/{id}`	`t/farmersmarket/fonts/abc123`
Per-tenant	`t/{tenantId}/{collection}/{id}`	`t/farmersmarket/designs/d-9982`
Per-store	`t/{tenantId}/s/{storeId}/{collection}/{id}`	`t/farmersmarket/s/oslo-main/esls/7318690123456`
Singleton	`t/{tenantId}/s/{storeId}/{collection}`	`t/farmersmarket/s/oslo-main/eccparameters`

Singletons are DTOs where at most one instance exists per store (or per tenant). The collection name itself is the full ID — there is no trailing item identifier. The eccparameters DTO is the canonical example: each store has exactly one set of ECC parameters.

Aliases¶

An alias is a secondary lookup path that maps an alternative ID to the canonical DTO. Aliases are declared in the DTO's proto definition and are maintained by the DTOflow Server alongside the primary ID index.

The canonical example is the link DTO, which is primarily keyed by t/{tenantId}/s/{storeId}/links/{linkId}. Because consumers frequently need to look up links by the ESL barcode they point to, the DTO also declares a by_storeesl alias that keys links at t/{tenantId}/s/{storeId}/esls/{barcode}/link. A reader can pass alias_name = "by_storeesl" in a ReadDtoRequest and supply the ESL-scoped path; the server resolves the alias and returns the same DTO as if the canonical ID had been used.

Aliases are also relevant to CQS subscriptions: a queue can subscribe to alias-qualified notifications (e.g., alias_name = "by_storeesl") to receive notifications only when the alias mapping itself changes, not every time the linked DTO's content changes. The canonical application — using a barcode-keyed alias as the deletion trigger for child DTOs whose primary key is not the barcode — is documented in Pattern: alias-watch cleanup.

Schema versioning¶

Minor versions (non-breaking)¶

Adding a field to an existing message, marking a field deprecated, or updating documentation comments are all non-breaking changes. These are tracked with an inline comment inside the message definition:

message Link {
  // minor version 3
  string id = 1;
  string design_id = 2;
  string item_id = 3;           // added in minor version 2
  repeated string tag_ids = 4;  // added in minor version 3
}

No new file is created; the existing .proto is updated in place. Consumers compiled against an older minor version silently ignore unknown fields (protobuf's default behavior), so rolling deployment is safe.

Major versions (breaking)¶

When a change cannot be made backward-compatible — for example, removing a field, renaming a field with semantic meaning, or restructuring the message hierarchy — a new .proto file is created with a new version suffix. The old version remains in the repository and continues to be served.

Example: link.v1 → link.v2

The link.v1 DTO represented links with a flat structure. The PLT-2484 migration (ESL link and image DTO refactor) introduced new variant types that could not be expressed within the existing field layout without breaking existing consumers. A link.v2 DTO was introduced in a new proto file. The CQS queue link-registry continues to consume link.v1 as a legacy bridge, while all new consumers subscribe to link.v2. Both versions are written and served concurrently during the migration window. Once all consumers are migrated, link.v1 is deprecated and eventually removed.

Reader API¶

The Reader API is defined in reader_service.proto (generated from the proto template). It is a read-only, stateless gRPC service.

`Read` — single DTO by ID¶

rpc Read(ReadDtoRequest) returns (DtoMessage)

message ReadDtoRequest {
  string id = 1;         // The canonical DTO ID
  string alias_name = 2; // Optional: use an alias namespace for the lookup
}

Returns the current version of the DTO. Returns NOT_FOUND if the DTO does not exist (including after deletion). Pass alias_name to resolve via a secondary alias index instead of the primary ID index.

`BatchRead` — up to 1000 DTOs in one call¶

rpc BatchRead(BatchReadDtosRequest) returns (BatchReadResponse)

message BatchReadDtosRequest {
  string parent = 1;         // Optional: shared parent for sharding optimization
  repeated string ids = 2;   // Up to 1000 IDs
  string alias_name = 3;     // Optional: alias namespace for all IDs
}

Returns all found DTOs. IDs that resolve to NOT_FOUND are silently omitted from the response — callers must compare the response count against the request IDs to detect deletions. The optional parent field hints to the server which storage partition to route the request to, improving read performance when all DTOs share a common ancestor.

`ReadChildren` — paginated children of a parent¶

rpc ReadChildren(DtoChildrenRequest) returns (QueryDtosResponse)

message DtoChildrenRequest {
  string parent = 1;      // The parent DTO ID whose children to retrieve
  int32 page_size = 2;    // Max items per page; server may return fewer
  string page_token = 3;  // Pagination token from previous response
  string alias_name = 4;  // Optional: resolve children via alias index
}

Returns a page of child DTOs along with a next_page_token. An absent next_page_token in the response signals the last page. Keep all request parameters constant across pages; changing any parameter with a page token in flight produces undefined behavior.

`ListChildren` — IDs only, paginated¶

rpc ListChildren(DtoChildrenRequest) returns (ListChildrenResponse)

Returns only the IDs of child DTOs (not the full message payloads). Use this when you need to enumerate all children before batch-reading, or when you only need the IDs for downstream processing. The response includes alias_name so callers know which alias namespace the returned IDs belong to.

`CountChildren` — count without fetching¶

rpc CountChildren(CountChildrenRequest) returns (CountChildrenResponse)

Returns the number of child DTOs under a given parent without reading any payloads. When alias_name is set, counts alias entries (which may include duplicates if multiple aliases point to the same DTO) rather than unique DTOs.

Writer API¶

The Writer API is a gRPC service for creating, updating, and deleting DTOs. All write operations are idempotent: writing the same DTO twice produces the same final state. Every write request carries an sla_timestamp — the business deadline by which downstream consumers should process this change — and an optional tracking_id for distributed tracing.

`Put` — create or update a single DTO¶

rpc Put(PutDtoRequest) returns (WriteResponse)

The Put request wraps the full DTO message plus sla_timestamp and tracking_id. If a DTO with the given ID already exists, it is replaced atomically. The WriteResponse contains counts of created, updated, and deleted DTOs.

`Delete` — remove a single DTO by ID¶

rpc Delete(DeleteDtoRequest) returns (WriteResponse)

message DeleteDtoRequest {
  google.protobuf.Timestamp sla_timestamp = 1;
  string id = 2;
  string tracking_id = 3;
}

Deleting a non-existent DTO is a no-op (idempotent). Subsequent reads for that ID return NOT_FOUND.

`DeleteChildren` — remove all children of a parent¶

rpc DeleteChildren(DeleteDtoChildrenRequest) returns (WriteResponse)

Atomically deletes all DTOs whose ID has ancestor_id as a path prefix. Useful when removing a store or tenant.

Streaming batch writes¶

For bulk operations, the Writer API provides three streaming RPCs:

PutAll(stream PutRequest) → WriteResponse — Replaces the entire dataset for a DTO type. Any existing DTO of that type not present in the stream is deleted. Intended for global/shared DTO types during a full reload.
PutMany(stream PutRequest) → WriteResponse — Creates or updates multiple DTOs from a stream without deleting any existing DTOs. Suitable for incremental updates.
PutAllChildren(stream PutRequest) → WriteResponse — For each unique parent of the DTOs in the stream, replaces all of its existing children with the children present in the stream. Intended for managing DTOs within a specific scope (e.g., a store's entire item catalogue).

`PutOrDeleteMany` — mixed create/update/delete stream¶

rpc PutOrDeleteMany(stream CreateUpdateOrDeleteRequest) returns (WriteResponse)

Each message in the stream is a oneof that is either a full DTO (create/update) or a deleted_id string (delete). This is the most flexible bulk write method and is used when a single reconciliation pass may need to both add new DTOs and remove obsolete ones.

SLA timestamp semantics¶

The sla_timestamp on every write request is the primary sort key used by CQS when prioritizing change notifications. A timestamp in the near future (or past) causes the notification to be delivered with high urgency. A timestamp far in the future allows the change to be processed at a lower priority. Setting sla_timestamp accurately is important: if every write uses the current time, CQS cannot distinguish urgent user-initiated changes from background batch operations.

History service¶

DTOflow Server retains every version of every DTO ever written. The History API exposes this audit trail.

service HistoryService {
  rpc ReadHistoric(ReadHistoricDtoRequest) returns (DtoMessage);
  rpc GetHistory(GetHistoryRequest) returns (GetHistoryResponse);
  rpc StreamHistory(StreamHistoryRequest) returns (stream HistoryEntry);
  rpc TailHistory(TailHistoryRequest) returns (stream HistoryEntry);
}

ReadHistoric returns the DTO payload at a specific point in time. Useful for debugging and for reconstructing past state during incident analysis.
GetHistory returns a list of HistoryEntry records — change type (NEW, CHANGED, DELETED), timestamp, and change metadata — for a single DTO, optionally filtered to a minimum version.
StreamHistory streams history entries for all DTOs under a parent, within a version range. Supports wait_if_none for waiting until a minimum version is available.
TailHistory streams from a given version forward, with an optional follow flag to keep the stream open and deliver new changes as they arrive. This is useful for building local caches or change-driven pipelines that need to stay in sync with a parent's entire history.

Integration testing¶

The DTOflow TestServer is a Docker image that implements the full gRPC surface (Reader, Writer, History, CQS, LFS) in memory. Services pull this image in their docker-compose files for integration tests and for local development when running without cloud credentials.

The TestServer is functionally identical to production for all API calls. It correctly enforces ID scoping, alias resolution, and CQS fan-out — making it sufficient for end-to-end integration tests.