DTOflow Platform

What DTOflow is

DTOflow is the data-sharing backbone of the Pricer EVO ESL system. Rather than having services call each other's REST APIs, services publish structured data objects (DTOs) to DTOflow and subscribe to changes via the Change Queue Service (CQS). This decouples producers from consumers: a writer publishes a DTO without knowing which services will consume it; a consumer receives a change notification without needing to poll the writer directly.

Contrast with REST coupling:

In a REST-based architecture, Service A calls Service B's API to obtain data. This creates direct coupling: A must know B's address, B's API contract, and B must be available at call time. If B is unavailable or slow, A is blocked.

In DTOflow: Service A writes a DTO to the DTOflow Server. The Change Queue Service delivers a notification to all interested consumers. Each consumer then reads the DTO from the DTOflow Server at its own pace. Services are isolated — neither writer nor reader has any runtime dependency on the other.

This pattern also enables natural fan-out. A single DTO write can trigger notifications to many independent consumers without the writer knowing or caring about any of them. Adding a new consumer is purely a configuration change; the writer is untouched.

The three components

DTOflow is composed of three independent services that together form the platform layer. Each component has a dedicated documentation page.

DTOflow Server (dtoflow-spanner)

The DTOflow Server is the central DTO store. It holds protobuf message instances, each addressed by a stable string ID following a tenant/store hierarchy. Services write DTOs through the gRPC Writer API and read them through the gRPC Reader API. A Docker-based TestServer (with CQS built in) is available for local development and integration testing without cloud dependencies.

See dtoflow-server.md for the full API reference and ID conventions.

Change Queue Service (dtoflow-changequeue)

The Change Queue Service delivers ordered change notifications to consuming services. Each consumer has one named queue. When a DTO is written or deleted, CQS fans out a notification to all queues that have subscribed to that DTO type. Consumers long-poll their queue, receive a batch of change notifications (IDs only), fetch the actual data from the DTOflow Server, process it, and acknowledge the notifications.

See cqs.md for queue configuration, the dequeue pattern, and priority semantics.

Large File System (dtoflow-lfs)

The Large File System is a GCS-backed binary asset store for files that are too large or binary to embed in protobuf DTOs — fonts, design preview images, rendered ESL images. Files are addressed by content hash, making the store naturally deduplicated and immutable. DTOs carry LFS paths as string fields rather than embedding file bytes.

See lfs.md for path conventions and the upload/read API.

How services interact with DTOflow

The typical data flow when a DTO changes:

sequenceDiagram
    participant Writer as Writing Service<br/>(e.g. studio-link-evaluator)
    participant DtoServer as DTOflow Server<br/>(dtoflow-spanner)
    participant CQS as Change Queue Service<br/>(dtoflow-changequeue)
    participant Reader as Consuming Service<br/>(e.g. studio-renderer)

    Writer->>DtoServer: WriterService.Put(dto, sla_timestamp)
    DtoServer-->>Writer: WriteResponse(new_version)
    DtoServer->>CQS: Internal: record change notification
    Reader->>CQS: BatchDequeue(queue_name, max_notifications, wait_time_ms)
    CQS-->>Reader: [DtoChangeNotification(dto_id, dto_type, sla_timestamp, ack_token)]
    Reader->>DtoServer: ReaderService.Read(id) or BatchRead(ids)
    DtoServer-->>Reader: DTO payload
    Reader->>CQS: BatchAcknowledge(queue_name, [ack_tokens])
    CQS-->>Reader: BatchAcknowledgeResponse

Step-by-step:

1. Write — The writing service calls WriterService.Put with the DTO payload and an sla_timestamp indicating the business deadline by which downstream consumers should have processed the change. DTOflow stores the DTO and notifies CQS.

2. Notification fan-out — DTOflow Server records the change internally and notifies CQS. CQS delivers a DtoChangeNotification to every queue that is subscribed to the changed DTO type. The notification contains the DTO ID and type, but not the payload itself.

3. Long-poll dequeue — The consuming service calls BatchDequeue on its named queue. If no notifications are available, the call blocks up to wait_time_milliseconds. When notifications arrive, the service receives a batch; the first item is always the highest-priority (earliest SLA), and the remaining items share the same path prefix (e.g., same store) as the first.

4. Fetch payload — The consumer calls ReaderService.Read or BatchRead on the DTOflow Server using the IDs from the notifications. If a DTO was deleted, the Reader returns NOT_FOUND — there is no separate "delete" notification type; deletion is inferred at read time.

5. Acknowledge — After successfully processing the change, the consumer calls BatchAcknowledge with the ack_tokens from the notifications. This permanently removes them from the queue.

Splitting a heavy notification. When a single notification implies very large fan-out (a design change touching every ESL in a multi-store tenant, a communication-pack change touching every linked ESL), the consumer can give it back instead of processing it inline: a streaming ReEnqueue call atomically acks the original and pushes back a list of smaller, tiered notifications that re-enter the same queue. Urgent traffic then interleaves between the pieces of the original job rather than waiting behind it. See Re-enqueue: splitting a heavy notification.

Core principles

Single ownership. Each DTO type is written by exactly one service. This makes it straightforward to reason about data provenance and avoids conflicting writes. If multiple services could write the same DTO, consistency guarantees would become much harder to provide.

Immutable IDs. A DTO's string ID is assigned at creation and never changes. The content (the protobuf payload) can be updated any number of times via Put, but the ID is stable. Services can safely cache or index DTO IDs.

Tenant/store hierarchy. Most DTO IDs are scoped to a tenant, and optionally to a store within that tenant. Some DTOs (e.g., esltype, eccmodel, eccfont) are global — shared across all tenants. Tenant-scoped IDs never cross tenant boundaries, enforcing data isolation as a first-class property of the data model.

Versioned schemas. DTO schemas evolve in two ways: non-breaking changes (adding fields, updating comments) are tracked with an in-file minor version comment and require no migration. Breaking changes (removing fields, changing semantics, restructuring the message) produce a new .proto file with a new version suffix (e.g., link.v2) and both versions coexist during the migration period.

CQS as the inter-service event bus. All inter-service change delivery flows through the Change Queue Service. Internally CQS uses per-DTO Pub/Sub topics as its persistence backbone — it drains these topics continuously and re-presents the notifications to consumers sorted by SLA priority and, when beneficial, grouped by tenant/store for batch efficiency. From a service's perspective there is one named queue per consumer; subscriptions and delivery guarantees are managed within the platform, visible in config files, and testable locally with the TestServer.