ADR-001: DTOflow for Inter-Service Data Sharing

Status: Accepted Date: 2025-01

Context

The Pricer EVO platform is a multi-service system where services need to share entity data — ESL assignments, rendered images, store configuration, item values, and link state. The naive approach of each service exposing its own REST API for peers to query creates tight coupling: every service must know the network address and API shape of every other service it depends on, retry logic must be duplicated everywhere, and scaling or redeploying one service risks breaking others.

The platform already operates a DTOflow infrastructure (Spanner-backed, with per-DTO read and write gRPC services). DTOflow makes entity state available as a persistent, versioned stream of records that any downstream service can subscribe to. The Change Queue Service (CQS) adds durable, consumer-owned notification queues on top of DTOflow, so that a service can restart and resume processing without missing notifications.

Direct REST calls between services were used in early integrations and proved problematic: callers needed to handle service unavailability, ordering was not guaranteed, and there was no built-in mechanism for a new consumer to receive historical state on startup.

Decision

Services on the EVO platform share entity data exclusively through DTOflow and CQS. A service that produces data writes it to a DTO via the DTOflow Writer gRPC API. A service that consumes data reads the current state via the Reader API and subscribes to changes via CQS. No service calls another service's REST API to read entity state.

Consequences of this rule:

• A producing service writes its DTOs and has no further responsibility to serve that data to peers.
• A consuming service owns its CQS queue and declares which DTO types it watches.
• Inter-service dependencies are expressed through DTO types and ID patterns, not through service endpoints.
• There is a single source of truth per DTO type: the service that holds the Writer for that DTO. No two services write to the same DTO type (except during planned migration windows covered by their own ADRs).

Consequences

Easier: - Services can be redeployed independently. A consumer that is offline when a producer writes will receive the missed notifications when it comes back online. - New consumers can onboard to an existing data feed without any change to the producer. - Scaling a high-throughput producer does not affect consumer latency directly. - Startup bootstrapping is clean: a service reads the current snapshot via the Reader API and then subscribes to incremental changes via CQS.

Different (not harder): - Introducing a new data dependency requires defining or reusing a DTO type. In practice this is simpler than adding an HTTP client: DTO schemas, gRPC services, and client libraries are all generated from the proto template, so the boilerplate cost is low and the interface is standardised. - End-to-end latency is the sum of write latency + CQS notification latency. The SLA timestamp and CQS long-polling mean that high-priority flows (e.g., a user-triggered link change) are delivered with sub-second notification latency when needed, while lower-priority background changes are batched. - Debugging a broken pipeline means identifying which service last wrote a DTO and which subscriber failed to react. DTOflow addresses this with a planned observability pillar: every write (including no-ops) will be logged with a tracking_id, making end-to-end flows fully traceable.

Trade-offs accepted: - DTOflow is an internal platform dependency. Every service is coupled to the DTOflow infrastructure, but not to each other. - The model favors eventual consistency over synchronous request-response. This is the correct trade-off for the ESL domain, where rendering, transmission, and status updates are all asynchronous by nature.