Skip to content

Large File System (LFS)

Purpose

The Large File System (LFS) is a GCS-backed binary asset store for files that are too large or not suitable to embed in protobuf DTOs. Protobuf messages are designed for structured data; embedding kilobytes or megabytes of raw binary in a DTO would inflate DTO storage, slow down reads, and make change notifications unnecessarily heavy. LFS solves this by storing binary assets separately in Google Cloud Storage and having DTOs carry only the path string that points to the asset.

LFS is content-addressed: the filename is always a SHA-256 hash of the file contents with the original extension appended. Because the filename encodes the content, LFS files are immutable — a given path always refers to the same bytes, forever.

The most important consequence of content-addressing is that it preserves DTOflow's downstream idempotency guarantee for binary assets. DTOflow does not trigger downstream notifications when a DTO is written with the same field values as its current state. Because the LFS path encodes the content hash, an unchanged binary asset produces the same path — so the DTO field does not change, and no downstream work is triggered. This extends the same "write the same DTO, nothing happens downstream" guarantee from structured fields to large binary assets.

File types

LFS stores the binary assets used by the ESL rendering and display pipeline:

  • Fonts (.ttf, .otf) — Typography resources uploaded by tenant administrators in the Pricer Studio Designer. Fonts are tenant-scoped: a font uploaded for farmersmarket is not accessible to other tenants.
  • Design preview images (.png, .svg) — Raster and vector previews of ESL label designs generated by the Studio Designer. These are stored per-tenant and referenced from design DTOs.
  • Rendered ESL images (.png) — Final rendered images produced by studio-renderer (and merged with ECC output by eslimage-merger), stored per-store. These pixel-perfect images are transmitted to the physical ESL display hardware by pricer-server. They are referenced from eslimage (and legacy renderedimage) DTOs.

Path convention

LFS paths follow a hierarchical pattern that encodes the owning system, tenant, and asset category, followed by the content-addressed filename. Note that LFS paths never include a store segment — they are always tenant-scoped, not store-scoped:

/{system}/t/{tenantId}/{category}/{sha256}.{ext}

Examples:

Asset type Example path
Tenant font /designer/t/farmersmarket/fonts/3a7bc9f1d2e4...abc.ttf
Design preview /designer/t/farmersmarket/previews/8f2d0e5a1c3b...def.png
Rendered ESL image /renderer/t/farmersmarket/eslimages/c4b1a2e9f0d7...456.png

The {system} prefix (e.g., designer, renderer, ecc) identifies which subsystem owns the file. This prevents path collisions when multiple systems store assets for the same tenant, and it allows GCS IAM policies to be applied per system.

The {sha256} component is the lowercase hexadecimal SHA-256 digest of the raw file bytes. Together with the extension, it forms the complete filename.

Deduplication scope: two uploads of identical bytes within the same {system} prefix will always land on the same GCS object — zero extra storage cost. However, different {system} prefixes store independent objects; /renderer/... and /ecc/... will each store their own copy even if the bytes are identical.

Tenant isolation

LFS is fully tenant-isolated for all tenant-specific data. Each LFS path embeds the tenant ID (/t/{tenantId}/), and GCS IAM policies are scoped to the {system}/t/{tenantId} prefix. One tenant's files are never accessible to another tenant, regardless of which service is performing the access.

Relation to DTOs

DTOs carry LFS paths as plain string fields, never file bytes. This keeps DTO payloads small and Spanner storage efficient. When a service needs the actual file, it uses the path to fetch it from GCS directly.

Example: the design DTO has a preview_path field:

message Design {
  // minor version 2
  string id = 1;
  string name = 2;
  string template_id = 3;
  string preview_path = 4;   // LFS path, e.g. "/designer/t/farmersmarket/previews/<sha256>.png"
  // ... other fields
}

Similarly, the renderedimage DTO references the final ESL image:

message RenderedImage {
  string id = 1;           // e.g. "t/farmersmarket/s/oslo-main/esls/7318690123456/renderedimage"
  string image_path = 2;   // LFS path to the rendered PNG
  // ... other fields
}

When studio-renderer produces a new rendered image, it: 1. Computes the SHA-256 hash and constructs the LFS path. 2. Writes the PNG bytes to GCS at that path. 3. Writes a renderedimage DTO to the DTOflow Server with image_path set to that LFS path. 4. CQS notifies pricer-server, which reads the eslimage DTO, retrieves the path, fetches the image bytes from GCS, and transmits them to the ESL hardware.

The DTO and the file are updated independently: a new rendered image always produces a new LFS path (different hash), and the DTO is updated to point to the new path.

Reading and writing LFS files

Services interact with LFS by reading and writing GCS directly. There is no intermediary service for backend-to-LFS communication — adding one would introduce an extra network hop with no benefit.

Writing a file

  1. Compute the SHA-256 hash of the file bytes.
  2. Construct the LFS path using the pattern documented on the DTO field: /{system}/t/{tenantId}/{category}/{sha256}.{ext}.
  3. Write to GCS using the create-only API with the doesNotExist() precondition:
  4. Java: Storage.BlobTargetOption.doesNotExist()
  5. Node: equivalent conditional create option
  6. If GCS returns 412: the object already exists with the same content (same hash → same bytes). Treat 412 as success — nothing more to do.
  7. If GCS returns a non-412 error: handle as a real failure.

Do not issue a GET before writing. A pre-flight read creates a race condition and double-bills the operation (both a Class B read and a Class A write). The doesNotExist() precondition is the correct and only mechanism. See PLT-2658 for background.

Reading a file

Fetch the GCS object directly using the path string stored in the DTO field. Backend services authenticate via IAM. The path is stable and immutable — it can be cached indefinitely.

External consumers (browser UIs, external integrations) do not have direct GCS access. They read files through the file-service REST API, which is exposed via Apigee.

Service responsibility

Backend services (e.g., studio-renderer, ecc-image-render-service) write files directly to GCS. There is no DTOflow intermediary service for writes.

Path patterns are defined in each DTO's documentation. Services are responsible for constructing the correct path. dtoflow-spanner may validate LFS path format in the future, but does not currently.

dtoflow-lfs (the Cloud Run service, formerly dtoflow-file-service) has one runtime role: it provides the REST API that external consumers use to read files. It is not in the write path for backend services.

GCS write semantics

LFS is insert-only. GCS is configured to reject overwrites: once a path exists, its bytes are immutable. This is enforced at the infrastructure level, not just by convention.

A write attempt to an existing path returns 412. Because paths are content-addressed, a 412 always means "this exact content is already stored at this path" — which is the desired outcome. Writers must handle 412 as success.

Garbage collection

DTOflow tracks which LFS paths are referenced by active DTOs. When a path is no longer referenced by any DTO, the platform will eventually delete the corresponding GCS object. This is a platform responsibility — writing services must not delete LFS files themselves.

Garbage collection is not yet implemented. Orphaned files currently accumulate in GCS until it is.