ADR-007: Content-Addressed LFS (Hash-Named Files)

Status: Accepted Date: 2025-06

Context

DTOflow's Large File Storage (LFS) service stores binary assets — rendered ESL images, design images, fonts, and other large blobs — that are referenced by DTO fields containing a file_path. The LFS service receives file bytes and returns a file_path that is then stored in the DTO record.

Two naming strategies were considered for files stored in LFS:

Option A — Semantic or UUID-based names: Each uploaded file is assigned a name derived from the entity it belongs to (e.g., the ESL barcode, the design ID) or a random UUID at upload time.

Option B — Content-addressed names (SHA-256 hash): Each uploaded file is named by the SHA-256 hash of its content. The UploadLfsFileResponse.file_path is {pattern}/{sha256}.{ext}.

The LFS service API exposes this in the response type:

message UploadLfsFileResponse {
    // Example: <pattern>/<sha256>.<ext>
    string file_path = 1;
}

Why UUID and semantic names were rejected

A UUID-named file for the same image bytes would produce a different path on every upload, even if the content is identical. Two renderers producing the same output for the same ESL would write two distinct files that are byte-for-byte equal. Over time, this creates unbounded storage growth as images are re-rendered and re-uploaded.

Semantic names (e.g., t/10001/s/S001/esls/ABC123/page1.png) bind the storage path to the entity, not the content. This means that when an ESL's image changes, the old file must be explicitly deleted or the new file written to a different path. Managing this lifecycle correctly requires the writer to track the previous path and issue a delete before or after each write — adding state management responsibility to every writer.

Decision

LFS files are named by the SHA-256 hash of their content. The file path returned by UploadLfsFileResponse is {directory_pattern}/{sha256}.{ext}, where directory_pattern is determined by the DTO type and the DTO ID (for tenant-scoped storage), and ext is the file extension provided by the uploader.

Two files with the same bytes within the same {system} prefix will always produce the same path. Uploading the same content twice is safe — GCS is configured create-only, so the second attempt returns 412 (object already exists), which is treated as success. Both calls result in the same file_path being stored in the DTO.

Consequences

Easier: - Downstream notifications are suppressed for unchanged files. DTOflow does not fire downstream notifications when a DTO is written with the same field values as its current state. Because the LFS path encodes the content hash, an unchanged binary asset produces the same path, leaves the DTO field unchanged, and suppresses all downstream work — exactly as with structured fields. This is the primary motivation for content-addressed naming. - Deduplication is automatic within a system prefix. If two ESLs are assigned the same design and produce identical render output, both DTO records reference the same file_path at zero additional storage cost. Note: deduplication applies within the same {system} prefix only — different system prefixes (/renderer/... vs /ecc/...) store independent GCS objects. - Re-uploads are safe and idempotent. A renderer that crashes mid-write and retries will produce the same file_path. GCS is configured create-only; the retry will receive a 412 (object already exists), which is treated as success. There is no risk of the DTO referencing a path that does not exist. - Cache efficiency. HTTP caches, CDNs, and client-side caches can use the hash-derived path as a stable cache key. The URL only changes when the content changes. - Immutable references. A file_path stored in a DTO can be treated as a permanent, stable reference to a specific version of the content. Auditing which version of an image was displayed at a given time is straightforward.

Harder: - No human-readable names. File paths in LFS are opaque hash strings. Debugging a specific image requires correlating the DTO record to the path, not reading the path directly. - Storage reclamation requires reference counting. Because multiple DTO records may reference the same file_path, a file cannot be safely deleted when any single DTO referencing it is deleted. Garbage collection of unused LFS files requires a separate reference-counting or sweeping mechanism (not yet implemented — files are currently not deleted). - Hash collisions are theoretically possible. SHA-256 collision resistance is well-established and not a practical concern at the scale of ESL images, but it is worth noting that the system treats hash equality as content equality.

Trade-offs accepted: - Storage reclamation is deferred. At the current scale of the platform, the deduplication benefit outweighs the cost of retaining unreferenced files. A garbage collection mechanism can be added later without changing the naming convention.