Skip to content

feat(upload): dedup encrypted binaries on the plaintext hash - #101

Open
riglar wants to merge 1 commit into
devfrom
feat/encrypted-binary-dedup
Open

feat(upload): dedup encrypted binaries on the plaintext hash#101
riglar wants to merge 1 commit into
devfrom
feat/encrypted-binary-dedup

Conversation

@riglar

@riglar riglar commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

CLI half of devicecloud-dev/dcd#1168. Platform half: devicecloud-dev/dcd#1169.

Problem

--encrypt uses a fresh random DEK per upload, so identical plaintext produces
different ciphertext every time. The dedup key was the hash of whatever gets
uploaded — so encryption silently disabled dedup entirely, and every run
re-uploaded and re-stored the full binary.

Confirmed on dev with a byte-identical test/fixtures/wikipedia.apk: two
--encrypt runs produced two distinct binary IDs (33c84e4c, cb3fdd3e), while
the plaintext run hit the cache. For a 300 MB app in CI that is a full re-upload
on every single run.

Change

Hash the plaintext first, dedup on that, and only encrypt on a miss:

shaPlain = H(plaintext)
dedup check { shaPlain, encrypted: true }
  hit  → return binaryId        // skips encryption AND upload
  miss → encrypt → sha = H(ciphertext) → upload → finalise { sha, shaPlain }

Because the check now happens before encryption, a hit skips the encryption work
too — an encrypted re-run is now faster than the old plaintext path, rather than
merely matching it.

binaries.sha is untouched and remains the ciphertext hash. That is deliberate:
the hosts that verify it (B2, Supabase Storage, storage-cache, Mac LRU) hold
ciphertext and no DEK, so they can only hash the bytes they physically have. The
plaintext hash rides alongside as shaPlain, sent only when encrypting.

The client re-checks the invariant itself

A dedup hit is honoured only if the server confirms the matched binary is itself
encrypted. This is not belt-and-braces for its own sake: a previously-uploaded
plaintext copy of the same app has the same plaintext hash, so a lookup
blind to encryption state would hand back an unencrypted binary to someone who
asked for encryption. It would have fired on my very first test, since the dev
plaintext wikipedia.apk row has exactly the bytes the --encrypt runs hashed.

The API applies the same predicate. This is the client declining to depend on
that, so an older or misbehaving deployment degrades into a redundant upload
rather than a silent loss of encryption. A response with encrypted absent
entirely — i.e. any deployment predating dcd#1168 — is likewise treated as a miss.

Compatibility

  • Requires the API side of dcd#1168 for the encrypted path to dedup. Against an
    older deployment the check finds nothing and behaviour is exactly as today, so
    this can merge and ship independently.
  • The unencrypted path is unchanged on the wire — still { sha }.
  • ApiGateway.checkForExistingUpload still accepts a bare sha string as well as
    the new lookup object, so existing callers are unaffected (covered by a test).

Cost worth naming

An encrypted upload that misses now hashes twice — once over the plaintext, once
over the ciphertext. Hashing is cheap next to encryption plus upload, and only the
miss path pays it; the hit path does one hash and stops.

Testing

  • 194 passing (up from 182), pnpm lint / typecheck / build clean.
  • New test/unit/encrypted-dedup.test.ts (12 cases) covers the wire format, the
    back-compat string signature, --ignore-sha-check, the unencrypted path being
    untouched, and that the lookup key is stable across runs — the property the
    ciphertext hash lacked.
  • The three that matter are the negative ones: a plaintext match is rejected
    when encryption was requested, a hit with encrypted omitted is rejected,
    and an encrypted match is reused without encrypting or uploading at all.

Also updates the --encrypt help text, which until now gave no hint either way
about dedup.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

`--encrypt` uses a fresh random DEK per upload, so identical plaintext produces
different ciphertext every time. Since the dedup key was the hash of whatever
gets uploaded, encryption silently disabled dedup entirely: every run re-uploaded
and re-stored the full binary. Verified on dev — two `--encrypt` runs of a
byte-identical APK produced two distinct binaries, while the plaintext run
deduped.

Hash the plaintext first, dedup on that, and only encrypt on a miss. Because the
check now happens before encryption, a hit skips the encryption work too, not
just the upload — an encrypted re-run is now faster than the old plaintext path
rather than merely matching it.

`binaries.sha` is untouched and remains the ciphertext hash: the hosts that
verify it (B2, Supabase Storage, storage-cache, Mac LRU) hold ciphertext and no
DEK, so they can only hash the bytes they actually have. The plaintext hash rides
alongside as `shaPlain` and is sent only when encrypting.

A dedup hit is honoured only if the server confirms the matched binary is itself
encrypted. This matters: a previously-uploaded *plaintext* copy of the same app
has the same plaintext hash, so a lookup blind to encryption state would hand
back an unencrypted binary to someone who asked for encryption. The API applies
the same predicate (devicecloud-dev/dcd#1168); this is the client refusing to
depend on that, so an older or misbehaving deployment degrades into a redundant
upload instead of a silent loss of encryption.

Requires the API side of dcd#1168 for the encrypted path to dedup; against an
older deployment the check finds nothing and behaviour is exactly as it is today.
The unencrypted path is unchanged on the wire.

One cost worth naming: an encrypted upload that misses now hashes twice (once
plaintext, once ciphertext). Hashing is cheap next to encrypt + upload, and only
the miss path pays it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@claude

claude Bot commented Jul 30, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant