The corpus stops carrying its own binaries

Why Care?

78 binaries were 90.5% of the corpus bytes. That single ratio caused every storage problem this project had: it is why the repo needed Git LFS, which is why jj corrupted a client repo earlier the same day; it is why a four-year-old laptop was a constraint; and it is why sharing work meant duplicating a 156 MB tree.

bin/ fixes all three at once, and the deeper win is not storage. Under content addressing a binary cannot conflict — a changed PDF is a different hash and therefore a different object. The whole multi-writer conflict problem collapses onto text, which is the only place merge works.

What shipped

src/binary/ — keys, optimization, the store, ingest and migration. All 21 BIN-* IDs green, ladder passing, mypy clean.

Totals  113 green · 0 red · 5 gated · 0 missing · 0 retired

Two scopes, one key:

remote        r2://<client-bucket>/corpora/bin/<ab>/<sha256><ext>   per client
local cache   ~/Library/Caches/corpora/bin/<ab>/<sha256><ext>       per MACHINE

bin/ is never in a repo. The cache is shared across every corpus — fetched once, used everywhere — while the remote stays per client, because two wrappers naming the same hash would reveal that two clients hold the same document.

The deliberate run, against the real corpus

Read-only on clients/reach-edu/corpus, all writes to a dev target:


binaries walked	34
source	140.0 MB
stored	79.6 MB (56.9%)
optimized	24 of 34
kept verbatim	9 `not_smaller`, 1 `text_loss`
wall clock	137s

Biggest win: 38.0 MB → 9.2 MB (24.1%) — the Bloomberg annual report.

56.9% is the honest number, not the 24% a single file suggests. Nine PDFs came out of Ghostscript larger than they went in, and were kept verbatim rather than rewritten for no gain. A blanket "compress everything" would have grown the corpus on a quarter of its files.

The guard that matters fired on a real file

One PDF lost enough of its text layer that the optimization was rejected and the original stored. That is BIN-05, and it is the reason the invariant is a threshold rather than an aspiration: on 34 real files it caught one that a size-only rule would have silently degraded. A corpus grounds factual claims; an optimization that costs extraction is not a saving.

Three decisions worth naming

Optimize, then hash. The key must address what you actually retrieve, so the artifact is finalised before it is named. Hashing first would leave a key pointing at bytes nobody has.

Both digests, always. binary_sha256 to fetch by, source_sha256 to cite by. Drop the second and "here is the source" points at something we altered and cannot prove faithful.

Eviction is cache eviction. Clearing a cache entry is lossless by construction — a stronger guarantee than git-annex's numcopies, needing none of its bookkeeping. Nothing here deletes a remote object.

One fix in Phase 1 code

LocalFsStore.stat() computed its hash by calling self.read(), which made "verification never downloads" untrue at the seam level. It now streams from disk and takes size from the filesystem. R2Store already kept the promise — it heads the object and reads the sha256 it writes into object metadata.

Still to do

Wire it into capture. ingest_binary exists; corpora add does not call it yet.
Write the frontmatter. BinaryRef.to_frontmatter() produces the block; nothing merges it into wrappers yet.
Migrate for real. The run above wrote to a dev target. Migrating a client corpus is a separate, confirmed step, and it deletes nothing (BIN-16).
Per-file report. IngestResult.source_path now exists so "which file lost text" is answerable; that run was not completed.

context-v/specs/Binary-Ingest-And-Bin-Store.md — 21 IDs, no open questions
ai-labs/context-v/reminders/Never-Run-JJ-In-A-Git-LFS-Repo.md — the incident that made binaries the hard part
ai-labs/studies/sync-and-content-version-control/context-v/profiles/Profile__Git-Annex.md — where the eviction-safety model came from