845 reads become one

Why Care?

Searching this corpus opened every file in it. 845 round-trips to R2, 1.2 to 5.8 seconds, to show fifty rows — because the four strings a search matches on live inside those files and nothing had written them down.

That cost had already produced one bug. The operator reported "search filters once, then stops", which turned out not to be the filter at all: an unsearched page took 0.48s and a search took up to 5.8s, so clearing the box issued a fast request while a slow one was still in flight, and whichever landed last won. A stale-response guard fixed the symptom last week. This fixes the cause.

What's New?

• A manifest in the corpus. One small file holding what a listing needs, so a search costs one read instead of 845.

• Ranked search. Two words in any order, morphological variants, results in relevance order rather than date order, and focus chips that carry counts.

• corpora reindex, and a rebuild affordance in the app that appears only where it would actually work.

• A gap in an older spec, closed — and its test inverted rather than deleted.

The numbers

Measured on a synthetic 845-source corpus shaped like reach-edu's: 241 tagged, 604 untagged, nine domain declarations.

Read counts, not seconds. A wall-clock promise passes on a fast laptop and rots quietly; a read count is what "does this open the file" actually means. The ten is the manifest plus the nine index.md files that declare what the domains are.

Nothing was written to a client bucket to get these. Reindexing writes into the corpus, which makes measuring against reach-edu on R2 the operator's call rather than an agent's.

Pagefind does not replace the manifest

The load-bearing decision, and the one that shaped everything else.

Pagefind has no incremental add — one new source means rebuilding the whole index. So it cannot be kept current on capture, and a search index silently lagging the corpus is precisely the failure this work exists to prevent. The two coexist with distinct jobs:

It also indexes the manifest, not the corpus — one extract, two consumers, so building the search index costs whatever the manifest already cost and nothing more.

And when the bundle's recorded fingerprint does not match the manifest actually present, the app says the ranking is stale and searches on the server instead. Serving stale ranking silently is worse than serving honest substring matching.

Four things the plan did not anticipate

Written up because each was invisible until something was built, run, or counted.

1. The bundle is 866 objects

Pagefind writes one fragment per record. On 845 sources that is 866 files and 4MB — the size is nothing, the object count is not. A delete-then-rewrite would be roughly 1,700 operations against a client's bucket every rebuild, for a corpus that may have gained a single source.

Fragment, index and filter files are named by a hash of their own contents, so a key already in the store is already right. Skipping those takes an unchanged rebuild from 866 writes to 14, and adding one source to 22. Orphans are still deleted, because a leftover chunk is one the runtime would happily fetch and read as current.

2. The tags had to be filters and text

Routing domains: through Pagefind's filters is the obvious fit — filters are faceted and return counts, which is exactly the "12 in Workforce Development · 832 in the corpus" shape the app already built by hand.

But an older promise says typing literacy must reach a source carrying strategy:adult-literacy-numeracy even when neither its title nor its excerpt says the word. Filters answer exact values, not words inside them. Tags that were only filters would have dropped that promise quietly. They go in both places: filters for narrowing and counting, prose for recall.

The builder also has to widen Pagefind's word characters to include : - _ / ., or a corpus key is torn apart before it ever reaches the filter.

3. Pagefind loads its filter index lazily

Until something asks for the filters, an identical query reports every count as absent. Same query, same bundle: {} before filters() has been called, real counts after. Measured while a test failed, not read in a doc.

4. Coverage was judged against the wrong number

Ranked search reorders rows the client already holds, which is what makes it instant rather than a round trip per keystroke — the manifest is what made fetching them all affordable. That is only correct while those rows are everything the filter selects; a hit ranked outside the window has no row to render and would vanish rather than rank low. So ranking stands down and says so.

Except the comparison was against the corpus total rather than the current listing's. Click any focus chip — 41 rows held, 847 in the corpus — and ranking stood down every time, wearing the face of a deliberate fallback.

Reading the code, it looks right. Driving it, the placeholder text changes and the answer is visibly worse.

The gap that closed, and the test that said so

Strategy-Focus.md had named this index and declined to build it:

the fix when it does is an index, not 845 reads … Not built. This paragraph exists so nobody discovers the limit from a wrong answer, and FOCUS-07 asserts both halves — the miss on a page load, the hit under search — so the day it changes, a test says so.

That day was this one. A source tagged into a strategy whose folder it does not live in used to be findable only by searching, because narrowing happened on the key. With a manifest it narrows on the row, and a plain page load finds it.

So FOCUS-07 was inverted, not retired. The ID is unchanged and both halves are still asserted — unindexed still misses, indexed finds — because an unindexed corpus has to go on working. Amending a spec because reality taught you something is good work; the difference from the other kind is which one you wrote first.

Under the hood

One row builder, not two

The failure mode a listing cache invites presents to a user as "search finds different things than the page does", and it is very hard to diagnose. Both paths now reach the same function through the same Entry type — the indexed path reads one out of the manifest, the unindexed path parses a file into one and throws it away. INDEX-07 asserts the agreement field for field; having a single function is what makes it true.

What the manifest deliberately does not store

• domain — derivable from the key. A stored derived value is one that drifts.

• binary_state — whether those bytes are on this machine. The manifest is shared. Caching that answer would have one laptop's downloads showing as present on a laptop that has never fetched them.

What it does store is the parse error of any file that would not parse. An index that quietly omits what it could not read reproduces the ImmuneCo failure — 13 sources silently absent from a count — with a speedup attached.

And no timestamp, which is why two rebuilds of an unchanged corpus are byte-identical. A built_at would make every bundle look stale the moment anybody reindexed.

Drift repairs itself, and names what it cannot see

A key in the store that the manifest has never seen is read individually, so an index five captures behind costs five reads rather than a rebuild. A key in the manifest and no longer in the store is dropped.

An edit in place is the case a key comparison cannot see — same key, changed content, from the CLI or by hand. INDEX-05 asserts that limit in executable form rather than hoping about it, so the day a stat-bearing list() closes it, a test goes red and says so.

prose_excerpt moved down a layer

The manifest has to store the same excerpt a direct read computes, or the arrival of an index silently changes what search finds. Importing it from the capture package was a cycle — src/capture/__init__ eagerly imports add, which now imports the manifest — so it moved to src/model/text.py, which is where a pure markdown-text function belonged anyway. The old call site still works.

Mutation testing

Every rule was broken in turn and its test confirmed red. One did not bite: INDEX-09 claimed nothing derived is listed as a source, and passed because the manifest is .jsonl and the extension filter excluded it regardless. The promise is about the prefix, so the test now writes a derived markdown file — the case that actually proves it, and the one a future summary or report would land in.

Added later the same day — a result that says why it matched

Two reports from the operator, and they turned out to be one bug.

First: context-vigilance-kit/splash shows title, tags, and content in its results, and this did not. It uses Pagefind's own PagefindUI, whose result card is a title, a set of meta tags, and the passage that matched with the terms marked. What corpora-builder showed instead was the first 240 characters of the body — the same opening paragraph on every source, which says nothing about why this one came back. And the domains: tags, which every row has carried since focus shipped, were never rendered at all.

Second: searches took longer than they should have.

Both fixed by one change

Pagefind returns matches lazily. The ranked list is cheap; each result's data() is a separate round trip for that record's fragment — and that fragment is where the marked excerpt lives. The code resolved data() for every match:

apprenticeship   615 results   search 59ms   data(all, sequential)  821ms / 615 fetches
                                             data(50, parallel)       1ms

615 fetches against local files. In the browser, against the sidecar, over HTTP. So the fix for the slowness and the fix for the presentation are the same one: resolve only what you draw — the count comes from the result list, which needs no fetch at all, and the visible page resolves in parallel and hands back the marked passage while it is there.

Then the page size, which was still 200. An unsearched page costs nothing per row because the rows are already in the browser; a ranked row costs a fragment fetch. Drawing 200 of those was 359 requests per search. Ranked results now draw 30 with a show more, and the count stays honest — "615 matching · 847 in the corpus · showing 30".

Finally, fragments, index shards and filter shards are named by a hash of their own contents, so a given name can never mean different bytes. Serving them immutable takes a repeated query to zero requests:

The excerpt is rendered as HTML, so it is first reduced to text plus <mark> and nothing else. Pagefind escapes fragment content itself — but "the library escapes it" is a claim that stops being true the day the library changes.

And a type that was lying

Adding the domains: tags to the card failed to compile: SourceRow in the client had never declared the field. The server has been sending it since focus shipped, the app has been receiving it, and the interface said it did not exist. The one place a lie like that is cheap to find is the moment somebody tries to use the field.

The results go under the search box

The first version of the above put results in the list further down the page, which is where they already were. The splash puts them in a panel under the input, and that is a different question being answered: the list is "show me everything matching, so I can work through it"; the panel is "did the thing I am thinking of come back", answered without leaving the box you are typing in. Both are wanted. Only one existed.

Eight results, each a title, the matching passage, and its domains: tags. Escape closes it, a click outside closes it, picking one opens the source. It renders from rows, not from Pagefind, on purpose — a corpus with no search index is the common case today, and a panel that only appeared after somebody ran reindex would look broken rather than unindexed. With an index the same rows arrive ranked and their excerpts arrive marked, so the panel improves without changing.

What was indexed was not what was worth reading

The panel made a mistake visible that the list had hidden. Every excerpt ended:

… strategy:adult-literacy-numeracy live strategies adult literacy numeracy
sources 2026 05 06 source 089.md live/strategies/adult-literacy-numeracy/…

The corpus key was being fed to Pagefind so that typing part of a path would rank a source. It worked. But Pagefind excerpts from the same text it matches on, so the path came back rendered as prose, in every single result.

Content indexed for matching has to be content worth reading. The key is out — it lives in meta.path, which is what the app joins on, and the loss is small because a filename here is a slug of its own title and the title is indexed. The domains: tags stay, but as words rather than raw tags: strategy:adult-literacy-numeracy goes in as "adult literacy numeracy", so it still answers a search for literacy and reads as English if it surfaces.

What's Next

• The operator walk-through. Both specs stay Draft until then. A green ledger proves the code does what the tests say; only the operator judges whether it does what they meant.

• Measuring against reach-edu itself, which means writing an index into a client bucket — a decision, not a step.

• list_stat() on the storage seam. ListObjectsV2 already returns ETag and size in the response list() discards. That would close the edit-in-place gap for zero extra requests — and it changes the interface everything sits on, so it wants its own decision.

• Multiple simultaneous focuses. Pagefind filters support it natively, which is the first time the "eventual shape" is cheap rather than invented. Still waiting on multi-valued data actually existing.

Related

• context-v/specs/Search-Index.md · Ranked-Search.md

• context-v/plans/Search-Index-Then-Pagefind.md — the two-phase plan and its risks

• changelog/2026-08-23_01.md — the navigation work whose BROWSE-17 this makes nearly unnecessary