One way to name, fetch, and file a source

Why Care?

corpora-builder exists because corpus assembly kept reappearing as a sub-feature inside other products. The risk that comes with that history is obvious in hindsight: three codebases quietly grow three answers to the same question, and by the time anyone compares them, each has real data on disk in its own shape.

That was about to happen. memopop shipped a source-approval surface in early August that fetches page content, shows it, and throws it away. Fixing that meant writing a source file — and the moment it wrote one, memopop would become the third generation of a schema that already existed twice, in augment-it's content-ingest service and in its strategy-curator view model.

So we stopped and compared them first. The good news is that nobody has to be talked out of anything.

What's New?

context-v/blueprints/Source-File-Schema-Reconciliation.md — the repo's first blueprint, and the first concrete artifact of the system design's Tension 1, Option B ("the canonical corpus.ts-descended domain model … shipped as copy-from sample code the siblings adopt knots-style").

It does three things:

Maps what already agrees, field by field, across the three implementations.
Rules on the three real divergences — lifecycle vocabulary, flat fields vs extra_metadata, and provenance naming.
Fixes a canonical field set with exactly one required field (url) and graceful degradation for everything else.

What already agreed

This is the part worth internalizing, because it means the convergence is cheap:

Concept	augment-it `corpus.ts`	memopop `source-with-extracts-md`	`strategy-curator`
Filename	`<YYYY-MM-DD>_<title-slug>.md`	`<date>_<query-slug>/` + `<title-slug>.md`	`source_slug`
When we pulled it	`fetched_at`	`fetched_at`	—
When it was published	`published_at`	`published_at`	`published_date`
Dedup identity	—	`normalized_url`	`normalized_url`
Lifecycle	—	`candidate → promoted` + `content_pulled`	`metadata-only \| fetched`
Binary sibling	`binary_asset{}` + `download_status`	`asset_path`	`binary_filename`

Same skeleton. The divergences are vocabulary, not model.

The three rulings

Keep both lifecycle axes. metadata-only | fetched and candidate | promoted are not competing vocabularies for one thing — they encode how much have we pulled and where is this in the analyst's decision. Collapsing them loses a representable state: promoted-but-not-yet-fetched. Both stay.

Flat for the known set, extra_metadata for the rest. Every canonical field is flat and hand-editable; unrecognized Jina/OG keys land in the catch-all. published_at is lifted out of it because sort/filter UIs read it as first-class — augment-it's liftPublishedAt already does exactly this.

verdict is a person; machine_verdict is a machine. Permanently two fields. memopop learned this the expensive way: a real deal carried 34 verdicts reading HTTP 200 (body verified), which are reachability results. Counting those as approvals is precisely the category error its membership gate exists to correct — reachability is not approval.

Adoption

Copy-from, knots-style. Per this repo's own README: "corpora-builder does not become a shared dependency of the other ai-labs apps — patterns travel knots-style (blueprints + copy-from sample code), not as an npm package the siblings import." A shared library spanning a Python orchestrator, a Node service, and a Tauri app would be three bindings and a release cadence nobody wants.

Divergence stays allowed — it just gets documented in the blueprint rather than invented silently. That is the difference between a convention and a truce.

It paid off the same day

memopop implemented the Python side immediately, and the blueprint caught a bug in the process. Jina Reader returns metadata as a pseudo-header inside the markdown:

Title: Vector database
URL Source: https://en.wikipedia.org/wiki/Vector_database
Published Time: 2023-06-12T20:54:51Z
Markdown Content:
<the actual article>

Left unparsed, that block poisons the excerpt and strands Published Time — the one authoritative signal for published_at, without which staleness is unanswerable. augment-it already lifts it (liftPublishedAt); memopop was not. Writing the Python counterpart was the blueprint doing its job on day one rather than a fourth convention being born.

What's Next

Four questions the blueprint deliberately leaves open, the sharpest being who owns URL normalization. memopop has canonical_url() in best_sources.py; if the JS side normalizes differently, cross-app dedup fails silently and nothing will announce it. Worth settling before augment-it writes files under this schema.