Fetched content stops evaporating — and a writer that deleted fields on every save

Why Care?

Two days ago the approval surface learned to fetch a page through Jina Reader so an analyst could check a source was real. It rendered the content into a pane and dropped it on the next click. Every verification cost a network round-trip and produced nothing durable.

That is a waste, but it is also a violation of a convention this repo already wrote down. agent-skills/source-with-extracts-md has said for months: "On save, pull cheap metadata + a ~200-char excerpt. Only on promote pull the full body." Two tiers, both persisted. We were doing neither — full body for candidates, kept for nothing.

Closing that gap turned up something worse, which is the more interesting half of this entry.

What's New?

• src/curation/source_file.py — the per-source file writer. Canonical filename grammar, full frontmatter round-trip, the two-tier lifecycle, and a Jina preamble parser.

• Previews now persist. /actions/fetch-source files what it fetches under inputs/sources/ when the deal is known.

• Approve mirrors the decision into every source's file, without re-fetching.

• metadata_only — the cheap tier, so a pasted link can name itself without paying to store content that may be rejected.

• serialize.clean_source stopped deleting fields it did not recognize.

How a source gets filed

inputs/
├── Sources.md              the LIST — what the membership gate enforces
└── sources/
    └── 2026-08-08_gmv-retention-the-marketplace-metric-most-ignore.md
                            ONE source — its content and extracts

Two files, two jobs. The list is the contract; the files are where content accumulates across runs.

---
url: https://en.wikipedia.org/wiki/Vector_database
normalized_url: https://en.wikipedia.org/wiki/Vector_database
title: Vector database
fetched_at: '2026-08-08T20:04:21+00:00'   # when WE pulled it
published_at: '2023-06-12'                # when it was written
status: promoted                          # the analyst's decision
content_pulled: true                      # what is on disk
excerpt: From Wikipedia, the free encyclopedia. A **vector database**…
---

status and content_pulled are deliberately independent. One is where the analyst got to; the other is how much we hold. A source can be promoted with nothing fetched yet, or cached before anyone has judged it — and a single field cannot say both.

Jina buries its metadata inside the content

Jina Reader does not return structured metadata. It prefixes a pseudo-header into the markdown itself:

Title: Vector database

URL Source: https://en.wikipedia.org/wiki/Vector_database

Published Time: 2023-06-12T20:54:51Z

Markdown Content:
From Wikipedia, the free encyclopedia…

Left in place, two things break. The excerpt reads "Title: … URL Source: … Markdown Content: …" — metadata masquerading as content. And Published Time sits unparsed in the body while published_at stays empty, which makes staleness unanswerable.

parse_jina_preamble() splits it, with a guard: the header only counts if Markdown Content: appears in the first 600 characters. The phrase can legitimately occur inside an article about markdown, and truncating a real document there would be silent data loss.

This is a convergence win, not a memopop fix — augment-it's content-ingest/src/corpus.ts already lifts the same field from the same preamble in liftPublishedAt. We wrote the Python side of a rule that existed in TypeScript, which is exactly what [[Source-File-Schema-Reconciliation]] is for.

The bug that mattered

While verifying, io/humain showed as dirty. Checking rather than assuming turned up real data loss in a live client file.

sensitivity had gone from 93 sources to 14. Eight of those were internal_only — the field that governs whether a source may be cited outside the firm. All eight had been silently downgraded to citable_externally. In a memo shared with an LP that is a leak, not a formatting glitch.

Two layers of the same mistake:

# serialize.clean_source — before
for key in FIELD_ORDER:     # ← anything else simply vanished
    ...

The UI's save payload omitted sensitivity, and the writer only emitted keys on its known list. Meanwhile the read path deliberately preserves hand-added keys. Read preserves, write destroys is about the worst asymmetry a hand-editable file format can have — every save became a quiet deletion.

Fixed on both sides. clean_source now appends unrecognized keys after the canonical block, so a field added by an analyst, or by a future agent, survives a round-trip:

for key, val in s.items():
    if key not in out and key not in FIELD_ORDER and val not in (None, "", [], {}):
        out[key] = val

The client data was restored by merging the pre-session values back in by URL — 85 citable_externally + 8 internal_only = 93, matching the committed original, with the analyst's 46 verdicts intact.

Verification

167 tests, 1 skipped. Two of them exist specifically so this cannot recur: one asserts a hand-added key survives a save, the other hammers five consecutive autosaves and checks internal_only is still standing at the end.

Live-fetched the real a16z and Wikipedia URLs end to end — correct title, correct published_at, correct filename slug, preamble stripped from the body.

What's Next

description is declared in the schema and nothing populates it; Jina's preamble does not carry one, so it needs OG metadata from the httpx fallback path. Harmless as an empty field, but it is declared-and-unfilled.

And the blueprint's open question is now live: canonical_url() here is doing URL normalization for dedup. If augment-it's JS side normalizes differently, cross-app dedup fails silently and nothing announces it.