refactor(validation): key citation validation by URL, not by per-file citation number
feat(curation): per-deal Sources.md codified-source loader (consumer side)
feat(curation): source_aggregator agent — produces Sources.md draft and HALTS for analyst curation

Three commits over the day, one architectural arc. The validator
finally sees every source in the corpus (was: 95% lost to a per-file
numbering collision); the writer can be configured to compose from
an analyst-approved source list only (was: composes from whatever
broad-search surfaced, including LLM URL inventions); and the new
aggregate_sources agent halts the pipeline mid-run with a curated
draft, so the analyst's curation step actually precedes the writing
step instead of being a cosmetic post-pass.

* src/agents/remove_invalid_sources.py — URL-keyed refactor across
  validate_url callers, _run_recovery_pass, save_source_validation_log,
  write_redacted_hallucinations_log, the agent function, and the CLI
  standalone. New helpers: _per_file_nums_for_invalid_urls,
  remove_invalid_url_citations resolve URL-set → per-file num-set for
  targeted markdown removal.

* src/curation/sources_md.py — frontmatter parser → SourcesMd /
  SourceEntry dataclasses; forgiving per-section tag matching.

* src/curation/fetch.py — URL → clean markdown via Jina Reader (free
  tier; no API key required) with httpx + BeautifulSoup fallback.

* src/agents/codified_section_researcher.py — replaces broad-search
  per-section research when Sources.md is in codified mode. Claude
  synthesizes per-section research notes from curated content with
  proper [^N] citations; falls back to raw-content dump if no
  ANTHROPIC_API_KEY.

* src/agents/perplexity_section_researcher.py — top-of-function
  short-circuit: if codified Sources.md detected, delegate to the
  codified researcher and return.

* src/agents/source_aggregator.py — new agent between cleanup_research
  and draft. Walks 1-research/*-research.md, dedupes URLs across
  sections (tagging each URL with the sections it appeared in),
  validates via Phase 1, drops hard-fails, sorts survivors by tier +
  occurrence count, writes outputs/<v>/Sources-aggregated.md, and
  sys.exit(0)s. Passes through silently when codified mode is already
  active so the second-phase run can continue to the writer.

* src/workflow.py — register aggregate_sources node, reroute
  cleanup_research → aggregate_sources → draft.

* templates/Sources-template.md — annotated template for analysts.

Changelog - 2026-05-22_02

The Analyst Gets the Pencil

Yesterday's URL Validation Grows a Body un-disabled the citation validator and gave it a body-sniff layer. We thought we had Phase 1 done. We ran the next ChromaDB memo as v0.0.8 to demonstrate the new behavior, and the validator's report was:

Results (pre-recovery): 6 valid, 0 uncertain, 0 invalid

Six citations. Across a memo that ought to have ~100 sources. Something was very wrong, and the wrong was older than yesterday's work.

This entry documents the three pieces that landed once we started pulling that thread: a structural refactor of the validator's data model (the proximate cause of the "6 instead of 100" problem), the full plumbing of an analyst-curated-source workflow on top of the now-working validator, and the conceptual shift that came with it — the analyst's curation step now precedes the writing step, instead of being a cosmetic post-pass over already-written prose.

Why

The reason citation validators in our pipeline kept under-performing on memo quality wasn't really about validators. It was about an underlying truth we kept forgetting: the writer was composing from whatever broad-search surfaced, including the LLM URL inventions that Tavily and Perplexity Sonar happily included alongside legitimate sources. The validator's job was to clean up after — to throw out the dead URLs after they'd already shaped the prose. By the time validation ran, the memo was already framed by sources the analyst had never approved.

The real fix isn't a better validator (though we'll take one). The real fix is to put the analyst in the middle of the loop: between the URL-aggregation step and the prose-writing step. Aggregate broad-search results, surface them to the human, let the human rank and prune, then compose only from approved sources. That's the Human-Curated Source Sets and Per-Firm RAG exploration in code.

Three things had to happen for that workflow to actually function:

1. The validator had to see every URL the corpus contained — not 6% of them.

2. The pipeline had to have a way to consume an analyst-curated source list and skip broad search when one was present.

3. The pipeline had to produce the source list itself from a broad-search pass so the analyst wasn't starting from a blank file.

We did all three today.

What Landed

1. URL-keyed validation (the structural refactor)

collect_all_citation_urls() previously returned {citation_num: url}. The function walked every section file and .update()-merged a per-file {num: url} dict — but per-file citation numbers are not globally unique. Every section file uses its own local [^1] through [^N], so each later file's [^1] overwrote every earlier file's [^1]. ChromaDB-v0.0.8 had ~100 unique URLs in 1-research/; the validator saw 6.

Fixed by keying everything by URL. New helpers _per_file_nums_for_invalid_urls() and remove_invalid_url_citations() resolve a set of invalid URLs back to per-file (file, citation_num) pairs so the existing per-file num remover can still target each occurrence precisely. The refactor rippled through _run_recovery_pass, save_source_validation_log, write_redacted_hallucinations_log, the agent function, and the CLI standalone — about 270 lines added, 230 removed, net +40 plus a fundamental change in what the validator actually validates.

Demonstration: running the now-URL-keyed cleanup against the existing ChromaDB-v0.0.8 corpus:

Found 39 unique citations to validate
Validating URLs (parallel, 10 workers)...
Results (pre-recovery): 25 valid, 3 uncertain, 11 invalid (across 39 unique URLs)

🗑️  Removing 11 invalid URL(s):
  Soft 404: body contains 'page not found' — https://docs.trychroma.com/docs/collections/update-data
  Soft 404: body contains 'page not found' — https://docs.trychroma.com/docs/overview/introduction
  Soft 404: body contains 'the page you're looking for' — https://metadesignsolutions.com/...
  HTTP 404 — https://techcrunch.com/2023/04/25/chroma-open-source-embedding-database-vector/
  HTTP 404 — https://techcrunch.com/2023/04/25/weaviate-nabs-50m-...
  HTTP 404 — https://techcrunch.com/2023/04/27/pinecone-a-vector-database-startup-...
  Hallucination pattern: ^https?://(www\.)?idc\.com/getdoc\.jsp\?containerId=US\d+$ — https://www.idc.com/getdoc.jsp?containerId=US50642323
  Hallucination pattern: ^https?://(www\.)?idc\.com/getdoc\.jsp\?containerId=US\d+$ — https://www.idc.com/getdoc.jsp?containerId=US51890524
  HTTP 404 — https://www.idc.com/getdoc.jsp?containerId=prUS51701224
  HTTP 404 — https://www.reuters.com/technology/pinecone-raises-100-million-...
  HTTP 404 — https://www.wsj.com/finance/investing/software-infrastructure-stocks-...

39 URLs validated, 11 dropped, 28% bad-source rate — including two perfect catches by yesterday's Step 3 hallucination-pattern regex on bare-containerId IDC URLs. That's what the pipeline was producing all along; we just couldn't see it because the validator was being fed 6 URLs to inspect.

2. Codified-mode consumer — Sources.md + codified_section_researcher

The pipeline now understands a deals/<deal>/inputs/Sources.md file with YAML frontmatter that lists curated URLs. When mode: codified, the per-section researcher short-circuits Perplexity Sonar entirely:

---
mode: codified
deal: ChromaDB
firm: alpha-partners
date_curated_initial: 2026-05-22
curated_by:
  - Michael Staton
sources:
  - url: https://www.trychroma.com/blog/series-a
    sections: [funding-terms, team]
    rank: 1
    sensitivity: citable_externally
    note: "Series A announcement; primary."
  - url: https://github.com/chroma-core/chroma
    sections: [technology-product, traction-milestones]
    rank: 1
    sensitivity: citable_externally
---

# ChromaDB — Curated Sources

## How this list was built ...
## Excluded — examined and rejected ...
## Open questions / coverage gaps ...

The new codified_section_researcher_agent fetches each curated URL once via Jina Reader (free, generous tier; no API key required), groups sources per section by their sections: tags, and writes per-section research files in 1-research/<NN-slug>-research.md matching Perplexity's output shape so the writer downstream consumes them without modification. When ANTHROPIC_API_KEY is present, Claude synthesizes proper research notes with [^N] citations referencing only the curated set; without it, the raw fetched markdown is dumped with a citation footer.

The short-circuit is at the top of perplexity_section_researcher_agent — three lines of code that change the entire memo composition story. If codified Sources.md exists, the broad-search providers never fire.

3. Source aggregator — the producer half

The piece that closes the loop. New source_aggregator_agent sits in the workflow between cleanup_research (Phase 1 validation gate) and draft (writer). Its job in three lines:

1. Walk 1-research/*-research.md from this run's broad-search output.

2. Validate every unique URL through Phase 1, drop hard-fails, sort the survivors by a tier+occurrence heuristic, write outputs/<v>/Sources-aggregated.md with mode: aggregated and the analyst-notes scaffold.

3. sys.exit(0).

That last bullet is the load-bearing one. The pipeline halts with a printed instruction block:

🛑 HALTING PIPELINE for analyst curation.

Next steps:
  1. Open: outputs/<v>/Sources-aggregated.md
  2. Reorder rows (top = primary), x-out sources that aren't quality
     or don't have the info you want, fill in any analyst notes.
  3. Copy or move the curated file to: inputs/Sources.md
     (or edit it in place and rename) — and set `mode: codified`.
  4. Re-run the pipeline. The codified-mode researcher will use
     ONLY your approved sources; no broad search, no model invention.

When the analyst re-runs the pipeline, the codified short-circuit at section_research kicks in, the aggregator detects codified mode and passes through silently (instead of halting again), and the writer composes from the approved corpus.

This is the workflow shape from the exploration realized in code: harvest → validate → HUMAN GATE → extract → write.

How It Composes

The pipeline now has two distinct phases when run on a fresh deal:

PHASE 1 (broad search → curation gate):
  dataroom → deck → research → section_research (broad via Perplexity)
    → cleanup_research → aggregate_sources → HALT
  ↓
  Analyst opens outputs/<v>/Sources-aggregated.md
  Reorders, prunes, sets mode: codified
  Copies to inputs/Sources.md
  ↓
PHASE 2 (codified → write):
  dataroom → deck → research → section_research (codified short-circuit)
    → cleanup_research → aggregate_sources (passes through; no halt)
    → draft → enrichment → citations → fact-check → validate → finalize

Both phases share the same workflow graph. The branch happens implicitly via file presence: no Sources.md = broad search + halt; codified Sources.md = curated + run-to-completion.

What's Still Soft

Three things that won't be proven by code review and will surface on first use:

1. sys.exit(0) halt cleanliness. LangGraph doesn't have a "pause" primitive, so we exit the process. If LangGraph's checkpointer wraps the exit in a way that produces an ugly stack trace, the fallback is a paused_for_curation: true state flag + a conditional edge in the graph — more invasive but more graceful.

2. Aggregated-file YAML well-formedness. The aggregator writes Sources.md by string concatenation (not yaml.safe_dump) because we want the analyst-friendly comment lines and the data interleaved. Titles with embedded quotes get a naive replace('"', "'") scrub. If the loader chokes when re-reading the file the aggregator just wrote, we'll know — and we'll switch to a hybrid serializer.

3. Recovery cost at scale. Each Phase 1 invalid URL triggers a Tavily search ($0.008) plus per-candidate validations ($0.001 each). The smoketest against v0.0.8 showed 11 recoveries attempted on 11 invalids — about $0.20 total. On larger memos with 200+ URLs and 30% invalidity, that's $1–2/run. Tolerable but worth caching, which is the next Phase 1 step (Step 6, .url-validation-cache.json).

Demonstrated on v0.0.10

The arc ran end-to-end on ChromaDB tonight, in two stages:

v0.0.9 — phase 1 (broad search → halt). Pipeline did the usual broad-search research, walked into aggregate_sources, validated all 51 unique URLs from the corpus (vs. the 22 the old non-URL-keyed cleanup_research_citations saw), dropped 23 hard-fails, kept 28. Wrote outputs/ChromaDB-v0.0.9/Sources-aggregated.md with the survivors sorted by tier+occurrence. sys.exit(0) halt was clean — exit code 0, no stack trace, just the printed curation-instructions block.

Analyst step. Five URLs added by hand to the aggregated file: two MindStudio AI-memory pieces (build-ai-second-brain-persistent-memory, hybrid-ai-memory-system-memarch-hermes-claude-code), three of the client's investors (the Lead and two others — the last requiring a quick lookup, since quietcapital.com is parked and the real site is quiet.com). The five additions got tagged into the existing decile-group outline's section names rather than across an outline change — outline switching mid-curation breaks tags, learned that one in real time. Aggregated file promoted to inputs/Sources.md, mode flipped from aggregated to codified (with the line-anchored regex required because there are two mode: occurrences in the file — one in a comment block).

v0.0.10 — phase 2 (codified → write). Codified short-circuit fired at section_research: 📚 Codified-source mode active — 33 hand-curated URLs from inputs/Sources.md. Codified researcher fetched 32 of 33 URLs cleanly via Jina (one analyst-added VC homepage came back too thin to use). Six of seven outline sections populated from the curated tags; one section's tags didn't match anything in the corpus and got the <needs-source> stub. Source aggregator detected codified mode at its turn in the graph and passed through silently. Writer + enrichment + citations + fact-checker + validator + scorecard + one-pager all ran to completion.

Quality score: 5.5/10 (v0.0.8 broad search) → 6.5/10 (v0.0.10 codified) on the same deal. One point of improvement from removing broad-search slop in favor of the analyst's curated 33. Final draft (49K), one-pager HTML + PDF, redacted-hallucinations.md (capturing 4 URLs that the post-fact-corrector validation pass dropped), all produced.

Two of the three "What's Still Soft" worries from above came up clean:

• sys.exit(0) halt cleanliness. Worked. LangGraph propagated SystemExit without complaint; main.py exited gracefully; the printed instructions block was the last thing on stdout.

• Aggregated-file YAML well-formedness. Round-trip-parse-tested at every edit during curation. The loader read back what the aggregator (and subsequently the analyst's manual additions) wrote, with all 33 sources resolving correctly.

• Recovery cost at scale. Less relevant for the codified run since the writer's source set is pre-validated — recovery only fires for invalid URLs, and codified URLs are the ones the analyst already approved. Still worth caching for the broad-search runs.

The non-obvious gotcha that did surface — and that the new agent skill captures — is that the orchestrator has two separate cleanup gates with different code paths: cleanup_research (defined inside workflow.py, still using the old per-file-num dedup, saw only 22 of 51 URLs) and cleanup_sections (the URL-keyed remove_invalid_sources_agent from yesterday's commit). Consolidating them is queued; the working hypothesis is that the workflow's cleanup_research_citations function should be replaced with a call into remove_invalid_sources_agent with a different gate_name parameter.

What's Next

The Trustworthy-Citations rollout had seven Phase 1 steps. After today, 1–5 are landed (+ redaction worksheet, + aggregator). Remaining:

And one piece that emerged today and isn't yet in the rollout plan:

The real next thing is proving the full loop with a real memo. The v0.0.9 run is fired as this entry goes to disk; we'll see whether the halt is clean, whether the aggregated file parses cleanly when the analyst re-saves it, and whether the codified researcher produces research files the writer can actually compose from.

References

• [[The-Analyst-Gets-the-Pencil]] — this entry

• [[2026-05-22_01|URL Validation Grows a Body]] — yesterday's prerequisite work

• [[Human-Curated-Source-Sets-and-Per-Firm-RAG-for-Memo-Narrative]] — the exploration this implements

• [[Trustworthy-Citations-Source-Harvester-Rollout]] — the multi-phase plan; today closed out Steps 3–5 and added the aggregator

• [[Separating-Retrieval-from-Generation-in-Agent-Pipelines]] — the architectural rationale

• [[Curating-only-valid-Sources-across-Runs]] — the original post-mortem that surfaced the 392-URL mess