Every URL was a guess — reconciling citations against what Perplexity actually retrieved

Why Care?

The memos read well and the sourcing didn't survive a partner clicking a footnote. The reason turned out to be one line.

Every sonar-pro response carries a search_results array — the documents Perplexity actually retrieved, each with a real URL, a real title, a real publication date. All ten call sites read exactly one thing:

research_content = response.choices[0].message.content

A grep for search_results across src/ returned zero hits. The ground truth arrived on every call and was discarded, while PERPLEXITY_RESEARCH_SYSTEM_PROMPT instructed the model to type out [Source Title](https://full-url.com) — asking a language model to recall URLs from memory and presenting the result as a citation.

Measured on one memo: 129 web citations, 37% fabricated, 34% pointing at real pages with the wrong title or date, 29% correct as written.

And the rate is not a property you can tune against. Two identical runs of the same deal, same code, same temperature: Executive Summary went 0% fabricated to 90%; Origins went 43% to 0%. You cannot threshold a distribution that wide. You have to close the surface.

What changed

Reconciliation. perplexity_sources.py keeps the retrieved-source array and rebuilds citation definitions against it: a genuinely-retrieved URL keeps its URL and gets its true title and dates; an invented one is substituted with the retrieved source that best supports the claim; a claim nothing supports loses its citation and its inline markers, because a [^7] with no definition is what makes these memos miserable to hand-edit. Prose is never rewritten.

Extraction as its own step. Coverage was being treated as a citation problem — 28 approved sources, 6 cited, so add a rule that every source must be cited. Backwards. A source goes uncited because nobody extracted anything from it, and a model asked to synthesise 28 sources into 750 words reads four and answers the rest from memory. Now every approved source gets a dedicated pass that files its quotes, stats and claims as LFM directives under # Extracts in the source's own file. On ChromaDB: 303 extracts from 26 sources, and source coverage per section went from 6-of-28 to 15-of-16.

Grounding — the check nothing else performs. Every quote and stat must appear verbatim in the stored document, checked against bytes already in memory. This is the only defense that catches a model faking having read a source: provenance says the URL was retrieved, liveness says it resolves, fact-verification says the claim is true — and all three pass a quotation that appears nowhere in the document it cites. It caught "25.6k stars on GitHub" attributed to an article that never says it.

Attribution audit, wired into the existing correction chain as fact_check → attribution_audit → fact_verify → fact_correct. A memo can pass every check and still be wrong about who a number belongs to. One Executive Summary was about to report a $113 million raise at $1.3 billion for a company raising $1.5M on a $30M cap — real figures, correctly cited, and about OpenRouter.

Citation hygiene, ported from cite-wide/src/utils/citationSpacing.ts: one space before an inline marker, marker after the punctuation, and one marker per source per paragraph. 46% of cited paragraphs repeated a citation; one 61-word paragraph carried [^1] three times and [^3] three times.

Two-tier fetch, enforced. is_explicitly_approved is deliberately stricter than the citation membership gate. Citing an unreviewed source is recoverable — the analyst sees it in the draft. Pulling full content for a candidate that may be rejected wastes a fetch, and writing it into the shared registry puts junk in the system of record where every other client sees it, permanently.

A durable foundation. Fetched content and extracts live in inputs/sources/*.md, not outputs/<version>/, so they are version-independent by construction; deck analysis is cached per deal by content hash. Generating another version is prose-shaping, not a re-read of the corpus.

Also: cli/triage_sources.py (separates was this retrieved from does it resolve from does it support the claim, and treats analyst warrant as outranking both), cli/export_web.py (webpage export with the reading-position ToC from the astro-knots blueprint, distinct from the print/PDF export), and surreal_registry.py (registration into augment-it's shared sources / source_usages tables — built, verified read-only, nothing written yet).

What this cost, honestly

Four bugs in this work were mine, found by measurement rather than by reading my own code:

• Reconciliation skipped entirely when retrieval returned nothing — the defense switched itself off in exactly the case it exists for. 25 example.com placeholders passed through untouched.

• The extractor verified one string and stored a different one, giving a false all-clear on 53 of 138 spans while reporting zero rejections. "0 rejected" looked like success and was the symptom.

• Pre-existing citations leaked into enrichment reconciliation and were judged against the wrong source set.

• The extracts parser's close-fence match let an unterminated block swallow the next directive and every heading between — silently, which is the exact failure the format exists to prevent.

Each was caught by checking the artifact instead of trusting the log line. That is the transferable lesson: a number that looks like success is the first thing to verify.

What's next

Section research in codified mode runs serially and re-prompts on ungrounded spans, so it took 20 minutes where the parallel Perplexity path took 3. That regression is mine and it is the first thing to fix — a pipeline that produces honest memos too slowly to use is not usable.

Then: <needs-analyst-judgment> instead of <needs-source> on group: synthesis sections, because a scorecard summary has no URL and saying it does is wrong; the SurrealDB write path; and a loud failure when a codified run finds zero sources for a section, which currently degrades into a confident, uncited memo.

Related: [[Extract-Sources-Where-Possible-with-Fallback-to-Ground-Content]] · [[Reconciling-Perplexity-Citations-Against-Retrieved-Sources]] · [[Citation-Coverage-Promoter]]