fix(validation): un-disable URL accessibility checks and replace HEAD-only validation with GET + body sniff

Phase 1 steps 1–2 of the Trustworthy-Citations rollout
(context-v/plans/Trustworthy-Citations-Source-Harvester-Rollout.md).

citation_validator.py:302 — drop the `len(issues) == 0` guard that
self-disabled URL accessibility checks whenever a citation had any
other issue, which in practice meant every real citation since date-
format warnings are common. The check now runs against every
extracted URL. First-run effect: many more URL warnings surface —
restored visibility, not a regression.

remove_invalid_sources.py — replace the urllib HEAD-only validate_url
with an httpx GET that reads the first ~32KB of the response body and
classifies into three new content-based verdicts:
  - SOFT_404_BODY  (-2): HTTP 200 but body contains "page not found",
    "no longer available", and similar phrases. Drops the URL.
  - PAYWALL_STUB   (-3): HTTP 200 but body contains "sign in to read",
    "subscribe to continue", and similar. Drops the URL. A subsequent
    commit will wire a reputable-publisher allow-list so WSJ/FT/Bloomberg
    paywalls become `verified-gated` rather than dropped.
  - HALLUCINATION_PATTERN (-1): unchanged regex preflight; renamed
    constant for clarity.
Both call sites (workflow agent + CLI entry) now drop on the three
negative sentinels via CONTENT_INVALID_CODES.

Non-HTML responses (PDF, JSON, plain text) are passed through on
status code alone — a 200 from a real PDF is a real source.

httpx is already a project dependency; no new packages added. The
existing ThreadPoolExecutor wrapping is preserved — httpx.Client (sync)
works inside threads.

* src/agents/citation_validator.py — drop bypass guard on URL check
* src/agents/remove_invalid_sources.py — httpx GET + body sniff +
  soft-404 / paywall classification + CONTENT_INVALID_CODES sentinel set

Changelog - 2026-05-22_01

URL Validation Grows a Body

For months the citation validator has been lying to itself. Not by intent — the bug was a single-line guard on line 302 of citation_validator.py:

if extracted_url and len(issues) == 0:  # Only check URL if no other issues

The author wrote it to be polite about latency. The effect was that the URL accessibility check ran only on citations that had zero other warnings — and since the validator routinely emits a warning for any non-standard date format, "Updated: N/A" formatting, or trivial citation-string quirks, the gate closed on essentially every real citation. The check had been a placebo for as long as anyone bothered to run it.

This commit removes the guard, which is the boring half of the work. The interesting half is what was sitting underneath: even when the URL check did run, it was a HEAD request that took an HTTP 200 as gospel. On the modern web that means almost nothing — enterprise SaaS sites return 200 with "page not found" bodies, Bloomberg returns 200 with a "sign in to continue" stub, Gartner returns 200 for any doc-id you can guess, and the validator was happily stamping all of it as VALID. So this commit also rewrites validate_url() in remove_invalid_sources.py to use an httpx GET that reads the first ~32KB of the body and classifies into three new verdicts beyond the raw HTTP status code: soft-404 (HTTP 200 + "page is gone" phrasing), paywall-stub (HTTP 200 + "sign in to read"), and hallucination-pattern (regex preflight, unchanged behavior).

Two files, ~110 lines added, ~34 removed. The blast radius is narrow; the visibility lift is large.

Why

We have seven versions of the ChromaDB memo in io/alpha-partners/deals/ChromaDB/outputs/. We ran the [[Curating-only-valid-Sources-across-Runs|cross-run source curation]] pass against all of them last week and got 392 "unique" URLs across 26 best-of catalogs. When we started clicking through the master list, the very first URL we checked was a clean 404 — AWS returns <title>404 Not Found</title> in the body. The orchestrator had logged it HTTP: 200. We dug in. The HEAD request was happily seeing AWS's CDN return 200 for the unknown path before the body got involved.

This is not a new problem. [[Faked-Sources-from-Perplexity]] documented it in December 2025. [[blueprints/Anti-Hallucination-Source-Validation-and-Removal]] proposed a fix. The fix shipped — and then the bypass on line 302 disabled it for everything but the cleanest citations, and the HEAD-only validator missed everything that wasn't a textbook 404. Two layers of silent failure, six months of memos with sources that didn't survive the first click.

This isn't an architecture problem. The architecture is fine. It's a "what we said we were doing versus what we were actually doing" problem, and the two-line nature of the fix is the embarrassing-and-true shape of it. The Trustworthy-Citations rollout plan ([[Trustworthy-Citations-Source-Harvester-Rollout]]) opens with this exact diagnosis as Phase 1 — Block the bleeding before any architecture work begins. This is steps 1 and 2 of that phase landing.

What Landed

citation_validator.py:302 — drop the self-disabling guard

- # 4. Check URL accessibility (optional - can be slow)
- if extracted_url and len(issues) == 0:  # Only check URL if no other issues
+ # 4. Check URL accessibility — runs on every citation with a URL,
+ # regardless of other issues. The prior `len(issues) == 0` guard
+ # self-disabled this check on most real citations because date
+ # warnings are common, masking the very URL failures we cared about.
+ if extracted_url:

Six insertions, one deletion, no other behavioral changes. The check itself is the same urllib request the file always had — only the gate that decided whether to run it has changed.

First-run effect: the next run will surface a lot more URL warnings than every previous run. That's restored visibility, not new breakage. The citations that surface as warnings now were always broken; the validator just stopped reporting them when it noticed a stray date-format issue.

remove_invalid_sources.py — validate_url() gets a body

The original function:

req = urllib.request.Request(url, method='HEAD', headers={...})
with urllib.request.urlopen(req, timeout=timeout, context=ctx) as response:
    return (url, response.getcode(), "OK")

The new function reads the first ~32KB of the response and runs two phrase-list sniffs against it:

with httpx.Client(
    follow_redirects=True,
    timeout=timeout,
    headers={..., "Range": "bytes=0-32768"},
) as client:
    response = client.get(url)

code = response.status_code
if code >= 400:
    return (url, code, f"HTTP {code}")

content_type = response.headers.get("content-type", "").lower()
if "html" not in content_type:
    # PDF, JSON, plain text — pass through on status code alone
    return (url, code, f"HTTP {code} ({content_type or 'non-html'})")

body_text = response.text.lower()

for phrase in SOFT_404_PHRASES:
    if phrase in body_text:
        return (url, SOFT_404_BODY, f"Soft 404: body contains '{phrase}'")

for phrase in PAYWALL_PHRASES:
    if phrase in body_text:
        return (url, PAYWALL_STUB, f"Paywall: body contains '{phrase}'")

return (url, code, f"HTTP {code} (body verified)")

Three new module-level constants carry the signal:

HALLUCINATION_PATTERN = -1
SOFT_404_BODY = -2
PAYWALL_STUB = -3
CONTENT_INVALID_CODES = {HALLUCINATION_PATTERN, SOFT_404_BODY, PAYWALL_STUB}

And both decision blocks (workflow agent + CLI entry, lines ~593 and ~787) collapse the per-sentinel checks into a single set test:

if http_code in CONTENT_INVALID_CODES:   # hallucination, soft-404, or paywall stub
    invalid_citations.add(num)
elif http_code in INVALID_HTTP_CODES:    # 404, 410
    invalid_citations.add(num)
elif http_code in POTENTIALLY_VALID_CODES:  # 401, 403, 429, 500…
    potentially_valid.add(num)
...

The phrase lists are conservative. Soft-404 picks up the textbook "page not found / no longer available / has been removed" variants. Paywall picks up "sign in to continue / subscribe to read / start your free trial." Both lists are easy to extend; they live next to the function and need no library work to grow.

Diagram: what the three layers catch

                ┌─── (regex preflight, no network) ──→ HALLUCINATION_PATTERN (-1)
                │       e.g. example.com, XXXXX, /path/to/, {template-var}
                │
validate_url ───┼─── httpx GET, follow redirects, Range: 0-32768 ───┐
                │                                                   │
                │   HTTP 4xx/5xx ───→ INVALID / POTENTIALLY_VALID    │
                │   non-HTML 2xx ───→ pass through (real PDF, etc.)  │
                │                                                   │
                └─── HTML 2xx body sniff ────────────────────────────┤
                       ├─ soft-404 phrase ───→ SOFT_404_BODY (-2)    │
                       ├─ paywall phrase ────→ PAYWALL_STUB (-3)     │
                       └─ neither            ───→ "HTTP 200 (body verified)"

What the changes deliberately do not do

• Paywalled real sources still drop. A WSJ, FT, or Bloomberg paywall hit goes to the invalid bucket today, the same as a soft-404. The next step (Phase 1 step 4 in the rollout plan) wires a reputable-publisher allow-list so those become verified-gated instead of dropped, but until then, this commit will over-drop genuine premium sources behind paywalls. Expected; documented; reversible per-source by the analyst in the interim.

• URL-drift recovery isn't here yet. When a real article gets re-slugged, the new validator drops it. Step 5 of Phase 1 wires the "search by title + publisher → revalidate → accept first Jaccard ≥ 0.6 match" recovery loop. Not in this commit.

• citation_validator.py still has its own urllib check in a separate code path for the post-cleanup pass. Both validators currently exist. Consolidating to a single validate_url is later Phase 1 cleanup, not this commit.

• No new dependencies. httpx was already in requirements.txt and pyproject.toml for research_enhanced.py. The existing ThreadPoolExecutor wrapping is preserved — httpx.Client (sync) works fine inside threads, no async refactor required.

What's Next

Phase 1 has five more steps queued behind these two — a focused day's work, all in the same blast radius:

After Phase 1, the next move is the Sources.md codified-mode loader in research_enhanced.py — a per-deal inputs/Sources.md (frontmatter + analyst notes) that locks the run to a hand-curated source set, skipping the broad search entirely. That's the spine of the [[Human-Curated-Source-Sets-and-Per-Firm-RAG-for-Memo-Narrative|human-curated source set]] exploration that captures where the pipeline is heading more broadly.

The immediate concrete payoff is a clean ChromaDB-v0.0.8 regeneration whose source catalog actually means what it says — the same seven runs of artifacts also become the calibration baseline for the measurement layer (claim-vs-source check) when that lands.

References

• [[Trustworthy-Citations-Source-Harvester-Rollout]] — the six-phase plan; this commit lands Phase 1 steps 1–2

• [[Human-Curated-Source-Sets-and-Per-Firm-RAG-for-Memo-Narrative]] — the broader exploration this aligns with

• [[Faked-Sources-from-Perplexity]] — December 2025 symptom record

• [[Curating-only-valid-Sources-across-Runs]] — the post-mortem that surfaced the v0.0.1–v0.0.7 wreckage

• [[Separating-Retrieval-from-Generation-in-Agent-Pipelines]] — why the validator alone is not enough; the architectural direction Phase 2+ moves toward