Changelog - 2025-12-16

Citation Pipeline Fixes: Definition Preservation & Duplicate Key Handling

Overview

Critical fixes to the citation pipeline that were causing citation definitions (the source references at the bottom of files) to be lost during processing. This was manifesting as inline citation markers ([^1], [^2]) appearing in the final output without corresponding definitions, breaking footnote anchor links.

Root Cause Analysis

The citation loss was occurring due to two separate issues:

Issue 1: Writer Validation Gap

The writer agent's polish_section_research() function validated that:

1. The same number of inline [^N] markers existed after polishing

2. A ### Citations header was present

However, it did NOT validate that citation definitions ([^N]: Source...) actually existed. Claude could return a ### Citations header with empty content, and the validation would pass.

Issue 2: Perplexity Duplicate Citation Keys

Perplexity Sonar Pro outputs multiple definitions with the same key:

[^3]: Source A about topic...
[^3]: Source B about same topic...
[^3]: Source C about same topic...

This caused downstream confusion as only one definition would survive, leaving orphaned inline markers.

Bug Fixes

Writer Agent (src/agents/writer.py)

FIXED: Citation definition validation now checks actual definition count, not just header presence

# Count definitions in original research
defs_before = len(re.findall(r'^\[\^[a-zA-Z0-9_]+\]:', research_content, re.MULTILINE))
defs_after = len(re.findall(r'^\[\^[a-zA-Z0-9_]+\]:', polished_content, re.MULTILINE))

if defs_before > 0 and defs_after == 0:
    print(f"⚠️  WARNING: Citation definitions lost! Before: {defs_before}, After: {defs_after}")
    return research_content  # Fall back to original

if defs_after < defs_before * 0.5:  # Lost more than half
    print(f"⚠️  WARNING: Too many citation definitions lost!")
    return research_content  # Fall back to original

NEW: Image embed preservation validation

• Detects image embeds ![desc](path) before polishing

• Validates they're preserved after polishing

• Restores missing images if Claude removes them

Perplexity Section Researcher (src/agents/perplexity_section_researcher.py)

NEW: fix_duplicate_citation_keys() function

Handles Perplexity's duplicate key problem by:

1. Detecting duplicate [^3]: definitions

2. Renumbering to unique sequential keys: [^3], [^4], [^5]

3. Expanding inline references: [^3] becomes [^3] [^4] [^5]

4. Preserving canonical citations like [^deck]

# Before fix:
# [^3]: Source A...
# [^3]: Source B...
# Inline: claims [^3]

# After fix:
# [^3]: Source A...
# [^4]: Source B...
# Inline: claims [^3] [^4]

Citation Enrichment (src/agents/citation_enrichment.py)

FIXED: renumber_citations_globally() now handles alphanumeric citations

• Supports both numeric ([^1]) and alphanumeric ([^deck], [^ehrtime]) keys

• Uses dict to keep ONE definition per citation number (deduplication)

• Processes longer keys first to prevent partial replacement bugs

# Now handles: [^1], [^deck], [^3b], [^ehrtime]
old_citations = set(re.findall(r'\[\^([a-zA-Z0-9_]+)\]', main_content))

New Features

Two-Gate Citation Validation (src/workflow.py)

NEW: Dual validation gates prevent hallucination propagation

1. Gate 1: cleanup_research_citations (after research, before writer)

   • Validates URLs in 1-research/ files

   • Removes hallucinated/404 URLs before writer sees them

   • Prevents false information from entering draft narrative

2. Gate 2: cleanup_sections_citations (after revision, before assembly)

   • Validates citations in 2-sections/ files

   • Catches new hallucinations from enrichment

   • Ensures clean citations before final assembly

New Agents

src/agents/remove_invalid_sources.py (NEW)

• URL validation with parallel HTTP HEAD requests

• Hallucination pattern detection

• Citation removal and reordering utilities

• Configurable invalid HTTP codes and patterns

src/agents/citation_assembly.py (NEW)

• Dedicated citation assembly logic

• Handles consolidation after validation

src/agents/inject_deck_images.py (NEW)

• Injects pitch deck screenshots into sections

• Preserves images through polishing pipeline

Files Changed

Core Agents

• src/agents/writer.py - Citation definition validation, image preservation

• src/agents/perplexity_section_researcher.py - Duplicate key fix

• src/agents/citation_enrichment.py - Alphanumeric citation support

• src/agents/citation_validator.py - Enhanced validation

• src/agents/deck_analyst.py - Deck analysis improvements

New Agents

• src/agents/remove_invalid_sources.py - URL validation and cleanup

• src/agents/citation_assembly.py - Citation assembly

• src/agents/inject_deck_images.py - Deck image injection

Workflow

• src/workflow.py - Two-gate validation, new agent integration

Documentation

• context-vigilance/Anti-Hallucination-Source-Validation-and-Removal.md (NEW)

• context-vigilance/Faked-Sources-from-Perplexity.md (NEW)

• context-vigilance/An-Agent-should-Reorder-and-Organize-Citations-on-Assembly.md (NEW)

• context-vigilance/issue-resolution/Correct-Citation-Pipeline-Accuracy-in-Multi-Agent-Research.md (NEW)

Testing

Tested on ProfileHealth memo:

• Research phase: 84 citations generated

• Gate 1: Removed 6 invalid citations (78 remaining)

• Writer: Citation definitions preserved (fix working)

• Citation enrichment: 262 → 30 unique citations

• Gate 2: Removed 15 invalid citations (404s)

• Final output: 13 citations with working anchor links

Technical Details

Citation Definition Regex

# Matches: [^1]:, [^deck]:, [^ehrtime]:, [^3b]:
r'^\[\^[a-zA-Z0-9_]+\]:'

Validation Fallback Logic

1. Check definition count before/after polishing

2. If definitions dropped to 0 → use original research

3. If definitions dropped by >50% → use original research

4. Otherwise → accept polished content

Duplicate Key Expansion

# Input: [^3] appears once, but 3 definitions exist for [^3]:
# Output: [^3] [^4] [^5] - expanded to reference all sources

Migration Notes

For Future Development

• Always validate citation DEFINITIONS, not just inline markers

• Handle alphanumeric citation keys (not just numeric)

• Run validation BEFORE writer to prevent hallucination propagation

• Use parallel URL validation for performance

Anti-Patterns Identified

• ❌ Checking only for ### Citations header (header can be empty)

• ❌ Counting only inline [^N] markers (definitions can be missing)

• ❌ Assuming Perplexity uses unique citation keys (it doesn't)

• ❌ Processing citations only at assembly time (too late)

Known Issues Resolved

Contributors

• Citation pipeline debugging and fixes

• Two-gate validation architecture

• Perplexity duplicate key handling