Changelog - 2026-01-27

Citation Pipeline Fixes: Alphanumeric Key Support & Duplicate Section Prevention

Overview

Fixes to the citation pipeline addressing two critical issues: (1) the writer agent's regex only matched numeric citation keys ([^1]) and missed alphanumeric keys ([^deck]), causing citation loss during polishing; and (2) the citation enrichment agent was appending duplicate ### Citations sections instead of merging them.

Root Cause Analysis

Issue 1: Numeric-Only Citation Regex in Writer

The writer agent's polish_section_research() function used \[\^(\d+)\] regex which only matched numeric citation keys. Alphanumeric keys like [^deck], [^source_name] were completely ignored during validation, causing:

• Research files with [^deck], [^1], [^2] would report only 2 citations (missing [^deck])

• LLM could silently rename [^deck] to [^10] and validation would pass

• Citation definitions became orphaned when keys changed

Issue 2: Duplicate Citation Sections in Enrichment

The citation enrichment agent split content on ### Citations but only handled the first occurrence. When Perplexity returned content that included existing citations, the code would append a new ### Citations section, resulting in:

### Citations
[^1]: Source A...

---

### Citations
[^1]: Source A...  (duplicate)
[^2]: Source B...  (new)

Bug Fixes

Writer Agent (src/agents/writer.py)

FIXED: Citation regex now supports alphanumeric keys

# Before (broken):
citations_before = len(set(re.findall(r'\[\^(\d+)\]', research_content)))

# After (fixed):
all_citations_before = set(re.findall(r'\[\^([a-zA-Z0-9_]+)\]', research_content))
citations_before = len(all_citations_before)

NEW: Citation key renaming detection

• Validates that inline citation keys after polishing match original keys

• Detects when LLM silently renames [^deck] to [^10]

• Falls back to original research if keys were renamed

# Check if LLM renumbered citations (inline keys don't match original)
if inline_keys_after != all_citations_before:
    renamed_keys = inline_keys_after - all_citations_before
    if renamed_keys:
        print(f"⚠️  WARNING: LLM renamed citation keys! New keys: {renamed_keys}")
        return research_content  # Fall back to original

NEW: Definition-to-inline key matching

• Validates that all inline citations have corresponding definitions

• Catches cases where LLM includes inline refs but omits definitions

missing_defs = inline_keys_after - definition_keys_after
if missing_defs:
    print(f"⚠️  WARNING: Missing definitions for inline citations: {missing_defs}")
    return research_content

UPDATED: Prompt improvements

• Explicitly instructs LLM not to rename citation keys

• Lists actual citation keys in prompt for clarity

• Requires copying definitions exactly as they appear

Citation Enrichment (src/agents/citation_enrichment.py)

FIXED: Duplicate citation section prevention

# Strip ALL citation sections from enriched content before adding merged block
while "### Citations" in final_content:
    parts = final_content.split("### Citations", 1)
    final_content = parts[0].rstrip()
    # Also remove any preceding "---" separator
    if final_content.endswith("---"):
        final_content = final_content[:-3].rstrip()

Resume Script (cli/resume_from_interruption.py)

FIXED: Checkpoint detection now includes all agents

• Added scorecard and integrate_scorecard checkpoints

• Fixed workflow order documentation in docstring

• Improved checkpoint detection logic for final draft

New Scripts

scripts/fix_duplicate_citations.py (NEW)

Utility script to clean up research files that have duplicate ### Citations sections.

python scripts/fix_duplicate_citations.py io/dark-matter/deals/ExoLux/outputs/ExoLux-v0.0.2/1-research

Features:

• Detects files with multiple ### Citations sections

• Consolidates all citation definitions into single section

• Preserves all unique definitions

• Sorts keys: alphabetic first, then numeric

Files Changed

Core Agents

• src/agents/writer.py - Alphanumeric citation regex, key renaming detection, definition matching

• src/agents/citation_enrichment.py - Duplicate section prevention

• src/agents/citation_assembly.py - Minor fixes

• src/agents/citation_validator.py - Minor fixes

• src/agents/validator.py - Minor fixes

CLI Tools

• cli/resume_from_interruption.py - Checkpoint detection improvements

New Scripts

• scripts/fix_duplicate_citations.py - Duplicate citation section cleanup utility

Testing

Tested on ExoLux and Freecurve deals:

• ExoLux: Citations properly preserved through polishing (12 unique citations)

• Freecurve: Full pipeline run from scratch, citations maintained

• Fix script: Successfully cleaned 3 research files with duplicate sections

Technical Details

Citation Key Regex Patterns

# Inline citations (excludes definitions)
r'\[\^([a-zA-Z0-9_]+)\](?!:)'

# Citation definitions
r'^\[\^([a-zA-Z0-9_]+)\]:'

Validation Logic Flow

1. Extract all citation keys before polishing

2. Polish content with LLM

3. Extract citation keys after polishing

4. Check: Did we lose any citations? → Fall back

5. Check: Did LLM rename any keys? → Fall back

6. Check: Do all inline refs have definitions? → Fall back

7. Accept polished content only if all checks pass

Known Issues Resolved

Migration Notes

For Existing Deals

If you have research files with duplicate citation sections, run:

python scripts/fix_duplicate_citations.py <path-to-1-research-folder>

Anti-Patterns Identified

• ❌ Using \[\^(\d+)\] to match citations (misses alphanumeric keys)

• ❌ Splitting on ### Citations without handling multiple occurrences

• ❌ Trusting LLM to preserve citation keys (it will rename them)

• ❌ Checking only citation count, not actual key values