Changelog - 2025-12-09 (#1)

Fix: 12Ps Scorecard Section Excuses in Final Draft

Overview

This release fixes an issue where the 12Ps Scorecard Summary section (section 08) would contain LLM "excuses" instead of actual scorecard data. The problem occurred when the citation enrichment agent attempted to enrich a placeholder section with Perplexity web research, which failed because Perplexity couldn't find the specific company data. The agent would then output meta-commentary like "I cannot add citations to claims about..." instead of useful content.

Problem

The workflow sequence had a timing issue with 12Ps outlines:

draft (placeholder section 08) → cite (tries to enrich with Perplexity) → ... → scorecard (generates actual data) → integrate_scorecard

Issues:

1. Citation enrichment agent called Perplexity Sonar Pro on the placeholder section 08

2. Perplexity couldn't find company-specific scorecard data via web search

3. The LLM wrote "excuses" explaining why it couldn't help: "I appreciate your detailed request, but I need to clarify an important limitation..."

4. These excuses were saved to the section file, corrupting it

5. integrate_scorecard should have replaced this content, but used basic file concatenation that didn't handle citations/TOC properly

Observed in: FortemNeuroscience memo, section 08-12ps-scorecard-summary.md

Solution: Two Fixes

Fix 1: Skip Scorecard Section During Citation Enrichment

For 12Ps outlines, skip section 08 during citation enrichment since it will be replaced by the scorecard agent anyway.

File: src/agents/citation_enrichment.py

# Check if using 12Ps outline (scorecard section will be generated separately)
outline_name = state.get("outline_name", "")
is_12ps_outline = "12Ps" in outline_name or "12ps" in outline_name

for section_file in section_files:
    section_name = section_file.stem.split("-", 1)[1].replace("--", " & ").replace("-", " ").title()

    # Skip scorecard section for 12Ps outlines - it will be replaced by scorecard agent
    if is_12ps_outline and ("scorecard" in section_file.stem.lower() or section_file.stem.startswith("08-")):
        print(f"  ⏭️  Skipping {section_name} (will be replaced by scorecard agent)")
        # Still include in sections_data for final assembly
        with open(section_file) as f:
            section_content = f.read()
        section_num = section_file.stem.split("-")[0]
        sections_data.append((section_num, section_name, section_content))
        continue

Fix 2: Use Canonical Assembly in integrate_scorecard

The integrate_scorecard function now uses the canonical assemble_draft module instead of basic file concatenation. This ensures proper:

• Citation renumbering (global consolidation)

• Table of Contents regeneration

File: src/workflow.py

# Reassemble final draft using canonical assembly (handles citations + TOC)
try:
    from cli.assemble_draft import assemble_final_draft
    from rich.console import Console
    console = Console()
    final_draft_path = assemble_final_draft(output_dir, console)
    word_count = final_draft_path.read_text().split()
    print(f"  ✓ Reassembled final draft with citations and TOC: {len(word_count)} words")
    return {"messages": [f"Scorecard integrated into section 8, final draft reassembled ({len(word_count)} words)"]}
except ImportError:
    # Fallback to basic assembly if cli module not available
    # ... (basic concatenation logic preserved as fallback)

Additional Fix: 12Ps Sections in improve_section.py

Added 12Ps outline section names to the SECTION_MAP in cli/improve_section.py so the section improvement tool works with 12Ps memos:

# 12Ps framework sections (direct-early-stage-12Ps outline)
"Origins": (2, "02-origins.md"),
"Opening": (3, "03-opening.md"),
"Organization": (4, "04-organization.md"),
"Offering": (5, "05-offering.md"),
"Opportunity": (6, "06-opportunity.md"),
"Risks & What Could Go Wrong": (7, "07-risks--what-could-go-wrong.md"),
"12Ps Scorecard Summary": (8, "08-12ps-scorecard-summary.md"),
"Closing Assessment": (10, "10-closing-assessment.md"),

Updated Workflow Flow (12Ps Outline)

draft → cite (SKIPS section 08) → toc → ... → scorecard → integrate_scorecard
                                                   ↓
                                         Generates 5-scorecard/12Ps-scorecard.md
                                                   ↓
                                         Copies to 08-12ps-scorecard-summary.md
                                                   ↓
                                         Reassembles with canonical assembly
                                         (citations + TOC)

Files Changed

Impact

Before This Fix

• Citation enrichment tried to enrich placeholder section 08 with Perplexity

• Perplexity failed to find company data, outputting excuses

• Final draft contained LLM meta-commentary instead of scorecard

• Users had to manually run assemble_draft.py to fix the issue

After This Fix

• Citation enrichment skips section 08 for 12Ps outlines

• Scorecard agent generates proper content in 5-scorecard/

• integrate_scorecard copies scorecard to section 08

• Final draft assembled with proper citations and TOC

• Zero manual intervention required

Example Output

📚 Enriching citations section-by-section...
  Enriching citations: Executive Summary...
  ✓ Executive Summary: 12 citations added
  Enriching citations: Origins...
  ✓ Origins: 18 citations added
  ...
  ⏭️  Skipping 12Ps Scorecard Summary (will be replaced by scorecard agent)
  Enriching citations: Funding Terms...
  ✓ Funding Terms: 8 citations added

Testing

Manual Verification

# Run memo generation for a 12Ps deal
source .venv/bin/activate
python -m src.main --firm dark-matter --deal FortemNeuroscience

# Verify section 08 contains actual scorecard (not excuses)
head -30 io/dark-matter/deals/FortemNeuroscience/outputs/FortemNeuroscience-v0.0.1/2-sections/08-12ps-scorecard-summary.md

Expected Behavior

After running memo generation:

1. Section 08 is skipped during citation enrichment (logged as "⏭️ Skipping")

2. 5-scorecard/12Ps-scorecard.md contains detailed 12Ps evaluation

3. 2-sections/08-12ps-scorecard-summary.md matches scorecard file

4. 4-final-draft.md includes full scorecard with proper TOC and citations

Root Cause Analysis

The "excuses" pattern occurs when:

1. An LLM agent is asked to enrich content that doesn't have enough context

2. External API (Perplexity) can't find relevant data via web search

3. Instead of failing gracefully, the LLM outputs meta-commentary about its limitations

Prevention Strategy:

• Skip sections that will be replaced by downstream agents

• Ensure replacement agents use proper assembly tools

• Add explicit skip conditions for outline-specific sections

Related Documentation

• changelog/2025-12-06_02.md - Original integrate_scorecard implementation

• templates/outlines/direct-early-stage-12Ps.yaml - 12Ps outline definition

• cli/assemble_draft.py - Canonical assembly with citation consolidation

Contributors

Implementation by Claude Code (Anthropic) with user guidance.