Lossless Changelog
component

Major Architecture Refactor: Section-by-Section Processing

Complete refactor of the enrichment pipeline to process memo sections individually rather than as a single monolithic document. This eliminates API timeout issues and enables proper citation management with Perplexity Sonar Pro.

Changelog - 2025-11-20

Major Architecture Refactor: Section-by-Section Processing

Overview

Complete refactor of the enrichment pipeline to process memo sections individually rather than as a single monolithic document. This eliminates API timeout issues and enables proper citation management with Perplexity Sonar Pro.

Breaking Changes

Writer Agent (src/agents/writer.py)

  • CHANGED: No longer assembles full memo during writing phase

  • CHANGED: Returns empty draft_sections dict instead of full_memo

  • Now saves only individual section files to 2-sections/ directory

  • Section assembly moved to citation enrichment agent

Enrichment Agents

All enrichment agents now process section files directly instead of operating on state:

Citation Enrichment (src/agents/citation_enrichment.py)

  • CHANGED: Now loads sections from 2-sections/ files instead of state

  • CHANGED: Citation format now uses markdown links: [^1]: YYYY, MMM DD. [Title](URL). Published: YYYY-MM-DD | Updated: N/A

  • CHANGED: Removed deprecated URL: https://... suffix

  • FIXED: renumber_citations_globally() now consolidates ALL citations into ONE block at the end

  • FIXED: Citations properly stripped from individual sections before consolidation

  • FIXED: Multiple citations separated by single space: text. [^1] [^2]

Socials Enrichment (src/agents/socials_enrichment.py)

  • CHANGED: Processes Team section file (04-team.md) directly

  • CHANGED: Uses get_latest_output_dir() helper instead of version manager

  • Adds LinkedIn links to team members in-place in section file

  • CHANGED: Processes each section file independently

  • CHANGED: No longer attempts to process full 50k+ char memo (avoiding timeouts)

  • Each section enriched with organization/entity hyperlinks separately

Trademark Enrichment (src/agents/trademark_enrichment.py)

  • NEW: Creates header.md file with company logo/trademark

  • CHANGED: No longer tries to insert into assembled memo

  • Header file used by finalizer during assembly

Visualization Enrichment (src/agents/visualization_enrichment.py)

  • DISABLED: Temporarily disabled pending refactor for section-by-section processing

  • Returns skip message until re-implemented

Finalizer (src/workflow.py)

  • CHANGED: No longer assembles sections (done by citation enrichment)

  • CHANGED: Now verifies 4-final-draft.md exists instead of creating it

  • CHANGED: Uses get_latest_output_dir() helper

New Features

Version Resolution (src/utils.py)

  • NEW: Added get_latest_output_dir() helper function

  • Dynamically finds most recent output directory by scanning filesystem

  • Eliminates version sync issues between agents

  • All enrichment agents now use consistent version resolution

Version Manager (src/versioning.py)

  • NEW: Added get_current_version() method

  • Returns latest version from versions.json

  • Used alongside get_next_version() for different use cases

Bug Fixes

Citation Format

  • FIXED: Citations now consistently use markdown link format

  • FIXED: Multiple citations properly spaced: [^1] [^2] not [^1][^2]

  • FIXED: Citation placement after punctuation with space: text. [^1]

  • FIXED: Consolidated citation block (ONE block at end, not duplicates)

Version Management

  • FIXED: Enrichment agents no longer fail due to version mismatches

  • FIXED: Dynamic directory resolution prevents "v0.0.2 vs v0.0.3" errors

  • FIXED: All agents now find same output directory consistently

API Timeout Prevention

  • FIXED: No more 50k+ character API payloads

  • FIXED: Each section processed independently (~5k chars each)

  • FIXED: Link enrichment no longer times out on large memos

Technical Details

Section-by-Section Flow

  1. Writer: Saves 10 individual section files to 2-sections/

  2. Trademark: Creates header.md with company logo (if provided)

  3. Socials: Enriches Team section with LinkedIn links

  4. Link: Enriches each section with organization hyperlinks

  5. Visualization: Skipped (pending refactor)

  6. Citation:

    • Loads each section file

    • Enriches with Perplexity Sonar Pro citations

    • Saves enriched section back

    • Renumbers citations globally

    • Consolidates into ONE citation block

    • Saves assembled 4-final-draft.md

  7. Validator: Validates final draft

  8. Finalizer: Verifies draft exists, saves state

Citation Renumbering Algorithm

PYTHON
# Each section has local citations [^1][^2][^3]
# Section 1: [^1][^2] -> renumbered to [^1][^2]
# Section 2: [^1][^2][^3] -> renumbered to [^3][^4][^5]
# Section 3: [^1] -> renumbered to [^6]
# Result: Global sequence [^1][^2][^3][^4][^5][^6]
# All citation definitions consolidated into ONE block at end

Files Changed

Core Agents

  • src/agents/writer.py - Section-only saving, no assembly

  • src/agents/citation_enrichment.py - Section-by-section + consolidation

  • src/agents/socials_enrichment.py - File-based processing

  • src/agents/link_enrichment.py - File-based processing

  • src/agents/trademark_enrichment.py - Header file creation

  • src/agents/visualization_enrichment.py - Temporary disable

Utilities

  • src/utils.py - NEW: Helper functions

  • src/versioning.py - Added get_current_version()

Workflow

  • src/workflow.py - Finalizer refactor

Configuration

  • CLAUDE.md - Updated troubleshooting notes

  • README.md - Updated architecture documentation

Migration Notes

For Future Development

  • All enrichment agents MUST process section files, not state

  • Use get_latest_output_dir() for consistent version resolution

  • Citation blocks MUST be stripped before assembly

  • Section headers added during assembly, not in section content

Deprecated Patterns

  • ❌ draft_sections["full_memo"]["content"] - NO LONGER POPULATED

  • ❌ version_mgr.get_current_version() alone - Use get_latest_output_dir() instead

  • ❌ Processing full memo in enrichment - Process sections individually

Performance Improvements

Before

  • Single 50k+ char API call for citations → frequent timeouts

  • Full memo reassembled after each enrichment → memory intensive

  • Version sync issues → manual intervention required

After

  • 10 separate ~5k char API calls → no timeouts

  • Sections enriched in-place → minimal memory overhead

  • Dynamic version resolution → zero manual intervention

Known Issues

Visualization Enrichment

  • Currently disabled, pending refactor for section-by-section processing

  • Will be re-enabled in future release

Section Headers

  • Duplicate headers may appear if section content includes its own header

  • Assembly process adds numbered headers (e.g., ## 1. Executive Summary)

  • Section files should contain content only, no headers

Testing

Tested on:

  • Avalanche VC Fund II memo (fund commitment, retrospective)

  • All 10 sections processed successfully

  • Citations properly formatted with markdown links

  • Single consolidated citation block at end

  • No API timeouts during link or citation enrichment

Contributors

  • Architecture refactor and bug fixes

  • Citation format standardization

  • Version resolution system

References

  • Issue: Citations losing format after pipeline execution

  • Issue: API timeouts on large memos (50k+ chars)

  • Issue: Version mismatch between enrichment agents

  • Issue: Duplicate citation blocks in final output