Lossless Changelog
component

Summary

Added a comprehensive scorecard evaluation system that scores completed investment memos against a structured 12-dimension framework (12Ps: Persona, Pain, Proposition, Problem, Possibility, Positioning, People, Process, Product, Potential, Progress, Plan).

Changelog: 12Ps Scorecard Evaluation System

Date: 2025-11-30 Type: New Feature Scope: Post-generation evaluation, new agent, CLI tool

Summary

Added a comprehensive scorecard evaluation system that scores completed investment memos against a structured 12-dimension framework (12Ps: Persona, Pain, Proposition, Problem, Possibility, Positioning, People, Process, Product, Potential, Progress, Plan). This enables quantified quality assessment with evidence, improvement suggestions, and diligence questions.

Problem

Investment memos lacked a structured, repeatable evaluation framework:

  • Quality was assessed subjectively (existing validator scores 0-10 without dimension breakdown)

  • No standardized way to identify specific strengths and weaknesses

  • No automated generation of follow-up diligence questions

  • No quantified benchmarking against a defined rubric

Solution

New Files Created

1. src/scorecard_loader.py (~370 lines)

YAML scorecard definition loader with:

PYTHON
@dataclass
class DimensionDefinition:
    name: str
    group: str
    number: int
    short_description: str
    full_description: str
    evaluation_guidance: EvaluationGuidance  # questions, evidence_sources, red_flags
    scoring_rubric: ScoringRubric  # 1-5 scale with descriptions

def load_scorecard(scorecard_name: str) -> ScorecardDefinition:
    """Load and parse YAML scorecard, with caching."""

Features:

  • Parses YAML scorecards from templates/scorecards/{type}/{name}.yaml

  • Caches loaded scorecards for reuse

  • Provides helpers: get_dimension_rubric(), get_percentile_label(), get_score_label()

2. src/agents/scorecard_evaluator.py (~550 lines)

New LangGraph agent that:

PYTHON
def scorecard_evaluator_agent(state: MemoState) -> Dict[str, Any]:
    """Score memo against scorecard dimensions, save artifacts."""

Evaluation process:

  1. Loads sections from 2-sections/*.md

  2. Scores each of 12 dimensions via Claude (temperature=0.3 for consistency)

  3. Extracts evidence citations and improvement suggestions per dimension

  4. Calculates group averages and overall score (1-5 scale)

  5. Identifies strengths (score ≥4) and concerns (score ≤2)

  6. Generates diligence questions from low-scoring dimension guidance

  7. Saves to 5-scorecard/12Ps-scorecard.md and .json

Retry logic for API errors with exponential backoff.

3. cli/score_memo.py (~490 lines)

Standalone CLI for scoring existing memos:

BASH
# Score by company name (finds latest version)
python cli/score_memo.py "Company" --scorecard hypernova-early-stage-12Ps

# Score by path
python cli/score_memo.py output/Company-v0.0.2 --scorecard hypernova-early-stage-12Ps

# Custom model
python cli/score_memo.py "Company" --model claude-sonnet-4-5-20250929

Output:

🎯 Scoring 12 dimensions...
   [1/12] Persona... 4/5
   [2/12] Pain... 3/5
   ...
✅ Scorecard complete!
   Overall Score: 3.6/5
   Strengths: 4
   Concerns: 2

State & Workflow Integration

src/state.py

Added new TypedDicts:

PYTHON
class DimensionScore(TypedDict, total=False):
    score: int              # 1-5 scale
    percentile: str         # e.g., "Top 10-25%"
    evidence: str           # Key evidence supporting the score
    improvements: List[str] # What would make this score higher

class ScorecardResults(TypedDict, total=False):
    scorecard_name: str
    overall_score: float
    dimensions: Dict[str, DimensionScore]
    groups: Dict[str, float]  # Group average scores
    strengths: List[str]      # Dimension IDs with scores >= 4
    concerns: List[str]       # Dimension IDs with scores <= 2
    diligence_questions: List[str]

Added to MemoState:

  • scorecard_name: Optional[str] - Scorecard to use (from company JSON)

  • scorecard_results: Optional[ScorecardResults] - Evaluation results

src/workflow.py

New workflow routing:

... → validate → scorecard → [finalize | human_review]

  • Added scorecard node after validate

  • Conditional routing now happens after scorecard (was after validate)

src/main.py

  • Loads scorecard field from data/{Company}.json

  • Passes scorecard_name to generate_memo()

  • Displays scorecard results at end:

Validation Score: 8.5/10
Scorecard Score: 3.8/5 (5 strengths, 1 concerns)

Output Structure

output/{Company}-v0.0.x/
├── 0-deck-analysis.json/.md
├── 1-research.json/.md
├── 2-sections/*.md
├── 3-validation.json/.md
├── 4-final-draft.md
├── 5-scorecard/              # NEW
│   ├── 12Ps-scorecard.md     # Human-readable report
│   └── 12Ps-scorecard.json   # Structured data
└── state.json

Scorecard Markdown Output

MARKDOWN
# Company 12Ps Scorecard Evaluation

**Company**: Example Corp
**Scorecard**: Hypernova Early-Stage 12Ps
**Date**: November 30, 2025

## Scorecard Summary

| Group | Dimensions | Avg Score |
|-------|-----------|-----------|
| Origins | Persona, Pain, Proposition | 3.7/5 |
| Opening | Problem, Possibility, Positioning | 3.3/5 |
...

**Overall Score: 3.5/5**

## Origins

| Persona | Pain | Proposition |
|---------|------|-------------|
| **4/5** | **3/5** | **4/5** |
| Top 10-25% | Top 50% | Top 10-25% |

### 1. Persona — **4/5**

**Evidence**: The memo clearly identifies target customers...

**What Could Make This Score Higher**:
- Include specific customer interviews or LOIs
- Quantify the number of target customers approached
...

## Key Findings

### Standout Strengths (Scores of 4+)
- **Persona (4/5)**: Clear target customer definition...
- **Product (4/5)**: Differentiated product design...

### Areas of Concern (Scores of 1-2)
- **Progress (2/5)**: Limited traction metrics provided...

### Critical Questions for Diligence
- [Progress] What specific milestones have been achieved in the past 6 months?
- [Progress] Are there any signed LOIs or pilot agreements?

Configuration

Add scorecard to company JSON to enable evaluation:

JSON
{
  "type": "direct",
  "mode": "justify",
  "scorecard": "hypernova-early-stage-12Ps",
  "description": "...",
  "url": "..."
}

Scorecard YAML files live in templates/scorecards/direct-early-stage-12Ps/.

Files Changed Summary

New Files (3)

FileLinesPurpose
src/scorecard_loader.py~370YAML scorecard parser with caching
src/agents/scorecard_evaluator.py~550LangGraph agent for dimension scoring
cli/score_memo.py~490Standalone CLI for existing memos

Modified Files (3)

FileChangesPurpose
src/state.py+30 linesDimensionScore, ScorecardResults TypedDicts
src/workflow.py+15 linesScorecard node, routing update
src/main.py+20 linesLoad scorecard, display results

Usage

Full Pipeline (with scorecard)

BASH
source .venv/bin/activate
python -m src.main "Company"  # Uses scorecard from data/Company.json if specified

Standalone Scoring (existing memo)

BASH
source .venv/bin/activate
python cli/score_memo.py "Company Name" --scorecard hypernova-early-stage-12Ps

Dependencies

No new dependencies. Uses existing:

  • langchain-anthropic for Claude API

  • pyyaml for scorecard parsing

Next Steps

  1. Create actual scorecard YAML in templates/scorecards/direct-early-stage-12Ps/

  2. Consider adding scorecard-driven section improvement suggestions

  3. Add scorecard comparison across multiple companies/versions

Contributors

Implementation by Claude Code (Anthropic) with user guidance.