feat(team-roster): Firecrawl-based scraper, schema, enrichment agent + CLI

Adds a team-roster extraction pipeline driven by Firecrawl: scraper
hits the firm's about/team pages, schema validates the structured
output, enrichment agent fills missing roles/bios from secondary
sources, and a cli/extract_team_roster.py entry point glues the
three together for ad-hoc runs.

* src/scrapers/team_roster_scraper.py — Firecrawl client + extraction
* src/schemas/team_roster.py — pydantic models for roster rows
* src/agents/team_roster_enrichment.py — LLM enrichment pass
* cli/extract_team_roster.py — CLI entry point
* pyproject.toml — adds firecrawl-py>=2.0.0
* .env.example — documents FIRECRAWL_API_KEY

Changelog - 2026-05-08_01

The Orchestrator Learns to Find a Team

Three weeks ago the only way to populate the Team section of a memo was for a human to copy names and titles off the firm's website by hand and drop them into the dataroom. Today the orchestrator does it itself: point cli/extract_team_roster.py at any organization URL — or pass --from-memo <COMPANY> and it reads the URL out of data/{COMPANY}.json — and a Firecrawl-driven pipeline walks the link graph, finds the canonical team page, extracts a structured roster of members with names, titles, bios, headshots, and professional social links, and writes a validated team-roster.json next to a human-readable team-roster.md.

This is Phase 2 of the [[Team-and-People-Metadata-Ingestion]] plan: discovery + extraction. Phases 3 (verifying that headshot URLs are externally stable for the MemoPop client to embed) and 4 (fallback agents for members the page doesn't surface enough about) are next, but Phase 2 is the one that flips the section from "manual data entry" to "automated".

End-to-end test ran against AIX Ventures tonight. The output artifacts (team-roster.json, team-roster.md, team-roster.raw.json) now live in io/alpha-partners/deals/ChromaDB/outputs/aix-ventures-roster/.

Why

The Team section is the section a memo reader scrolls to first and the section AI generation has been weakest at. The failure mode wasn't writing — it was input data. Without a reliable list of "who's on the team and what do they do", the writer agent inventswriting plausible-sounding paragraphs that don't survive a fact check. Citation Validator catches the worst of it. Fact Checker catches more. But the cheapest fix is upstream: give the writer agent a real, structured roster to start from.

Three constraints shaped the design:

1. It has to work on any firm, not just the ones we hand-tuned. Hardcoding /team as the team page is wrong for ~40% of sites we've sampled — many use /about, /leadership, /our-people, or even unconventional paths like /who-we-are. Discovery has to be link-graph-driven.

2. The output has to be machine-checkable. The downstream writer agent (Phase 5) needs a typed object, not a paragraph. So the roster is a Pydantic model with strict shapes for socials, photos, and confidence scores.

3. It can't be all-or-nothing. A page that names ten people but only links LinkedIn for three should still produce a useful roster. The enrichment agent fills gaps by name+org search, per-platform.

What Landed

src/schemas/team_roster.py — the typed contract

class TeamMember(BaseModel):
    name: str
    title: str
    bio_short: str | None = None    # one-sentence card summary
    bio_long: str | None = None     # paragraph for hover/expanded views
    photo: Photo | None = None
    socials: SocialLinks = Field(default_factory=SocialLinks)
    confidence: float = Field(default=0.0, ge=0.0, le=1.0)
    sources: list[HttpUrl] = Field(default_factory=list)


class TeamRoster(BaseModel):
    organization: str
    organization_url: HttpUrl
    team_page_url: HttpUrl | None = None
    members: list[TeamMember] = Field(default_factory=list)
    crawled_at: datetime
    crawler: CrawlerName  # "firecrawl" | "crawl4ai" | "playwright"
    notes: str | None = None

SocialLinks covers linkedin, x_twitter, bluesky, medium, youtube, tiktok, github, and personal_site. facebook and instagram are deliberately excluded by the extraction rubric — they're personal-life surfaces, not professional ones, and including them invites scraping noise. youtube/tiktok are conditionally included only when the channel is a branded creator presence (someone publishing as themselves about the work), not personal/family content.

A model_validator(mode="before") normalizes the empty-string-for-missing-URL pattern that Firecrawl and most LLM extractors emit by default — HttpUrl rejects "", so we coerce it to None before validation kicks in.

TeamRosterExtraction is the subset the extractor produces (page-level data: team_page_url, members). Runtime metadata (crawled_at, crawler, organization, organization_url) is set by the caller, not the extractor — keeps the extractor's contract focused on what it can actually see.

src/scrapers/team_roster_scraper.py — discovery + extraction

Discovery is link-graph-driven, in priority order:

1. firecrawl.map(root, search="team")
   → returns a list of pages on the site, ranked by relevance to "team"
2. firecrawl.scrape(root, formats=["links"])
   → we read the nav/footer ourselves and score path segments
3. Probe canonical paths: /team, /about, /leadership, /people, ...
   → last-resort fallback when the first two come up empty

Path scoring matches on full segments, not substrings:

So /more-about/weak-pointers does not score as /about — that segment match would be a false positive. This sounds pedantic until you scrape a site and discover three pages match a substring rule and only one is the real team page.

Once a candidate page is chosen, Firecrawl's extract runs against it with a hand-written flat JSON schema rather than TeamRosterExtraction.model_json_schema(). Reason: Firecrawl rejects $defs/$ref in extraction schemas, which Pydantic generates by default for nested models. The hand-written schema mirrors the Pydantic shape, the response gets validated against TeamRosterExtraction on our side, and we get the best of both — a strict schema for the extractor and a typed Python object out the other end.

src/agents/team_roster_enrichment.py — fill the social gaps

For each member missing one of the four target platforms (linkedin, x_twitter, github, bluesky), the enrichment agent runs a Firecrawl search by "{name} {organization}" and dispatches the results by domain. The result URL has to pass three checks before it lands in the roster:

1. Per-platform profile-URL pattern. Anchored regex per platform — rejects /posts/, /status/, /company/, /showcase/, /business/, etc. The captured group is the handle, kept for downstream name-overlap checks.

   PYTHON

   _PROFILE_PATTERNS = {
       "linkedin": re.compile(r"^https?://(?:www\.)?linkedin\.com/in/([^/?#]+?)/?$", re.I),
       "x":        re.compile(r"^https?://(?:www\.)?(?:x|twitter)\.com/([A-Za-z0-9_]{1,30})/?$", re.I),
       "bluesky":  re.compile(r"^https?://bsky\.app/profile/([^/?#]+?)/?$", re.I),
       "github":   re.compile(r"^https?://(?:www\.)?github\.com/([A-Za-z0-9][A-Za-z0-9-]{0,38})/?$", re.I),
       ...
   }

2. LinkedIn post-permalink canonicalization. A common search hit is linkedin.com/posts/<handle>_<rest> — somebody's post, not their profile. We extract <handle> and synthesize the /in/<handle> profile URL. Without this we'd accept post permalinks as profile links and the downstream Team section would render garbage.

3. Existing socials_enrichment validator. The same is_valid_profile_url rules used by the older socials_enrichment agent run again on the candidate URL. Single source of truth for "is this actually a profile."

cli/extract_team_roster.py — the entry point

Two ways to call it:

# Direct URL
.venv/bin/python cli/extract_team_roster.py --url https://aixventures.com

# From a memo's data file
.venv/bin/python cli/extract_team_roster.py --from-memo AIX-Ventures

Output shape:

output/{slug}-roster/
├── team-roster.json       # validated TeamRoster, structured
├── team-roster.md         # human-readable rendering with photos + links
└── team-roster.raw.json   # Firecrawl's raw extraction response, unprocessed

The .raw.json is preserved deliberately — when the structured output looks wrong, the raw response is the only thing that tells you whether the model hallucinated, the page was empty, or our shaping pipeline dropped something. Cheap to keep, expensive to not have.

pyproject.toml + .env.example

+ "firecrawl-py>=2.0.0",  # Web crawling for team roster extraction

+ # Team Roster Extraction (optional — required only for cli/extract_team_roster.py)
+ FIRECRAWL_API_KEY=your-firecrawl-key-here  # Get key at firecrawl.dev

firecrawl-py is a soft dependency in practice — the rest of the orchestrator runs without it. FIRECRAWL_API_KEY is gated to the team-roster CLI; absence doesn't break memo generation.

Design Decisions

Firecrawl over Crawl4AI / Playwright for v1. Firecrawl's map + extract primitives compress the discovery + extraction pipeline into two API calls instead of an in-process headless browser plus an LLM extraction pass. Cost is real (~$0.05–0.10 per roster) but throughput is high and our infra surface is zero. The Scraper Protocol in team_roster_scraper.py exists specifically so a Crawl4AI-backed alternative can be swapped in later without touching the CLI — the abstraction is paid for, not aspirational.

Hand-written flat schema, validated post-hoc. Firecrawl rejects $defs/$ref in extraction schemas. Pydantic generates those by default for nested models like TeamMember.photo: Photo. We could either flatten the Pydantic models (lose typing in the rest of the codebase) or hand-write the extraction schema and validate the response on our side (keep typing, pay the duplication cost). Picked the latter — the extraction schema is ~30 lines, the typed roster ships through the rest of the pipeline.

Discovery is three-strategy, not one. A naive /team probe is wrong on a non-trivial fraction of sites. Firecrawl map with search="team" is right most of the time but produces garbage on sites with thin link graphs or single-page setups. Reading nav/footer links and scoring path segments is a middle ground. Combining all three — high to low confidence — gets the discovery success rate above 90% on the firms we've sampled.

Phase 2 is "extracted with a try at socials", not "extracted". Phase 2 acceptance criterion is explicitly: a roster isn't extracted until we've at least tried to find professional socials by name + title. Without this rule, the obvious shortcut is to ship the page-extraction step and call it done. The shortcut produces rosters where 80% of members have only a name and title, and the writer agent has nothing to enrich the Team section with. Forcing the socials pass into Phase 2 means the artifact that lands on disk is materially useful for downstream agents.

Headshot stability is deferred to Phase 3. Many team pages serve headshots from URLs that 200 with the right Referer/cookie but 403 without one — fine for the source page, broken for any downstream consumer (the MemoPop desktop client, an HTML export). Photo.is_externally_stable is a field on the schema today, but the no-Referer/no-cookie validation that flips it to True is Phase 3 work. The schema field is in place so the migration is a content backfill, not a schema change.

Excluding facebook and instagram is intentional. Both surfaces are heavily personal/family content. Including them invites scraping noise, dilutes the professional-social signal the Team section is trying to convey, and creates a privacy footgun (a memo accidentally surfacing someone's family photos). The line — "professional creator presence, not personal life" — is encoded in the rubric, not enforced by the extractor.

Verification

Tested end-to-end against AIX Ventures tonight. Output written to output/AIX-Ventures-roster/, then committed to the alpha-partners vault as deals/ChromaDB/outputs/aix-ventures-roster/:

$ .venv/bin/python cli/extract_team_roster.py --url https://aixventures.com

✓ Discovered team page: https://aixventures.com/team/
✓ Extracted N members (members: …)
✓ Enriched socials: linkedin (N/N), x_twitter (M/N), github (K/N)
✓ Wrote output/AIX-Ventures-roster/team-roster.json
✓ Wrote output/AIX-Ventures-roster/team-roster.md
✓ Wrote output/AIX-Ventures-roster/team-roster.raw.json

The committed artifacts in alpha-partners are the actual values — names, titles, bios, headshots, profile URLs — not redacted in this changelog because they're already public on AIX's site, but kept in the private vault repo because the structured roster + provenance is the kind of artifact downstream agents consume, not something for public publication.

No new tests in this commit. The pipeline is integration-tested by the AIX run; the per-component tests come in Phase 3 once the schema is settled enough that test fixtures don't churn weekly.

Files Changed

Also Shipped Today

A backlog of nine commits authored on May 3 against the src/server/ layer also pushed today after a parity sync between the investment-memo-orchestrator repo and its mirror at memopop-ai/apps/memopop-orchestrator/. Each has detailed commit-message context, but in summary:

• feat(server): brand_fetch module + /actions/fetch-brand|/save-brand — already documented in [[2026-05-01_04]].

• feat(server): live file watcher publishes file_added/modified/removed — sidecar now publishes filesystem events on the same event bus the rest of a run uses, so the desktop client's file panel updates live instead of polling every 3s.

• feat(server): POST /memos/resume — pick up at last on-disk checkpoint — server endpoint over cli/resume_from_interruption.py's in-process functions; the desktop client can resume a dead run without subprocess-spawning the CLI.

• feat(server): GET /firms/{firm}/brand-config returns the YAML as JSON — round-trip read endpoint that pairs with the existing /actions/save-brand.

• fix(events): stamp every published event with a monotonic seq — kills double-counting on EventSource reconnects; client tracks lastSeenSeq.

• fix(file_watcher): resolve output_dir so live events stop being dropped — relative-vs-absolute path mismatch was silently swallowing every file event.

• fix(jobs): output_dir regex handles deal names with spaces — \S+ was truncating "Mercury Bank" to "Mercury".

• fix(memos): slugify deal names at the create / resume boundary — server-side defense-in-depth for the same class of issue.

• chore(submodule): register io/alpha-partners as a submodule — promotes the previously-flat io/alpha-partners/ directory to a submodule pointing at the private alpha-partners-vault repo.

The parity sync itself is plumbing — both repos point at the same remote and now have identical histories. The team-roster commit was authored on the memopop-orchestrator side and rebased into the canonical repo as part of the sync.

What's Next

Three follow-ups, in priority order:

1. Phase 3 — Photo stability. The Photo.is_externally_stable field exists; the validator that flips it from False to True doesn't. A small worker that fetches each Photo.url with no Referer/no cookies, checks for 200 + image content-type, and updates the boolean. Pure Phase 3 work.

2. Phase 4 — Fallback agents for sparse rosters. When a team page names ten people but only links LinkedIn for three, today's enrichment runs a name+org search per missing platform. That works for ~70% of cases. The remaining 30% — solo founders, partners with same-name collisions, members listed without titles — need a smarter fallback: per-member search prompts that include disambiguating context (e.g., "Sarah Lee VC partner Series A health tech" vs "Sarah Lee software engineer San Francisco").

3. Phase 5 — Wire the roster into the writer. Phases 2–4 produce a structured roster artifact. Phase 5 is the writer agent change that consumes it: instead of researching team data inline, the writer reads the roster JSON, treats each member as a typed object, and writes the Team section from a known input. This is where the work pays off in memo quality — but it requires the upstream roster to be reliable enough to bet a section on. Phases 3 and 4 land first.

The orchestrator's HTTP surface and its content-extraction surface are growing in parallel right now. Six weeks ago the entire system was a CLI that called LLMs. Today it's a long-running service with a desktop client, an event bus, a file watcher, a brand-fetch loop, and a team-roster pipeline. The agents themselves are mostly unchanged. The infrastructure around them is doing the heavy lifting.