feat(server): milestone extraction + firm-scoped deal scaffold

Two changes that turn the FastAPI sidecar from "it runs" into "you can
watch it run, and the output lands where you said it should."

* MilestoneExtractor (src/server/milestones.py): regex maps for ~30
  emoji-tagged agent prints, grouped into 10 pipeline stages. LogSink
  feeds every line through it; matches are published as 'milestone'
  events on the JobEventBus alongside the raw 'log' events. Static
  prints are deduped so summary blocks don't re-fire.

* scaffold_firm_deal_dir (src/server/app.py): POST /memos now writes
  io/{firm}/deals/{deal}/inputs/deal.json from the request fields
  before queuing the job. Without it, paths.resolve_deal_context()
  silently fell back to legacy output/ — files landed outside the
  firm, even when --firm was passed. Idempotent: existing configs
  (e.g., curated via CLI) are preserved.

Test coverage: pytest now runs in tests/ (configured in pyproject),
12 tests covering Pydantic validation, log-sink line splitting, event
bus replay, milestone recognition + dedup, scaffold creation +
preservation, and path-traversal rejection. ~0.16s end to end.

No changes to agents, workflow, or generate_memo(). The CLI path is
unaffected.

Changelog - 2026-05-01_02

Milestones Out of the Logs, Files Into the Firm

Yesterday the orchestrator got an HTTP front door. Today it learns to narrate. The FastAPI sidecar now emits structured milestone events alongside its raw log stream — clients no longer have to grep for ✅ SECTION RESEARCH COMPLETE to know research is done; they just listen for {"type": "milestone", "stage": "research", "level": "success", ...}. And a missing scaffold step that was sending firm-scoped runs to legacy output/ is fixed: a firm-scoped POST /memos now writes the deal config the orchestrator's path resolver requires, so artifacts land where the user said they should.

Zero changes to the workflow or any agent. generate_memo() is identical. python -m src.main still works exactly as it did. The CLI doesn't even know any of this happened.

Why

Two distinct problems surfaced when the desktop client (memopop-native) drove its first end-to-end memo through the sidecar.

The first was the user not being able to tell what was happening. A live tail of stdout is sexy — and it stays sexy — but it's not what most operators want as their primary signal. Watching the agents write a 67k-character draft over 17 minutes, what they want to know is "where are we, what's left." The orchestrator's agent prints already encode this perfectly (🔍 PERPLEXITY SECTION RESEARCH (PARALLEL), [3/10] Market Context, ✅ All sections passed fact-check) — they just needed to be lifted into structured events with stage and level, so a UI can render them as a checklist instead of asking the user to read the logs.

The second was a silent fallback bug. A run was kicked off with firm="alpha-partners" and company_name="ChromaDB" from the desktop app. Output landed in output/ChromaDB-v0.0.1/ rather than io/alpha-partners/deals/ChromaDB/outputs/.... The cause: paths.resolve_deal_context() only honors a passed firm if ctx.exists() is true — meaning io/{firm}/deals/{deal}/inputs/deal.json already exists. The CLI flow assumes you pre-create that file (or data/{Company}.json in legacy mode); if it's missing, the resolver silently falls through to legacy output. The CLI tells you in a yellow warning. The API just… did the run, in the wrong place.

The CLI is one user (Mike) who knows the convention. The API surface has to be friendlier than that.

What Landed

src/server/milestones.py — turning prints into events

A MilestoneExtractor with a list of ~30 compiled regex patterns covering the orchestrator's emoji-tagged status prints across ten pipeline stages:

Each match returns a JSON-shaped milestone:

{
  "type": "milestone",
  "id": "a3f29c8e1b",
  "stage": "research",
  "level": "info",
  "label": "Researched: Market Context",
  "detail": "18 citations",
  "ts": "2026-05-01T18:42:11.293Z"
}

label and detail are formatted from regex capture groups ({0}, {1}, ...), so ✓ [03] Market Context: 18 citations becomes label="Researched: Market Context", detail="18 citations". Static milestones (no captures) are deduped — the orchestrator sometimes prints ✅ SECTION RESEARCH COMPLETE once at the end of a stage and again in a summary block; the bus only fires it once.

src/server/events.py — LogSink now feeds the extractor

LogSink accepts an optional milestone_extractor. When it splits a log line on \n and publishes it as a log event, it now also calls extractor.process(line); if a milestone comes back, it's published as a separate milestone event on the same JobEventBus. SSE subscribers see both event types interleaved in chronological order. The bus's 2000-event backlog covers both.

sink_out = LogSink(bus, mirror=sys.__stdout__, milestone_extractor=extractor)
sink_err = LogSink(bus, mirror=sys.__stderr__, milestone_extractor=extractor)

One extractor per job, shared between stdout and stderr — its dedup state is per-run.

src/server/app.py::scaffold_firm_deal_dir — the missing scaffold

POST /memos now runs a small idempotent scaffold step before submitting the job:

@app.post("/memos", ...)
async def create_memo(request: CreateMemoRequest) -> CreateMemoResponse:
    scaffold_firm_deal_dir(request)
    job = _registry(app).submit(request)
    return CreateMemoResponse(...)

If the request includes a firm, it creates io/{firm}/deals/{company_name}/inputs/ and writes a deal.json populated only from fields the request actually carried — type, mode, url, description, stage, notes, deck, dataroom, outline, scorecard, trademark_light, trademark_dark. If inputs/deal.json already exists (the user curated it via the CLI, hand-edited, etc.), it's left untouched; only the directory tree is created lazily.

The result: when generate_memo() reaches create_artifact_directory(firm=firm) and that calls paths.resolve_deal_context(), the resolver finds the config and routes the run through the firm-scoped path. Output lands at io/{firm}/deals/{company}/outputs/{company}-{version}/ — exactly where the firm-pill in the UI implies.

When firm isn't provided, the scaffold is a no-op and the run still falls through to legacy output/. CLI users who rely on that behavior aren't disturbed.

tests/ — pytest is real now

pyproject.toml gained a [tool.pytest.ini_options] block (testpaths = ["tests"], asyncio_mode = "auto"). The test suite now covers:

12 tests, ~0.16s end to end. None of them spend a cent on Anthropic — generate_memo() is never called. They use FastAPI's TestClient, pytest.tmp_path, monkeypatch.setenv("MEMO_IO_ROOT", ...), and direct unit calls into the helpers.

A small refactor was needed for the path-traversal test: the target.relative_to(out) logic was inline in the get_artifact endpoint. It's now a module-level resolve_artifact_path(out, sub) that the endpoint calls and the test exercises directly. Cleaner factoring; identical behavior.

Design Decisions

Pattern matching, not orchestrator instrumentation. The agents could have been changed to emit explicit "stage start / stage complete" events. They weren't. Instead, the sidecar pattern-matches the prints the agents already produce. Reasons: (a) zero touch to the agent code keeps the CLI behavior identical, (b) the prints already exist and are well-shaped, (c) when an agent's prose changes, we update one regex in milestones.py rather than chasing through 30 agent files. The cost is fragility — a print rewrite breaks a milestone — but the test suite can grow with the catalog and the failure mode is graceful (you fall back to the raw log, which still works).

Milestone events on the same bus as logs, not a separate channel. A separate /memos/{id}/milestones endpoint was tempting, but the SSE bus already carries the chronological order both event types need. Clients render them where they want; both arrive on the same stream with the same ts.

The scaffold writes only what the request carries. deal.json is the orchestrator's contract — it accepts a wide set of optional fields. The scaffold doesn't try to invent defaults; it copies through only what the request explicitly provided. If the user later wants to enrich the file (add stage, edit notes), they can — the scaffold won't fight them.

Idempotency over overwrite. A user who runs the CLI to set up a deal, then later runs the API for the same firm+company, should not lose their hand-curated config. The scaffold checks for an existing inputs/deal.json and bails out untouched. If the API request had richer fields than the existing config, those richer fields are silently ignored — but the run still routes to the right place because ctx.exists() returns True, which is the only thing resolve_deal_context() cares about.

Pytest config in pyproject.toml, not a separate pytest.ini. Matches the modern Python convention and keeps the repo's tool config in one file. asyncio_mode = "auto" so @pytest.mark.asyncio decorators aren't needed on every async test.

Verification

$ .venv/bin/python -m pytest tests/ -v
======================== 12 passed in 0.16s ========================

All twelve green. The tests are fast enough to run on every save in development if you wire it up.

Manual smoke against a live server (after the breaking change, before this commit, to confirm the symptom):

$ curl -X POST http://127.0.0.1:8765/memos \
    -H 'Content-Type: application/json' \
    -d '{"company_name":"ChromaDB","firm":"alpha-partners","memo_mode":"consider"}'
{"job_id":"...","status":"queued"}

$ ls io/alpha-partners/deals/
(empty before this change — files landed in output/ChromaDB-v0.0.1/)

$ ls io/alpha-partners/deals/ChromaDB/inputs/   # after this change
deal.json

The 17-minute run that triggered all this still produced a complete 67k-character memo at output/ChromaDB-v0.0.1/. The scaffold fix changes where future runs go; previous artifacts are untouched.

Files Changed

What's Next

The desktop client (memopop-native) has been built out to consume these events alongside this work — a fullscreen JobView with a phase checklist (left), the live log stream (center, dominant), and a polling artifact browser (right). That's documented in memopop-ai's own changelog; the orchestrator doesn't need to know about it.

A few orchestrator-side follow-ups worth keeping on the radar:

1. More milestone coverage. The current 30 patterns hit the loudest agent prints. Smaller agents (citation_validator, fact_corrector per-section progress, scorecard per-dimension scoring) still flow only through raw logs. Adding patterns is one-line each.

2. Estimated time remaining. A simple model — average duration per stage observed across recent runs — could let the API surface an ETA on GET /memos/{id}. Useful for the desktop client to render "≈ 12 min remaining."

3. Resume from artifact tree. If a run crashed mid-flight (the desktop client closing a window currently kills the sidecar process via WindowEvent::Destroyed), state.json is on disk and python -m src.main --resume works. The API doesn't expose a resume verb yet. POST /memos/{id}/resume would be a short follow-up.

4. Persistent job history. Process memory is the store today; restarting the server loses the in-flight registry. SQLite-backed persistence is a 2-hour follow-up if/when the use case justifies it.

None of these block the current loop from feeling tight.