feat(server): persisted logs, never-overwrite versioning, hardened sidecar lifecycle

Two changes that turn the FastAPI sidecar from "it runs" into "you can
watch it run, and the output lands where you said it should."

* MilestoneExtractor (src/server/milestones.py): regex maps for ~30
  emoji-tagged agent prints, grouped into 10 pipeline stages. LogSink
  feeds every line through it; matches are published as 'milestone'
  events on the JobEventBus alongside the raw 'log' events. Static
  prints are deduped so summary blocks don't re-fire.

* scaffold_firm_deal_dir (src/server/app.py): POST /memos now writes
  io/{firm}/deals/{deal}/inputs/deal.json from the request fields
  before queuing the job. Without it, paths.resolve_deal_context()
  silently fell back to legacy output/ — files landed outside the
  firm, even when --firm was passed. Idempotent: existing configs
  (e.g., curated via CLI) are preserved.

Five concrete fixes after first contact with the desktop client. Every one
of them was a foot-gun the moment a non-Mike user (or even Mike) tried to
drive a real run.

* Logs persist on every run end (success OR failure). JSONL + plain text,
  written into the run's output dir AND a global `.logs/runs/` mirror so
  "send me a log file" is one upload. JobEventBus gains an optional `tap`
  callback; Job carries an uncapped events_log captured via that tap so
  the bus's MAX_BACKLOG (for SSE replay) doesn't lose anything from disk.

* VersionManager.get_next_version now considers BOTH versions.json AND any
  `{Company}-vX.Y.Z` directories on disk, returning max+1. Will not
  silently overwrite a version directory — full stop. Catches the case
  where artifacts were moved manually or a prior run crashed before
  recording. Three regression tests pin it down.

* SidecarManager.ensure_running self-heals. Always probes /healthz first,
  regardless of what state thinks; on failure, drops the stale handle and
  spawns fresh. No more "edit code, kill PID, hope for the best" loop.

* POST /actions/stop-sidecar — Rust dispatcher action that calls the
  manager's shutdown directly. Bypasses the Python sidecar entirely so it
  works when the Python is wedged (rate-limited, deadlocked, mid-retry).
  SIGKILL via stored CommandChild — a promise, not a request.

* Tauri shell handles WindowEvent::CloseRequested alongside Destroyed,
  so quitting MemoPop reliably terminates the Python child instead of
  leaving stale code holding the port.

Frontend (memopop-ai): JobView replaces the modal with a fullscreen 3-pane
(checklist · log stream · live file browser); event log capped at 2k in
store and 1k rendered with rAF-throttled scroll to fix memory bloat and
webview repaint stalls; STOP button wired to the new action.

18 tests pass in ~0.16s.

Changelog - 2026-05-01_03

Hardened: Logs Land, Versions Bump, the Sidecar Heals Itself

First-contact session with the desktop client surfaced exactly the failure modes you'd expect when the same person who built the orchestrator hands their laptop to itself, pressed for time, in the middle of a working night. A log viewer that ate memory until the screen went black. A versioning function that silently overwrote a directory because the registry was empty. A Python sidecar that didn't die when the app quit, holding stale code through three iteration cycles. A Stop button that didn't exist. None of these were new features — they're the things-that-should-have-been-ironclad about a service surface we shipped the day before.

This pass closes all of it. The fixes are small, individually unremarkable; the discipline is in finishing them off in one push so the service is actually usable end-to-end.

Why

Yesterday's 2026-04-30_01.md shipped a FastAPI sidecar wrapping generate_memo(). Today's 2026-05-01_02.md added structured milestone events plus the deal-directory scaffold so firm-scoped runs land where the user expects. Then we drove a real run from MemoPop and discovered four distinct ways the integration could (and did) bite:

1. The log viewer ate memory and the webview started skipping frames. A 17-minute run produced thousands of stdout lines, each one a reactive $state push and a DOM <div> with pre-wrap layout. By minute 20 the screen was going black on a 30-second cadence — Svelte's deep-reactivity proxies plus full {#each} re-realization were measurable in stop-the-world GC sweeps.

2. A second run on the same company silently overwrote the first run's directory. VersionManager.get_next_version only consulted versions.json. If an entry was missing — because the previous run crashed, was stopped, was moved manually, or used legacy paths — it returned v0.0.1 and create_artifact_directory cheerfully wrote on top of the existing tree.

3. The Python sidecar outlived MemoPop. Closing the Tauri window fired WindowEvent::Destroyed only sometimes (depending on platform and how the app was quit). The child Python process stayed alive on a closed port, preventing fresh sidecars from binding 8765 and — worse — keeping stale code in memory through subsequent debug cycles. Every code change required a manual kill {pid} from a separate terminal.

4. There was no Stop button. Once you POSTed /memos, your only recourse to abort a runaway run was the same manual kill. For a 30-minute LLM run that costs $5–20 of API credit, that's a real ergonomics gap.

Two further realizations followed from #3:

• Even with a window-close handler, if the sidecar process were already running stale code (because it predated the most recent edit), the dispatcher would happily route requests into the wrong code path. The dispatcher needed to self-heal — to detect when its sidecar handle was dead and spawn a fresh one without ceremony.

• Logs needed to be persisted to disk. Until that happened, the only artifact of a failed run was whatever ended up in output/ plus the live SSE stream. If a user reported "the run did something weird," there was no file to send back. "Send me the log" had to be a one-line instruction.

What Landed

src/server/log_persistence.py — full event stream, two formats, two locations

A new persist_job_logs(job_id, company_name, output_dir, version, events) writes the run's full event stream in two formats and to two places:

The double-underscore in the global mirror's filename is a deliberate sentinel: ls .logs/runs/ | grep '__Stripe' filters across all jobs for a single deal. Filenames are sanitized aggressively — a company name like Acme/Co: ../etc/passwd lands as dangerz__Acme_Co___etc_passwd.txt rather than escaping the directory.

The plain-text format mirrors what the JobView's LogStream pane shows:

[18:42:11] 📁 Created new output directory: io/alpha-partners/deals/ChromaDB/outputs/ChromaDB-v0.0.2
[18:42:11] ◆ [start] Run started — Output directory: io/alpha-partners/deals/ChromaDB/outputs/ChromaDB-v0.0.2
[18:42:13] 🔍 PERPLEXITY SECTION RESEARCH (PARALLEL)
[18:42:13] ◆ [research] Researching sections
[18:43:02]   ✓ [01] Executive Summary: 12 citations
...
[19:02:18] ✗ error: Anthropic rate limit exceeded
    Traceback (most recent call last):
      File "src/agents/researcher.py", line 142, in research
      ...

Errors include the full traceback.format_exc(). JSONL preserves it in a traceback field; the text version indents the traceback under the error line for visual nesting.

.logs/ was added to .gitignore so the mirror never accidentally commits.

src/server/events.py — bus gains a tap for off-bus capture

JobEventBus.__init__ accepts an optional tap: callable(dict) -> None invoked synchronously on every publish from the publishing thread. Used to accumulate the full event history off-bus (the in-memory backlog stays capped at MAX_BACKLOG = 2000 for SSE-replay performance; the tap captures the unbounded stream for disk persistence).

src/server/jobs.py — Job carries events_log, persistence runs on every terminal path

@dataclass
class Job:
    ...
    events_log: list[dict] = field(default_factory=list)

# In submit():
events_log: list[dict] = []
bus = JobEventBus(self._loop, tap=events_log.append)
job = Job(..., bus=bus, events_log=events_log)

_run's finally block now calls persist_job_logs(...) regardless of how the run ended — success, exception, even if output_dir was never populated (early failure scenario). For early failures, a new _scan_logs_for_output_dir(events_log) helper recovers the path from the orchestrator's 📁 Created new output directory: … print, so the failure log lands alongside whatever artifacts did get written.

src/versioning.py — disk-aware next-version

get_next_version no longer trusts only versions.json:

def get_next_version(self, company_name: str) -> MemoVersion:
    candidates: List[MemoVersion] = []

    # 1. Latest from versions.json (when recorded).
    if company_name in self.versions_data:
        latest = self.versions_data[company_name]["latest_version"]
        candidates.append(MemoVersion.from_string(latest))

    # 2. Highest version directory present on disk.
    candidates.extend(self._scan_disk_versions(company_name))

    if not candidates:
        return MemoVersion(0, 0, 1)

    highest = max(candidates, key=lambda v: (v.major, v.minor, v.patch))
    return highest.increment_patch()

The new _scan_disk_versions walks every io/{firm}/deals/*/outputs/ (firm mode) or the legacy output/ (otherwise) for directories matching {Company}-vX.Y.Z. The candidate set is the union of the registry and the disk; we bump above the maximum either way. Three tests pin this down: existing dir on disk with empty registry → v0.0.2; registry says v0.0.2 but disk has v0.0.3 → v0.0.4; clean slate → v0.0.1.

MemoVersion doesn't implement __lt__, so the max() call uses an explicit key=lambda v: (v.major, v.minor, v.patch). Worth flagging in case anyone wants to add ordering semantics to MemoVersion later.

src-tauri/src/api/sidecar.rs — ensure_running self-heals

pub async fn ensure_running<R: tauri::Runtime>(
    &self,
    app: &tauri::AppHandle<R>,
    repo_path: &str,
) -> Result<(), ApiError> {
    // ALWAYS probe /healthz first, regardless of what state thinks.
    if self.healthz_ok().await {
        return Ok(());
    }

    // Sidecar isn't responding. Drop any stale handle (best-effort kill).
    if let Ok(mut state) = self.state.lock() {
        if let Some(child) = state.child.take() {
            let _ = child.kill();
        }
        state.repo_path = None;
    }

    // ... spawn fresh process and wait for healthz ...
}

Previous behavior short-circuited on state.child.is_some() — meaning a child handle that pointed at a now-dead process would prevent re-spawning, and the dispatcher would forward into a closed port. This was the bug that wasted hours on the integration debug cycle: edit Python code, kill the sidecar manually, restart MemoPop, hope the Tauri shell would actually spawn the new code. With self-heal, none of those steps are necessary anymore — POST /memos is the only ritual.

Cost: a 250ms /healthz round-trip on every /memos call. For a workflow that takes 15–45 minutes, this is invisible.

src-tauri/src/api/router.rs — POST /actions/stop-sidecar

("POST", "/actions/stop-sidecar") => {
    let manager = sidecar::manager(&app);
    manager.shutdown();
    Ok(serde_json::json!({"stopped": true}))
}

Routes to SidecarManager::shutdown() directly — does NOT round-trip through the FastAPI server. The whole point of "stop" is that Python may be wedged; child.kill() on the stored CommandChild sends SIGKILL via tauri-plugin-shell. Promise, not request.

src-tauri/src/lib.rs — CloseRequested joins Destroyed

.on_window_event(|window, event| {
    match event {
        tauri::WindowEvent::CloseRequested { .. }
        | tauri::WindowEvent::Destroyed => {
            if let Some(manager) = window.app_handle().try_state::<api::sidecar::SidecarManager>() {
                manager.shutdown();
            }
        }
        _ => {}
    }
})

Cmd+W, Cmd+Q, the red dot — any path that closes the window now reliably kills the sidecar. Self-healing covers the kill -9 and crash cases; this covers the normal-quit case.

Frontend (memopop-ai) — fullscreen JobView, perf-bounded log pane, Stop button

The big visible change. The previous tabbed JobLogViewer modal was deleted and replaced with JobView — fullscreen, three columns:

Performance fixes for the log pane (issues observed in real usage):

• flow.svelte.ts: events array capped at 2,000 in the store via rolling-window slice(-2000). Milestones uncapped (small list).

• LogStream.svelte: rendered lines capped at 1,000 with a "⋯ N earlier lines hidden — full log on disk in server-log.txt" banner at the top.

• Auto-scroll coalesced via requestAnimationFrame — multiple events arriving in one frame now trigger one scrollHeight read, not N.

Stop button (red, only visible while running) calls POST /actions/stop-sidecar and locally transitions the flow to a cancelled state. Even if the action errors (sidecar already dead), flow.markCancelled('Stopped by user') runs unconditionally — the user's intent is honored regardless.

Layout: +layout.svelte switched from min-height: 100vh to height: 100vh with min-height: 0 on .content, so the JobView's internal flex chain sizes correctly.

Design Decisions

Persisted logs use two formats and two locations on purpose. JSONL is for grep / jq / programmatic analysis: jq 'select(.type=="error")' server-log.jsonl gives every failure point in one query. Plain text is for human eyeballs and for the support flow ("paste the last 50 lines"). The run-dir copy keeps the log alongside the artifacts it describes; the .logs/runs/ mirror is the file you ask a user to upload when something went wrong.

The bus's tap is sync, not async. JobEventBus.publish is called from any thread (the worker thread that runs generate_memo(), mainly). The tap fires inline with no await and no scheduling — the caller pays for it directly. This is correct for events_log.append: append is O(1), there's no I/O. If we ever want to do something heavier per event, we'd add a separate async hook.

The disk scan walks every deal under the firm. Could've been a single-deal scan if we passed the deal name into get_next_version, but the call site only has the sanitized company name. Walking deals/*/outputs/ is O(deals × versions), bounded by tens of items in practice. Trivial. The benefit: covers the case where deal_name and sanitized_company_name diverge (e.g., "Cool Co" → Cool_Co-v0.0.1 inside deals/Cool Co/outputs/).

Stop bypasses Python. Earliest design considered a POST /shutdown endpoint inside FastAPI, but that requires Python to be responsive — exactly the failure mode Stop must handle. Routing the action through the Rust dispatcher to the manager's child.kill() is the only design that works when the orchestrator is hung in a retry loop or rate-limited by Anthropic.

Self-heal pays a 250ms health check on every /memos call. Acceptable: the workflow itself is 15–45 minutes. The check is the difference between "edits-on-disk just work" and "manual kill {pid} between every iteration." We chose ergonomics.

Stop is SIGKILL, not SIGTERM-then-SIGKILL. Graceful shutdown would let the orchestrator's finally blocks run — including persist_job_logs. SIGKILL skips that, so cancelled runs don't get the structured log file. Trade-off: SIGKILL is reliable when SIGTERM-handlers themselves are wedged. We can add a --graceful-stop mode later that SIGTERMs first and SIGKILLs after a 5-second grace; for v1, the bias is "Stop means stop."

JobView replaced the modal entirely. The previous tabbed modal was 760px wide, centered on a 1920px screen. The user's words: "this will be THE THING THEY ARE TRYING TO DO, WITH IMPORTANCE so they will likely not have it as a small part of their screen." Three columns at full viewport width gives logs the dominance they deserve, milestones the visibility they need, and the file panel the satisfaction-of-watching-files-flash-into-existence factor that's surprisingly motivating.

Verification

$ .venv/bin/python -m pytest tests/ -v
======================== 18 passed in 0.16s ========================

Eighteen tests, six new since the previous changelog:

Manual end-to-end after the fixes landed: a new POST /memos for ChromaDB on alpha-partners, with io/alpha-partners/deals/ChromaDB/outputs/ChromaDB-v0.0.1/ already present from a prior run. Output landed at …/ChromaDB-v0.0.2/. Checklist updated live. Stop button killed the run cleanly. No more orphan sidecars between iteration cycles.

Files Changed

Orchestrator (investment-memo-orchestrator/)

Tauri shell (memopop-ai/apps/memopop-native/src-tauri/)

Frontend (memopop-ai/apps/memopop-native/src/)

What's Next

Three near-term items, none of them urgent:

1. Graceful Stop variant. A Stop with logs mode that SIGTERMs first (giving Python a 5-second window to run persist_job_logs) and SIGKILLs as a fallback. Today's Stop is bulletproof but loses the structured log on cancel. Cheap to add when it bites.

2. Sidecar version marker. Right now the dispatcher self-heals on dead sidecars but won't replace a stale live one (one running old code). A version hash exposed at /healthz would let the dispatcher detect stale code and replace it. Not urgent — the more common path is sidecar dying after an edit (covered by self-heal) — but worth doing before more than one user is in the loop.

3. Configurable sidecar port. Hardcoded 8765. If a user has their own service on that port, today's failure is opaque. Probing a small port range (8765–8775) and remembering which one stuck would handle this gracefully.

The desktop journey from "click an outline" to "see a memo land in the right firm at the right version with a real Stop button and a saved log file" is now closed. Time to test it on someone who isn't us.