Lossless Changelog
fleet

Roll-up, deliberate: pnpm rollup:sync replaces sync-on-build

The roll-up loader we shipped four hours ago hit the GitHub Content API on every build — 30 seconds per `pnpm dev`, auth plumbed through CI, every iteration eating the rate limit. Replaced it with a deliberate-sync model: a human runs `pnpm rollup:sync` when content drifts, the result lands in `splash/src/rollup/` as committable markdown, and from then on builds are pure file IO. Build time fell from 30s to 1.1s. CI auth needed: zero.

Michael StatonMusic Editoraugmented with Claude Code on Claude Opus 4.7 (1M context)

Why Care?

If you're building a splash page that aggregates content from child repos, the obvious instinct is to fetch on every build — convenient, always fresh, treats the children as live data. We tried that this afternoon. It worked. It was also wrong.

The build was 30 seconds. CI needed GITHUB_TOKEN plumbed through. Local pnpm dev paid the API cost on every restart. Iterating on layout meant either rate-limiting yourself or living with the latency. None of those problems are individually fatal; collectively they make the splash worse to maintain.

This entry documents the pivot — same outcome (rolled-up changelog and context-v feeds with provenance), entirely different shape (deliberate sync instead of build-time fetch). The result is faster, simpler, and matches how the rest of this tree already handles cross-repo content (submodule pointers are explicit snapshots; rolled-up content should be too).

What's New?

  • pnpm rollup:sync — a CLI script that fetches each plugin submodule's changelog/ and context-v/ via the GitHub Content API and writes the results to splash/src/rollup/ as plain markdown files. Run it deliberately; commit the result.

  • splash/src/rollup/ — the synced content directory. 52 files at this push: 22 changelog entries + 30 context-v entries spread across 8 plugin slugs. Committed to the repo, not gitignored. git diff src/rollup/ shows exactly what each sync changed.

  • A small unionLoader in content.config.ts that reads two directories — the parent's own (../changelog, ../context-v, live) plus src/rollup/ (synced) — and merges them into one Astro content collection with provenance preserved.

  • The Astro build no longer touches the network. No GITHUB_TOKEN env var passed in the workflow, no submodules: recursive checkout. CI installs deps, runs pnpm build, deploys. That's it.

  • Build time: 30s → ~1s. Verified with the time shell builtin: pnpm build now finishes in 1.05–1.15s consistently. Local pnpm dev is similarly instant on restart.

  • A blueprint at astro-knots/context-v/blueprints/Generate-a-Pseudomonorepo-Rollup-of-Changelog-and-Context.md codifies the pattern for future Lossless splash sites (memopop-site is the obvious next adopter).

The architecture, before and after

Before (sync-on-build)

pnpm build  →  Astro content sync
            →  rollupLoader fires
            →  ~70 GitHub Content API calls
            →  parse frontmatter, store entries
            →  render pages
            →  done in 30s

Required: GITHUB_TOKEN exposed to the build env. Authenticated rate limit (5000 req/hr) was the ceiling — anonymous (60 req/hr) ran out almost immediately during dev.

After (sync-on-demand)

pnpm rollup:sync   (human, occasional)
   ↓
fetchRolledUp()  →  ~70 API calls  →  write 52 .md files to src/rollup/
   ↓
git add src/rollup/  →  commit  →  push
   ↓
pnpm build   (CI or local, fast)
   →  Astro content sync
   →  unionLoader fires
   →  reads ../changelog, ../context-v (parent's own — live)
   →  reads src/rollup/changelog, src/rollup/context-v (synced — committed)
   →  merges, renders pages
   →  done in 1s

Required at build time: nothing beyond Astro itself. The API calls happen at sync time, not build time, and at human cadence.

Component changes

splash/scripts/rollup-sync.ts (new)

The CLI. Highlights:

TS
async function main(): Promise<void> {
  await loadDotEnv();                            // read splash/.env if present

  await rm(ROLLUP_ROOT, { recursive: true, force: true });   // wipe so removals propagate
  await mkdir(CHANGELOG_OUT, { recursive: true });
  await mkdir(CONTEXT_V_OUT, { recursive: true });

  const changelog = await fetchRolledUp({
    collectionName: 'changelog',
    remotePath: 'changelog',
    remoteFallbackPaths: ['context-v/changelogs'],   // legacy honor
    gitmodulesPath: GITMODULES,
  });
  for (const entry of changelog.entries) {
    const out = resolve(CHANGELOG_OUT, entry.from, entry.fromPath);
    await mkdir(dirname(out), { recursive: true });
    await writeFile(out, materialize(entry), 'utf8');
  }
  // ...same for context-v, with a filter that excludes 'changelogs/' so legacy
  // entries don't double-appear...
}

Each written file gets a provenance-injected frontmatter and a one-line marker comment so anyone opening one knows it's synced and shouldn't be hand-edited:

MARKDOWN
---
title: "Hex citations, day one"
date_created: 2025-06-07
[…original frontmatter…]
from: cite-wide
from_path: 2025-06-07_01.md
---
<!-- Rolled up from cite-wide/changelog/2025-06-07_01.md. Edit at the source, not here. Re-run `pnpm rollup:sync` to refresh. -->

## Why Care?
[…body…]

The script runs via Node 22's --experimental-strip-types flag — no transpiler dep, no tsx, no tsm:

JSON
"rollup:sync": "node --experimental-strip-types --no-warnings scripts/rollup-sync.ts"

splash/src/loaders/rollupFetch.ts (new — refactor of the old loader)

The fetcher is now a pure function fetchRolledUp(options) → FetchResult that's not tied to Astro's loader API. The CLI calls it directly. Same parsing, same legacy-fallback handling, same provenance injection. Just decoupled from the build pipeline.

TS
export async function fetchRolledUp(options: RollupFetchOptions): Promise<FetchResult>

Per-submodule errors don't sink the whole fetch — they get reported in result.perSubmodule[i].error and the rest of the run continues.

splash/src/loaders/rollupLoader.ts (deleted)

The Astro loader-object form is gone. Astro doesn't know rollup-sync exists; it just sees plain markdown files in src/rollup/.

splash/src/content.config.ts — new unionLoader

TS
function unionLoader(options: UnionLoaderOptions) {
  return {
    name: `union-loader:${options.collectionName}`,
    load: async ({ store, parseData, logger }): Promise<void> => {
      store.clear();

      // 1. Local parent content (e.g. ../changelog) — read directly, no sync needed
      for await (const file of fsGlob('**/*.md', { cwd: options.localDir })) {
        // …read, parse frontmatter, inject from='content-farm', store…
      }

      // 2. Synced rollup (src/rollup/changelog) — committed files
      for await (const file of fsGlob('**/*.md', { cwd: options.rollupDir })) {
        // …read, parse frontmatter, store (provenance already in frontmatter)…
      }
    },
  };
}

The local-parent half is what makes editing content-farm/changelog/<new-entry>.md show up in pnpm dev immediately — the splash's own ship notes don't need a sync step. Only submodule content syncs.

.github/workflows/pages.yml — three deletions

DIFF
       - name: Checkout
         uses: actions/checkout@v4
         with:
-          # Fetch submodules so context-v/ aggregations from plugin-modules
-          # are available at build time…
-          submodules: recursive
           fetch-depth: 1

       - name: Build
         run: pnpm build
-        env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Cleaner workflow, less to maintain, fewer failure modes.

Schema and parsing remain identical

Everything that was already true about the previous version stays true:

  • All frontmatter fields are optional via lenient z.preprocess coercion.

  • Schemas use .passthrough() to preserve unknown fields.

  • The loader catches per-entry parse errors and stores raw frontmatter rather than failing the build over a stylistic quirk.

  • Frontmatter parsing uses our in-tree ~150-line YAML subset parser at src/loaders/frontmatter.ts — no gray-matter, no js-yaml.

  • Astro is on @latest (currently 6.2.2). Tailwind isn't installed (the splash uses hand-authored CSS via two-tier tokens in BaseLayout.astro).

The whole point of changing the trigger mechanism was to keep the content handling untouched. Same data, same UI, different cadence.

What this enables

For the human maintaining the splash:

  • pnpm dev restart costs ~1s, not 30s.

  • Editing a parent-level changelog entry (the one you're reading) shows up instantly in dev — the parent dir is read live, no sync step needed.

  • Reviewing a sync PR shows actual content diffs. You can see at a glance what was added, removed, or modified across the rolled-up feed.

For CI:

  • Workflow is leaner. No submodules: recursive (saves checkout time and bandwidth). No GITHUB_TOKEN env passthrough (one fewer thing that can mis-fire).

  • Build determinism. The deployed splash reflects exactly what's committed at HEAD — not whatever GitHub's API happened to return at build time.

  • No rate-limit anxiety. Even if some other workflow on the org is hammering the API, this build is unaffected.

For the next Lossless splash adopter:

  • The pattern is documented in astro-knots/context-v/blueprints/Generate-a-Pseudomonorepo-Rollup-of-Changelog-and-Context.md with a complete file checklist.

  • memopop-site is the obvious next candidate. Same shape: pseudomonorepo (memopop-ai), child repos (memopop-orchestrator, memopop-native), splash app (memopop-site). Adopt-time is small now that the loader and CLI exist.

Trade-offs we accepted

This isn't a free win. Things we gave up:

  1. Always-fresh content. The splash now reflects whatever was synced last. Mitigation: pnpm rollup:sync is a single command; weekly cadence is reasonable; a future scheduled CI workflow can sync automatically and open a PR.

  2. A larger repo. 52 markdown files now live in src/rollup/. About 60KB total — negligible vs. anything else in the tree.

  3. Sync drift. A developer pushes a noteworthy plugin entry and forgets to sync the parent. The splash silently lags. Mitigation: the cron-the-sync follow-up below.

These are all bounded concerns. The previous architecture had unbounded concerns (build time, auth complexity, rate limits) that compounded across every build.

Honest middle (the conversation that produced this)

The pivot came from one question: "I ran pnpm build, it took 30 seconds. I ran pnpm dev, same. Should this just pull in content on a build flag? Or is this a non-issue I am just anxious?"

The non-issue framing was wrong. The 30-second build was the visible symptom; the deeper problems (CI auth, rate limits, no diff visibility, build non-determinism) were less obvious but more important. A scheduled human-cadence sync addresses all of them at once.

Worth saying explicitly: the user's instinct to ask the question saved us from shipping the wrong design. This is the kind of thing that's hard to spot reviewing a PR — the original sync-on-build worked, generated the right pages, looked correct. The drawbacks only became apparent when you tried to use it for real work.

What's next

  • Cron-the-sync. A scheduled GitHub Actions workflow that runs pnpm rollup:sync weekly, commits if anything changed, opens a PR. Removes the human cadence dependency without giving up diff visibility. Deferred until the manual cadence becomes painful — which may never happen for this splash.

  • Caching in the sync. Currently pnpm rollup:sync always re-fetches everything (~70 API calls). A SHA-keyed cache at splash/.cache/rollup/ would short-circuit unchanged submodules. Would let the script be cron-able more aggressively without burning the rate limit.

  • Roll-up convention reference update. pseudomonorepos/references/content-rollup.md (the skill ref written earlier today) is framed around sync-on-build. It should be revised to point at the new blueprint — astro-knots/context-v/blueprints/Generate-a-Pseudomonorepo-Rollup-of-Changelog-and-Context.md — as the current authoritative pattern.

  • memopop-site adoption. The reference implementation is content-farm/splash; the next site to adopt should be memopop-site since it's the other Lossless splash in the family. About a half-day's work given the components are now extracted.