63 pages, 9 plugins: the roll-up loader ships

Why Care?

Until this push, the content-farm splash rendered a thin slice of itself. /changelog showed three entries. /context-v showed five. The 9 plugin submodules below — sitting on origin/development with their own changelogs and notes — were invisible.

That gap was the entire point of the seeding round earlier today and it's the entire point of this one. The seed put content in each plugin's repo; this push wires the loader that fetches it.

If you're reading this, you can now visit /changelog on the splash and see ship notes from image-gin, cite-wide, perplexed, and the rest, sorted by date across all sources, with provenance on every card so you know where each entry came from. You can click into a cite-wide blueprint and read it without ever leaving the splash. The splash finally renders the farm, not just the parent repo.

What's New?

• A working roll-up loader. Custom Astro content-collection loader that reads content-farm/.gitmodules, derives each submodule's {owner}/{repo} and tracked branch, hits the GitHub Content API at build time, recurses into subdirectories, parses frontmatter, and merges everything into one Astro content collection with provenance metadata on every entry.

• 63 pages on the latest build. Up from 9 (local-only). The breakdown: 1 hero, 1 plugin gallery, 1 changelog index + 26 changelog detail pages (3 from content-farm + 23 from the 9 submodules), 1 context-v index + 36 context-v detail pages.

• Legacy context-v/changelogs/ paths handled. cite-wide and image-gin store changelog-shaped files there from before the convention solidified. The loader hits that path as a fallback per submodule and routes those entries to the changelog feed (not context-v) — preserving them while honoring the current convention.

• Path aliases declared in tsconfig.json (@components, @layouts, @loaders, @content, @pages, @/*) so loader code never reaches for guesswork relative paths. Per the Lossless aspiration that import paths read like meaning, not directory archaeology.

• No new runtime dependencies. The loader uses Node's built-in fetch for the Content API and an in-tree 180-line frontmatter parser scoped to our format — no gray-matter, no js-yaml. Per Astro Knots tech hierarchy: "fewer dependencies is always better."

• Astro bumped to 6.2.2 (was 5.16, against the skill's "always @latest" rule). Tailwind removed entirely from the splash since it was never used.

• All frontmatter fields optional. Schemas use z.preprocess lenient coercion; loader catches per-entry parse errors and stores raw frontmatter rather than failing the build over a stylistic quirk in one file.

• GITHUB_TOKEN plumbed through in the GitHub Pages workflow's build step so the loader hits the 5000 req/hr authenticated rate limit instead of the 60 req/hr anonymous one. With ~100+ API calls per build across all submodules, this is the difference between every-build-deploys and rate-limited-failures.

How the loader works (code-level)

Four files under splash/src/loaders/, each with one responsibility.

1. parseGitmodules.ts — read the parent's submodule registry

export interface SubmoduleEntry {
  name: string;          // [submodule "..."] section name
  path: string;          // local path relative to .gitmodules
  url: string;           // https://github.com/{owner}/{repo}.git
  branch: string;        // defaults to 'development'
  ownerRepo: string;     // derived: '{owner}/{repo}', no .git suffix
  slug: string;          // derived: leaf name, e.g. 'cite-wide'
}

export async function parseGitmodules(path: string): Promise<SubmoduleEntry[]>

Parses the INI-shaped .gitmodules file by hand (no dep). Skips non-GitHub remotes. Defaults branch = development when omitted.

2. githubContentApi.ts — minimal authed Content API client

export async function listMarkdownRecursive(
  ownerRepo: string,
  path: string,
  ref: string,
): Promise<ContentEntry[]>;

export async function fetchRawFile(downloadUrl: string): Promise<string>;

Uses Node's built-in fetch. Reads GITHUB_TOKEN or GITHUB_API_TOKEN from process.env. Treats 404 as "directory missing" (silent skip). Throws on rate-limit-403 with a clear message about which env var to set.

3. frontmatter.ts — in-tree YAML subset parser

export function parseFrontmatter(text: string): {
  data: Record<string, unknown>;
  body: string;
};

Scoped to the YAML subset our content actually uses: top-level key: value, quoted strings, block-style arrays (- item lines), flow-style arrays ([a, b]), booleans, numbers. Doesn't support nested mappings, anchors, multi-line block scalars — we don't author those. ~150 lines, zero dependencies.

The decision to write this rather than reach for gray-matter came from a sharp course-correction during the build session. The Astro Knots tech hierarchy is unambiguous: "fewer dependencies is always better." gray-matter pulls in a YAML parser plus its own machinery; we control the frontmatter format and parse only what we author.

4. rollupLoader.ts — composes the above into an Astro loader-object

export function rollupLoader(options: RollupLoaderOptions) {
  return {
    name: `rollup-loader:${options.collectionName}`,
    load: async (api) => runLoad(api, options),
  };
}

The function:

1. Globs options.localDir for *.md.

2. Calls parseGitmodules(options.gitmodulesPath) for the submodule list.

3. For each submodule: hits listMarkdownRecursive against options.remotePath plus any remoteFallbackPaths (where the legacy context-v/changelogs/ lives).

4. Fetches each file's raw content; parses frontmatter; injects provenance fields (from, from_path, optional legacy).

5. Calls Astro's parseData to schema-validate; on failure, logs a warning and stores the raw frontmatter so one bad file can't fail the whole build.

6. Stores entries via store.set({ id, data, body }).

IDs are prefixed with provenance to avoid collisions across sources: cite-wide/2025-06-07_01, content-farm/2026-05-04_04. URLs follow.

Wiring it into content.config.ts

const changelog = defineCollection({
  loader: rollupLoader({
    collectionName: 'changelog',
    localDir: PARENT_CHANGELOG,
    remotePath: 'changelog',
    remoteFallbackPaths: ['context-v/changelogs'],   // ← legacy honor
    gitmodulesPath: GITMODULES,
  }),
  schema: changelogSchema,
});

const contextV = defineCollection({
  loader: rollupLoader({
    collectionName: 'context-v',
    localDir: PARENT_CONTEXT_V,
    remotePath: 'context-v',
    gitmodulesPath: GITMODULES,
    filter: (relPath, source) =>
      // Don't double-pull legacy changelog files into context-v —
      // they're surfaced on /changelog instead.
      !(source === 'submodule' && relPath.startsWith('changelogs/')),
  }),
  schema: contextVSchema,
});

The filter callback on the context-v collection prevents the same cite-wide/context-v/changelogs/*.md files from showing up on both /changelog (intended) and /context-v (would be redundant).

The honest middle (what didn't go smoothly)

Three sharp course-corrections during the build, each producing a real improvement.

1. gray-matter was the wrong default. First pass added gray-matter to dependencies for frontmatter parsing. The pushback was immediate and correct — Astro Knots tech hierarchy says fewer deps wins, and we control the format. Removed the dep, wrote frontmatter.ts from scratch, ~150 lines, no transitive deps. Build is leaner and easier to audit.

2. Hard frontmatter validation was the wrong default. Initial schemas had from: z.string() (required) on the provenance fields. The pushback: "NO HARD VALIDATION ON FRONTMATTER, ALL FRONTMATTER OPTIONAL, ELEGANTLY HANDLE INCONSISTENCIES." Made every field — including provenance — pass through the lenient preprocessor stack. Loader now wraps parseData in a try/catch and stores raw frontmatter on validation failure rather than failing the build:

let parsed: Record<string, unknown>;
try {
  parsed = await parseData({ id, data: entry.data });
} catch (err) {
  logger.warn(`${tag} schema couldn't validate ${entry.id}: ${err.message}; storing raw frontmatter.`);
  parsed = { ...entry.data };  // fall through with what we have
}

A single weird YAML quirk in cite-wide's history can no longer brick the whole splash.

3. Astro 5.16 was the wrong baseline. package.json declared astro@^5.16.6 from when memopop-site was scaffolded. The astro-knots skill's setup playbook says "Always @latest unless explicitly told otherwise" — so this was drift. Bumped to astro@latest, which currently resolves to 6.2.2. While we were there, dropped @astrojs/tailwind and tailwindcss entirely since the splash never used either (it's all hand-authored CSS with two-tier tokens via CSS variables in BaseLayout.astro).

Three iterations is faster than a single pass that would have shipped any of those mistakes. Worth saying out loud: live course-correction is how this kind of thing reaches a defensible state.

Provenance, rendered

Every rolled-up entry carries data.from = 'cite-wide' (or whichever submodule it came from) and data.from_path = 'context-v/blueprints/Lossless-Citation-Standards.md'. The UI surfaces this:

• Listing cards show a small ◆ cite-wide tag in the meta row, only when the source isn't content-farm.

• Detail pages show the same tag in the article header.

• Legacy fallback entries (the context-v/changelogs/*.md files in cite-wide and image-gin) get an additional legacy pill so a reader knows where the file actually lives in the source repo.

The visual treatment is deliberately understated. Provenance is information, not decoration.

What rolled up (by the numbers)

(Counts from the latest build's loader output. The split between "26 changelog pages" mentioned at the top and "25 entries" here: the changelog index page + 25 detail pages = 26.)

The legacy fallback path matters more than expected — 11 of the 25 changelog entries come from context-v/changelogs/ placements, not from a top-level changelog/. Without honoring the legacy path the splash would surface less than half of the actual ship history.

Build-time and rate-limit shape

A typical clean build now does:

• ~10 directory-listing API calls (one per submodule, plus recursion into context-v/ subdirs)

• ~50–60 raw-file fetches (one per .md found)

• Total: ~60–70 GitHub API calls per build

At 60 req/hr anonymous, that's one build every hour at best. At 5000 req/hr authenticated, that's effectively unlimited.

Hence the workflow change:

- name: Build
  run: pnpm build
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

The auto-provided GITHUB_TOKEN doesn't have explicit access to other repos in the org — but for reading public repo content, it doesn't need to; public content is universally readable. The token's only job here is to identify the request for rate-limit accounting. 60 → 5000 with one line.

Path aliases — small thing, big readability win

tsconfig.json now declares:

{
  "compilerOptions": {
    "baseUrl": ".",
    "paths": {
      "@/*": ["src/*"],
      "@components/*": ["src/components/*"],
      "@layouts/*": ["src/layouts/*"],
      "@loaders/*": ["src/loaders/*"],
      "@content/*": ["src/content/*"],
      "@pages/*": ["src/pages/*"]
    }
  }
}

So loader files import each other as @loaders/parseGitmodules rather than './parseGitmodules', and content.config.ts imports the loader as @loaders/rollupLoader. Astro respects these via its Vite config; the build resolves them without any extra setup.

Convention codified in the splash README so the next site that copies this layout starts there.

What's next

• Watch the next deploy. The first run with the loader pushed will be the real test. If it succeeds, lossless-group.github.io/content-farm/changelog/ will show the full rolled-up feed within a couple of minutes. If anything breaks, the loader logs verbosely under each [rollup:<collection>] tag — including which submodule failed, which file was unparseable, and which entries got stored with raw frontmatter.

• Caching. Right now the loader re-fetches everything on every build. Adding a per-submodule SHA-keyed cache (.cache/rollup/{owner}/{repo}/{sha}.json) would cut the API budget dramatically and survive across local dev iterations. Logged for next round.

• The roll-up convention rolls forward. Now that there's a working reference implementation, the same rollupLoader can serve ai-labs/memopop-ai/apps/memopop-site and any future Lossless splash. The loader itself is repo-agnostic — only the gitmodulesPath and localDir change per site.

• Cite-wide and image-gin's context-v/changelogs/ placement still warrants a quiet normalization pass at some point. The legacy fallback handles it for now, but the changelog skill's aspiration is changelog/ parallel to context-v/. Whenever those two repos get touched next, moving the files is one git mv per file.