Changelog — 2026-05-09 (#1)

Sweep: llms.txt + sitemap + robots.txt across every site and splash

Why Care?

Yesterday the question was "how do we measure visits." Today the question was "how do visitors and AI crawlers find anything in the first place." SEO has been a checkbox for years; GEO — making content legible to generative engines — is becoming the second checkbox. The llmstxt.org spec is the cleanest way to do both at once: a small /llms.txt link index for fast triage, a larger /llms-full.txt corpus dump for one-fetch ingest, plus the long-standing sitemap-and-robots pair for traditional crawlers.

The interesting part isn't the spec — it's that the same per-site shape held up across the fleet. One markdown-template-plus-assembler pattern, eight different content collections, eight different voices. Site authors keep editing prose in src/llms/*.md; the endpoint code never changes. The pseudomonorepo posture earned its keep again: every site implements the same surface, none of them shares the implementation, none of them drifted.

What's New?

/llms.txt and /llms-full.txt per the llmstxt.org spec

• /llms.txt — a link index of every published essay, note, context-v document, and changelog entry, sorted by date_modified desc, with a "if you're an LLM and you want everything, fetch /llms-full.txt" pointer at the top

• /llms-full.txt — concatenated raw markdown corpus across the same collections, with per-entry metadata headers, prerendered at build time so the multi-MB response never assembles on a request hot path

• Markdown-template-plus-assembler split — all voice and structural prose lives in src/llms/*.md; the endpoint code (src/pages/llms.txt.ts, src/pages/llms-full.txt.ts) is a dumb token substituter. Wording changes never touch TypeScript.

• Defensive publish/private gate — e.data.publish !== false && e.data.private !== true filters drafts before either endpoint emits, even though the page templates today don't yet apply the same gate. Belt-and-suspenders for when frontmatter private: true becomes load-bearing.

• Token-substitution scheme — plain {{TOKEN}} placeholders, no Mustache, no Handlebars. Adding a dynamic value means registering it in the substitution map and documenting it in src/llms/README.md.

@astrojs/sitemap integration

• Auto-generates sitemap-index.xml + sitemap-0.xml from every prerendered page Astro emits

• Filter excludes the two llms.txt endpoints (they serve LLMs, not search engines) and the 404 page

• site: set to the canonical production origin in each astro.config.mjs so absolute URLs resolve correctly

robots.txt pointing at the sitemap

• Allow-everything default (every site is intentionally indexable; brand-kit and design-system pages already carry their own noindex,nofollow meta where needed)

• Sitemap: directive at the top, pointing crawlers at sitemap-index.xml for those that don't honor the <link rel="sitemap"> document hint

BaseLayout <link> hints

Two new lines in every site's BaseLayout.astro:

<link rel="sitemap" type="application/xml" href="/sitemap-index.xml" />
<link rel="alternate" type="text/markdown" href="/llms.txt" title="llms.txt for this site" />

The rel="alternate" type="text/markdown" hint is the convention emerging around llms.txt — it tells GEO crawlers that the markdown surface is a first-class alternate to the rendered HTML, not just a curiosity.

Where it landed

This is a parent-level changelog; the per-submodule diffs live in each child repo. Same shape, different content collections feeding it.

The two committed surfaces were intentional first-followers — cv-kit/splash because it's the smallest editable surface (good for shaking out the shape), mpstaton-site because it carries the largest content corpus (good for stress-testing the corpus assembler). Everything else is the same shape, queued for child-repo commits and parent-repo submodule-pointer bumps.

How the day actually went

The LFM v0.3.0 + Astro 6.3.1 + OpenPanel sweep on 5/8 was the precondition. Once every site was on the same Astro and the same Analytics shape, layering an additional discoverability surface across the fleet became identical-modulo-IDs work. Same pattern: write the canonical reference once (mpstaton-site was the testbed since it has the richest content collections — essays, notes, context-v documents, changelog), copy the src/llms/*.md + endpoint pair into each sibling, regenerate the corpus index against that site's content shapes, paste the BaseLayout <link> lines.

The hardest part wasn't any one site — it was the discipline of not extracting prematurely. Twice today the temptation showed up to publish a @knots/llms-txt package: same endpoint code in eight repos, same template tokens, same publish/private gate. Twice we walked it back. The endpoints differ in which collections they iterate over, which fields they pull, which gate they apply — and those differences are the whole point. A shared package would force every site through one config object, and the moment a site needs a new collection (or a different sort order, or a custom token) the abstraction is in the way. Copy-pattern, not shared package. Same call we made on the <Analytics /> component yesterday, same reasoning.

The cv-kit splash was the first to commit because its content surface is small enough that the assembler converged in one shot. mpstaton-site shipped the same day but was held back from a same-day commit because it was bundled with the next morning's "Scroll Report" work — the combined commit landed 5/10 (b3b5788 feature(llms-txt): add seo geo optimization for llms-txt specification).

What's still open

• Submodule-pointer bumps in this parent repo. The sites/* and splash submodules with WIP working trees need their child-repo commits + pushes, then a parent-repo bump to track the new HEADs. Routine; queued for the next batch.

• Sitemap on splash repos. All four splashes have llms.txt + robots.txt, but @astrojs/sitemap integration is currently only on the Vercel-deployed sites (mpstaton-site). Splashes are GitHub Pages; sitemap there needs a pass to verify the absolute-URL behavior under the Pages base path.

• Vercel + Pages canonical URLs. A few sites still have placeholder site: values in astro.config.mjs; double-check before deploy so <loc> entries in the sitemap and absolute links in llms-full.txt resolve.

• Per-site llms-full.txt in hypernova-site. Currently only llms.txt is staged there — the corpus-assembler half is queued.

Related

• [[2026-05-08_01]] — yesterday's analytics + Astro + LFM sweep that this work sat on top of

• [[Codifying-a-Comprehensive-Extended-Markdown-Flavor-and-Shared-Package]] — the LFM spec, indirectly relevant since parseMarkdown is what shapes the corpus the llms-full.txt endpoint emits

• llmstxt.org — the spec

• sites/mpstaton-site/src/llms/README.md — the canonical token + template documentation that all sister sites' implementations follow