Agent-readable documentation

Supacharger Docs publishes complementary discovery formats for traditional search crawlers and AI agents. These artifacts are generated from the same Markdown sources as the rendered documentation so they remain aligned during every production build.

Public endpoints

Endpoint	Purpose
/robots.txt	Declares crawler access and the absolute XML sitemap location.
/sitemap.xml	Exhaustive standard list of public rendered routes.
/llms.txt	Concise Markdown index with descriptions and links to raw documentation pages.
/llms-full.txt	Complete public documentation combined into one Markdown document.
/docs/<route>.md	Raw Markdown alternative for an individual rendered documentation route.

For example, the hosted setup wizard is available as rendered HTML at /docs/guides/setup-wizard and as Markdown at /docs/guides/setup-wizard.md.

Discovery links

Every rendered page advertises /llms.txt with:

<link rel="describedby" type="text/markdown" href="/llms.txt">

Every rendered documentation page also advertises its matching Markdown resource:

<link
  rel="alternate"
  type="text/markdown"
  href="/docs/guides/setup-wizard.md"
>

The conventional root filename and these link relations let an agent discover the machine-oriented representation without guessing or scraping the rendered navigation.

Build behavior

The Docusaurus sitemap plugin generates sitemap.xml. docusaurus-plugin-llms generates llms.txt, llms-full.txt, and individual Markdown pages. A small local Docusaurus plugin generates robots.txt after the build so its policy cannot drift away from noIndex.

Do not edit files under build/; they are disposable output. Edit the source document, its front matter, or the generator configuration and rebuild.

Descriptions in llms.txt come from each page's description front matter when present. Public pages should therefore have a concise, factual description that distinguishes the page from nearby documentation.

Public and private builds

Production builds are indexable by default. Development uses Docusaurus noIndex, and a private preview or staging deployment must set:

SITE_NO_INDEX=true

That setting prevents sitemap generation, emits robots rules that disallow crawling, and adds Docusaurus no-index metadata.

During the temporary private-review period, HIDE_DOCS=true builds the content behind a browser-side development gate instead of omitting it. A valid token query parameter unlocks the site for the current tab and is propagated to internal links. The gated build remains noIndex and omits search and machine-readable documentation bundles. This is a review convenience rather than secure authentication because the token is present in the static client bundle. Restore build-time omission when the review period ends.

Maintenance checks

After adding, moving, or deleting documentation, run a production build and verify:

pnpm run build

Then confirm:

• build/robots.txt allows production crawling and points to the canonical sitemap;
• build/sitemap.xml contains the rendered public route;
• build/llms.txt links to the generated Markdown route;
• the individual .md file exists under build/;
• build/llms-full.txt contains the new content; and
• the rendered HTML includes both describedby and Markdown alternate links.

llms.txt complements, rather than replaces, robots policy, XML sitemaps, canonical URLs, structured data, accurate content, and ordinary search-engine indexing.