Documentation
Onboarding
Getting Started
Power & Battery
Hardware & Components
Build & Assembly
Radio & RF Design
Frequencies & Regulations
Firmware & Software
USB & Connectivity
GPS & Navigation
Morse & Communication
Modes & Operation
Field Operations & Rescue
Building Effectively
Build Variants
Project & Reference
Docs Project & Reference Robots and Sitemap

Robots and Sitemap

Edit on GitHub

The robots.txt and sitemap files: what they contain, why they exist, and how they are generated and verified.

Robots and Sitemap

Two small files tell search engines how to crawl the site. Both are generated at build time and verified by CI, so they cannot silently go stale.

robots.txt

The file at /robots.txt declares:

  • Which crawlers may access the site
  • Where the sitemap lives
  • The disallowed paths (none for a public static site)
CODE
User-agent: *
Allow: /

Sitemap: https://aegis-beacon.vercel.app/sitemap-index.xml

The sitemap

Astro’s sitemap integration generates a sitemap of every public page at build time. For large sites it emits a sitemap index pointing to multiple sitemap files. The CI verifies that the built sitemap covers the expected page count (see CI/CD).

Why they matter

FileEffect
robots.txtKeeps crawlers efficient and pointed at the sitemap
SitemapTells engines which pages exist, including new wiki pages without waiting for link discovery

Verification

The CI script checks:

  1. robots.txt exists and references the sitemap
  2. The sitemap exists and parses
  3. Every internal link resolves (no 404s)

A new page that fails these checks blocks the merge: that is the pipeline keeping search visibility honest.