What This Is
A sitemap indexing framework that creates machine-readable indexes at `/llms.txt` and `/llms-full.txt` at the domain root, containing contextual descriptions of core routes.
What Exists Today
• Configured `/llms.txt` maps at root.
• llms.txt Linter formatting check engine.
What It Is Intended to Do
Designed to format domain indexing paths so autonomous agents and search scraper crawlers digest site structures cleanly, reducing token overhead cost during crawlings.
What We Are Measuring
• Bot request counts targeting `/llms.txt` compared to standard sitemap XML path queries.
Current Evidence
• Trial Status: Baseline collection active.
• Observations: Crawlers access llms.txt at an observed frequency; specific influence on search ranking weights is not isolated.
• Limitations: Many AI crawlers do not disclose user-agent parameters, obscuring crawl tracking logs.
What We Cannot Yet Claim
We do not claim that deploying llms.txt compliance guarantees immediate AI indexing or eliminates search scraper costs.
Initialize Substrate
Explore how to deploy llms.txt Compliance templates for your startup.
Explore Pricing