llms.txt Support
llms.txt is a community standard for publishing an index of a site's documentation pages — similar to sitemap.xml but targeted at language models.
What It Is
A plain-text file at {docs_root}/llms.txt that lists the documentation pages for a project, one per line, with a short title:
# FastAPI
## Docs
- [Tutorial - User Guide](https://fastapi.tiangolo.com/tutorial/): Main tutorial.
- [Advanced User Guide](https://fastapi.tiangolo.com/advanced/): Advanced usage.
- [Deployment](https://fastapi.tiangolo.com/deployment/): How to deploy.
...
How DocuLayer Uses It
When llms.txt is present, DocuLayer:
- Fetches and parses the index (one HTTP request)
- Keyword-scores each entry against your query using term overlap
- Fetches only the top 1–3 pages most likely to answer the query
This replaces naive crawling of the entire docs site.
Without llms.txt: fetch root page → parse links → fetch each → search
With llms.txt: parse index → score entries → fetch 1–3 pages → search
Packages with Built-in Support
These packages have hardcoded llms.txt URLs in DocuLayer's shortcut table — zero extra resolution steps:
| Package | Docs URL |
|---|---|
anthropic | docs.anthropic.com |
astro | docs.astro.build |
fastapi | fastapi.tiangolo.com |
httpx | www.python-httpx.org |
langchain | python.langchain.com |
nextjs | nextjs.org/docs |
openai | platform.openai.com/docs |
pydantic | docs.pydantic.dev |
react | react.dev |
shadcn | ui.shadcn.com |
supabase | supabase.com/docs |
svelte | svelte.dev/docs |
tailwindcss | tailwindcss.com/docs |
vite | vitejs.dev/guide |
vue | vuejs.org/guide |
Any URL Works
Passing a direct URL — including to a site not in the table — triggers automatic llms.txt probing:
doculayer fetch https://my-internal-docs.company.com
DocuLayer will probe https://my-internal-docs.company.com/llms.txt and use it if found, or fall back to HTML parsing of the root page.
Publishing llms.txt for Your Project
If you maintain a documentation site, publish an llms.txt at the root. The format is simple:
# Project Name
## Section Name
- [Page Title](https://your-docs.com/page/): Short description.
- [Another Page](https://your-docs.com/other/): Another description.
DocuLayer will discover it automatically for any user who passes your docs URL.