Skip to content
AskVault Docs
Try Free →

llms.txt explained

Written by Joseph Russo, Principal Architect, Networks and Tech, AskVault · Reviewed by Allison Chan
Last updated: · 4 min read

What llms.txt is

A plain-text Markdown file at your domain's root (yoursite.co/llms.txt) that:

  • Lists your important pages in a structured format.
  • Provides a content index an AI tool can read in one request.
  • Describes each page in a short line.

Different from robots.txt, which grants or denies crawler access. llms.txt only describes content; it grants and denies nothing.

Why publish one

AI engines (ChatGPT Search, Perplexity, Claude, Gemini) answer from what they can find and read. A curated index can help any tool that chooses to read it:

  • Find your canonical pages without crawling the whole site.
  • Read clean Markdown instead of parsing HTML navigation.

Whether a given engine reads llms.txt is up to that engine, and most have not documented it. Treat the file as a cheap, low-risk signal, not a ranking lever.

llms.txt. The content index defined by the proposal. A list of important pages with descriptions.

llms-full.txt. A common companion convention: the full text of your pages in one file. It is not part of the core llms.txt proposal.

robots.txt. Where AI crawler access is actually controlled. Each major AI crawler publishes a user-agent token you can allow or disallow.

Sample llms.txt

# AskVault


> AI customer support and helpdesk platform.


## Docs


- [Getting Started](https://docs.askvault.co/getting-started/): Set up AskVault
- [Embed on React](https://docs.askvault.co/how-to/embed-ai-chatbot-on-react-website/): React widget setup
- [Skills overview](https://docs.askvault.co/skills/skills-overview/): Built-in skills


## API


- [API getting started](https://docs.askvault.co/api/getting-started/): Auth and first request

One H1 with the site name, an optional blockquote summary, then H2 sections of Markdown links. The spec at llmstxt.org defines the structure.

Sample robots.txt rules for AI crawlers

User-agent: GPTBot
Allow: /


User-agent: OAI-SearchBot
Allow: /


User-agent: PerplexityBot
Allow: /


User-agent: ClaudeBot
Allow: /


User-agent: Google-Extended
Allow: /


User-agent: Bytespider
Disallow: /

Allows the named AI crawlers and blocks one. Compliance with robots.txt is voluntary on the crawler's side, so it is a request, not an enforcement mechanism.

Implementation

About 30 minutes:

  1. Create /public/llms.txt in your site root.
  2. List your top 10 to 30 pages with descriptions.
  3. Review /public/robots.txt and set the AI crawler rules you want.
  4. Optional: /public/llms-full.txt with full page text.
  5. Deploy.
  6. Verify at yoursite.co/llms.txt.

Tools

  • Generators. Some CMS and docs frameworks can generate the file from your content.
  • AskVault's own docs site generates its llms.txt and llms-full.txt from its content collection at build time. This is for our documentation, not a feature for your workspace.

Limits

  • Size. Keep llms.txt short; it is an index, not a content dump. Put full text in llms-full.txt.
  • Format. Follow the Markdown structure in the spec.

Common pitfalls

File too long. Trim to essentials; long files may be truncated by the tool reading them.

Wrong format. Start with a single H1 and use Markdown links.

Outdated content. Regenerate when pages change; stale entries point tools at the wrong page.

Using llms.txt to block crawlers. It cannot. Use robots.txt.

FAQ

Is llms.txt an official standard?

No. It is a community proposal published at llmstxt.org, not a standard from a standards body. Whether an AI engine reads the file is that engine's decision, and most have not documented whether or how they use it.

Does AskVault generate an llms.txt file for my workspace automatically?

No. AskVault does not create or host an llms.txt for your workspace. If you want one, publish it on your own site. Separately, if you turn on NLWeb for a workspace, AskVault serves a public NLWeb endpoint for it: a discovery URL and an ask URL that agents supporting the NLWeb protocol can send questions to. NLWeb is off by default.

Will all AI crawlers respect my llms.txt?

llms.txt is not an access-control file, so there is nothing to respect or enforce. To allow or block an AI crawler, use robots.txt with that crawler's published user-agent token.

Will this guarantee AI engines cite me?

No. It may help a tool that reads it find the right page; it guarantees nothing.

Should I copy entire docs into llms-full.txt?

Only if you want your full text available in one file. Keep llms.txt itself as the short index.

Was this page helpful?