# llms.txt vs robots.txt: What Each One Actually Does

**Author:** John Morabito (Founder, /winston)
**Published:** September 19, 2026
**Reading time:** 8 minutes
**Canonical:** https://www.winstondigitalmarketing.com/playbooks/llms-txt-vs-robots-txt/

Two plain text files, both sitting at the root of your domain, both brought up in the same breath any time someone starts talking about AI crawlers. That is where the resemblance ends. One of them is a crawl control file with a written specification and thirty years of practice behind it. The other is a two-year-old proposal for a curated index that no major AI company has publicly committed to reading. People keep asking me which one they should be using, as if it were a choice between two versions of the same thing, and the confusion produces real mistakes. I have watched a client add llms.txt expecting it to keep a bot out, and I have watched another block every AI agent in robots.txt and then ask why assistants never mention them.

## What robots.txt actually does

robots.txt is a permission file for crawling. You list user agents, you tell each one which URL paths it may request, and a crawler that respects the convention reads the file before it fetches anything else on your site. It started as an informal agreement in the mid-nineties and was eventually written up as RFC 9309 in 2022, which is why the companies running crawlers now state plainly in their documentation that they follow it. There is no enforcement behind any of it. A crawler that wants to ignore the file will ignore the file. It is a posted sign rather than a lock, and if you need an actual block you need rules at the server or the edge.

The part people get wrong is the difference between crawling and indexing. Disallowing a URL tells the crawler not to fetch the page. It does not tell the search engine to leave the URL out of its index. Google has documented this for years: a URL you blocked can still appear in results if other pages link to it, usually with no description, because the engine knows the address exists and was never permitted to read what is there. If you want a page out of the index, you have to let it be crawled and serve a noindex directive on the page itself. Blocking it in robots.txt is what prevents the engine from ever seeing that directive.

## The AI crawler wrinkle

The AI companies do not run one crawler each. They run several, and the different agents do different jobs. OpenAI publishes GPTBot, OAI-SearchBot and ChatGPT-User. Anthropic publishes ClaudeBot, Claude-SearchBot and Claude-User. Perplexity has PerplexityBot. Google added a token called Google-Extended that does not crawl anything at all; it exists so you can tell Google whether content Googlebot already fetched may be used for Gemini training.

The distinction that matters to your business is that some of those agents are gathering content that feeds model training, and some of them fetch a page in the moment, while a user is waiting, so the assistant can answer the question and cite a source. Those are different transactions. The first one takes your content and gives you nothing back directly. The second one is how your link ends up in front of somebody who is actively looking for what you sell. A blanket disallow aimed at anything with AI in the name takes out both.

So write the file agent by agent and decide deliberately. Whether you want to contribute to training and whether you want to be retrievable are separate questions with separate answers, and robots.txt is perfectly capable of answering them differently in the same file.

## What llms.txt is proposing

llms.txt came out of a proposal from Jeremy Howard of Answer.AI in September 2024. The idea is a markdown file at your root that works as a curated index: a line or two saying what the site is, then a linked list of the pages you consider most useful, pointing where possible at clean text versions of them. The argument behind it is reasonable. A model working inside a limited context window should not have to chew through your navigation, your cookie banner and your footer to reach the three pages that answer the question it was asked.

What it is not is a control file. It permits nothing and restricts nothing. Publishing one has no effect on whether a crawler is allowed to fetch anything. It is a suggestion in file form, and it only does anything at all if the thing reading your site chooses to look for it. Adoption so far runs heavily on the publisher side. Plenty of sites have added the file, and I have not seen the major providers document llms.txt support in their own crawler documentation. That may change. For now I treat it as a cheap bet on a format that might get adopted, and I would not build anything on top of it. The [llms.txt guide](https://www.winstondigitalmarketing.com/playbooks/llms-txt-guide/) covers the format itself and how to write one.

## Neither file is a ranking lever

robots.txt can only take you out of the running. There is no configuration of it that makes an assistant more likely to cite you; the best it does is avoid getting in your own way. llms.txt has no citation mechanism attached to it either. At best it saves a model a little work, assuming it reads the file.

What actually determines whether you show up in an AI answer is duller and harder. The page has to render its content in the HTML rather than after a script runs. It has to be reachable through real internal links, which is the whole subject of [internal linking for AI crawlers](https://www.winstondigitalmarketing.com/playbooks/internal-linking-for-ai-crawlers/). And the writing has to answer a specific question in a self-contained passage that can be lifted out and quoted without the surrounding page, which is what [writing content AI actually cites](https://www.winstondigitalmarketing.com/playbooks/how-to-write-content-ai-cites/) is about. That work is what moves the number, and no configuration of either text file substitutes for it.

## The stance I take on both

For a client site, this is the order I work in:

- Read the current robots.txt line by line and find out what it is actually blocking. Stale disallow rules from an old CMS are extremely common, and so is a pattern that blocks CSS or JavaScript the page needs to render.
- Split the AI agents into training and retrieval, and make a deliberate call on each group rather than one sweep. Retrieval agents stay allowed for almost every business I work with.
- Serve clean markdown versions of the important pages, because that is the thing llms.txt is meant to point at and it is useful on its own. We do this on this site with a worker that serves a markdown twin of every page to bots that ask for it, which is covered in the [agent-ready website plan](https://www.winstondigitalmarketing.com/playbooks/agent-ready-website-4-week-plan/).
- Publish llms.txt as an index of those markdown files, then keep it current. A stale index is worse than none, because it advertises pages that have moved.
- Check the server logs afterward to see which agents are actually showing up and what they are requesting, rather than assuming the file did what you meant.

That whole list is maybe a day of work, and the first two items are the ones that carry the weight. If robots.txt is excluding the agents that would have cited you, no amount of curation downstream fixes it. If you want us to look at what your files are doing, the [free AI visibility audit](https://www.winstondigitalmarketing.com/audit/) reads both of them, and the [GEO service](https://www.winstondigitalmarketing.com/services/generative-engine-optimization/) is where we do the rest of it.

## Frequently asked questions

### What is the difference between llms.txt and robots.txt?

robots.txt is a crawl control file. It lists user agents and tells each one which URL paths on your site it is allowed to request, and well-behaved crawlers read it before fetching anything. It has a formal specification behind it and decades of use, and every significant crawler operator documents which of its agents obey it. llms.txt is a curation file. It is a proposed markdown index that sits at your root, describes what the site is, and links to the pages you consider most useful, ideally in a clean text form. It restricts nothing and permits nothing. One file answers the question may you fetch this. The other answers the question here is what is worth fetching. They are not two versions of the same idea and one does not replace the other.

### Does robots.txt block AI crawlers?

It does for the ones that choose to honor it, which includes the crawlers run by the major AI companies, since they publish their user agent names specifically so publishers can address them in the file. It does nothing about scrapers that ignore the file, because robots.txt has no enforcement mechanism. It is a posted sign rather than a lock, and if you need an actual block you need server-side or edge-level rules. The more useful point is that a blanket block is usually the wrong move. AI companies run separate agents for separate jobs, and some of them fetch your page at the moment a user asks a question so the assistant can answer and cite you. Blocking those removes you from the answers.

### Is llms.txt an official standard?

No. It is a proposal, published by Jeremy Howard of Answer.AI in September 2024, and it has not gone through any standards body. That matters mostly because of what it implies about the other side of the transaction. Adoption so far is heavily on the publisher side: a lot of sites have added the file. I have not seen the major AI providers document in their own crawler documentation that they read it. robots.txt is different in kind here. It began as an informal convention too, but it was eventually written up as RFC 9309 in 2022, and the companies running the crawlers state plainly that they follow it. Treat llms.txt as a low-cost bet on a format that may or may not get picked up.

### Will adding llms.txt improve my AI visibility?

Probably not on its own, and you should be suspicious of anyone who promises it will. There is no ranking or citation mechanism attached to the file. At best it saves a model some work by pointing it at clean versions of your best pages, which only helps if the crawler reads the file in the first place. The things that actually move whether an assistant cites you are more ordinary: pages that render their content in the HTML rather than after a script runs, crawlable internal links so the important pages are reachable, and writing that answers a specific question in a self-contained passage a model can lift. Add llms.txt because it costs an hour, not because you expect it to do the work.

### Should I block AI crawlers in robots.txt?

Answer it as two separate questions rather than one. The first is whether you want your content used to train models, which is a licensing and principle question and a legitimate reason to disallow the training agents. The second is whether you want to be retrievable, meaning you want an assistant to be able to fetch your page while answering a user and put your link in the response. For most businesses trying to get found, the answer to the second is yes, because that is the traffic and the attribution. Since the AI companies publish separate user agents for these different jobs, you can answer the two questions differently in the same file. Write it agent by agent instead of sweeping everything with an AI in the name into one disallow.