Let’s start with the conclusion: llms.txt will not automatically allow your brand to be referenced by ChatGPT. It is a Markdown map placed in the root directory of the website, allowing AI Agent and real-time retrieval in the inference stage to bypass the navigation bar, advertisements and JavaScript in your web page and read clean content directly. Done right, it saves AI parsing costs; done wrong, it’s just a file that no one reads. This article explains the syntax specifications, placement, crawler reading principles, and its current real boundaries at once.
What is llms.txt and what is the difference between it and robots.txt
llms.txt is a proposed specification proposed by Jeremy Howard of Answer.AI in September 2024. The file name is fixed to llms.txt, placed in the root directory of the domain, and the content is pure Markdown. Its job is to "curate": list the pages in the entire website that are most worthy of being read by AI in a streamlined list, with a link and a description for each one. It is not a configuration file, and it does not control who can enter. This is the prerequisite for all subsequent implementations.
What does a qualified llms.txt look like?
The specification has very light requirements on format, but the order is important. The entire file is written in standard Markdown, allowing any language model to be parsed at zero cost. The skeleton from top to bottom is as follows:
- An H1 title, write the name of the website or project. This is the only required field in the specification.
- Following a blockquote (starting with >), use a sentence or two to explain what this website is doing. This is the first summary that the AI reads when crawling the entire archive.
- If necessary, add a few paragraphs of plain text to explain the background or instructions for use, but do not have a second H1 in the entire file.
- There are several H2 blocks, and a list of Markdown links is placed under each H2. The format is [page name] (website): one sentence description; H2 is used for classification, such as "services", "documents" and "resources".
- A block named ## Optional (can be omitted): The links listed here are secondary content and can be skipped entirely when the AI context length is tight.
Where to put it and what format to provide it in?
There is no flexibility in location: the file must be placed in the root directory of the domain, and the URL is https://yourdomain/llms.txt. AI cannot find it if it is inserted into a subdirectory or subdomain. Use UTF-8 for encoding. It is recommended that the Content-Type returned by the server be set to text/plain or text/markdown. Do not let the browser download it as a file. Two matching conventions are worth doing together: the first is the optional llms-full.txt, which embeds the entire site content into a single large file, which is suitable for situations where you want AI to read it all at once, but the file is large and difficult to update, so small and medium-sized websites usually skip it first; the second is the .md mirror page, which adds .md after the original URL (for example, /pricing corresponds to /pricing.md) to provide a pure Markdown version, llms.txt The links in it can point directly to clean content instead of HTML full of layout tags. Tenten GEO's /pricing.md does this to provide machine-readable pricing.
How does the AI crawler read llms.txt (reading principle)
Let’s first separate two things that are often conflated. There are two points in time when AI comes into contact with the content of your website: the first is the training phase, when crawlers such as GPTBot, ClaudeBot, and Google-Extended crawl web pages on a large scale and feeds them to the model; the second is the inference phase, when users ask questions and AI needs to check information in real time, it is captured on-site by the browsing function of PerplexityBot, ChatGPT, or various AI Agents. llms.txt mainly works on the latter. Tools that support it will first grab the llms.txt in the root directory, use it as a navigation map, decide which links to expand and read next, and then grab the corresponding Markdown, saving the cost of "downloading the entire page of HTML, stripping off the navigation bar and script, and then guessing which paragraph is the main text."
But one thing needs to be made clear: As of now, OpenAI, Anthropic, Perplexity, and Google have not publicly committed to using llms.txt as a formal signal for training or ranking, and Google’s attitude is even reserved. At this stage, those who actually actively read llms.txt are mostly developer tools and Agent frameworks, such as some code assistants, file retrievers, and RAG or Agent processes you build yourself. Therefore, positioning llms.txt as "paving the way for Agent and self-built search" will be much more honest than positioning it as "AI version of SEO ranking switch".
How to verify after deployment and six common mistakes
- Misplaced: Stuffed in /docs/llms.txt or subdomain, AI cannot find it at all, it only recognizes the domain root directory.
- Use it as robots.txt: Write Disallow in llms.txt to block crawlers. If the format is wrong, nothing will happen. Please go back to robots.txt for access control.
- The link points to dirty HTML: the list links to a page full of pop-ups and sidebars, which means that the parsing cost is not saved. Try to match the .md version.
- Stuffing too many links at once: dumping hundreds of URLs from the entire site will dilute the focus. llms.txt is a curation, not a sitemap. Just pick the pages you want to be cited the most.
- Descriptions should be written in marketing language: the description behind each link should be a factual sentence, allowing AI to determine whether it should be expanded, not an advertising slogan.
- Once built, forget about it: if the page is not updated simultaneously after revisions, pricing changes, or the page goes offline, points will be deducted if the AI reads expired information.
Verification is not difficult. First, use a browser to directly open https://yourdomain/llms.txt and confirm that the response is plain text, does not have 404, and has not been downloaded as a file. Then paste the entire content into ChatGPT or Claude and ask it to "describe what services this website provides based on this llms.txt." If it can repeat it correctly, it means the structure is clear and AI can understand it. Finally, go back and check whether robots.txt has blocked the AI user-agent you want to let go. This step is most often missed, but it will make all the previous efforts in vain.
Should it be done and when should it be done?
Honest advice: llms.txt is low-cost, low-risk, and worth doing, but don't expect citation growth on its own. It is the outermost and cheapest piece of the entire AI visibility project. What really determines whether you will be cited by AI is whether the content itself can be cleanly crawled, whether the brand entity is credible enough in the eyes of AI, and whether key paragraphs can be directly extracted. llms.txt just paves a way for these contents to travel. Whether there is any goods on the road depends on your content.
llms.txt is a courtesy, not a guarantee. It makes it easier for an AI that's willing to read to read you, but it doesn't force an AI that doesn't want to read to start reading you.— Tenten GEO
After deploying llms.txt according to this article, the next question is usually: Have I been read and referenced by the six major AI engines? This must rely on actual measurement, not guesswork. If you want to know where your visibility gap lies, you can go to /contact to schedule a 30-minute GEO diagnosis. We will give you a visibility snapshot of the six major engines on the spot. By the way, you can check whether your llms.txt has actually been read.
