llms.txt Deployment Guide
llms.txt Guide: When to Use It and How to Maintain It
An optional llms.txt implementation guide: proposal syntax, templates, curated links, Markdown exports, six deployment checks, and maintenance. Google Search ignores the file; use by other systems requires separate verification.
v1.0 · June 2026 · Tenten GEO Research
CH.01
What llms.txt Is, and an Honest Look at Where It Stands
llms.txt is a curated Markdown overview for agents that use it. The current proposal allows a file at the site root or a subpath such as /docs/llms.txt, covering that path’s content. It is a community proposal; adoption and behavior depend on the agent.
Jeremy Howard published the llmstxt.org proposal in September 2024 and updated it to v2 on 10 August 2026. It gives agents concise context and selected links to detailed content. The file can describe the whole site from the root or a particular section from its own path.
The proposal lists documentation-site adopters, but publishing a file does not prove that an agent uses it. Google’s current guidance says Search ignores llms.txt: maintaining it neither helps nor harms Search visibility or rankings. This is a Google Search statement, not a rule for every provider.
Our take
Choose llms.txt when an identified tool or agent workflow uses it and the maintenance is justified. Estimate writing, deployment, link checks, and synchronization for your own site. It is not a mandatory first audit item, a near-zero-risk investment, or evidence of citation gains.
- →llms.txt can give participating agents selected context and links; it does not establish a ranking or citation improvement
- →An agent can use llms.txt when its workflow supports it or it is given the file; there is no universal first-stop requirement
- →This guide mentions geo.tenten.co/llms.txt and /pricing.md as reference URLs to inspect. Content and responses can change; no fixed line count, current plan count, or universal agent adoption is established here.
CH.02
Proposal Syntax and a Copyable Example
The proposal uses Markdown: a project H1, optional summary and context, sections, and selected links. These conventions help participating workflows interpret the file; they do not guarantee every model understands it.
| Element | Syntax | Required | Purpose |
|---|---|---|---|
| Site name | # Project or brand name | Yes (the only required field) | The file's single H1: tells AI whose site this is |
| Summary | > A blockquote | Strongly recommended | One to three sentences on who you are, what you do, and for whom: AI's first impression of your site |
| Supporting notes | Plain paragraphs or lists (no heading) | No | Context needed to interpret the rest: e.g. service model, methodology |
| Sections | ## Section name | No | Group links with H2s: Services, Docs, Resources, Contact… |
| Link list | - [Title](URL): one-line description | No | One link per line; the description tells AI why this page is worth reading |
| Optional section | ## Optional | No | A spec reserved word: AI can skip this block when context runs short: put secondary links here |
Elements of the llmstxt.org community proposal; consult the current specification, not a universal agent contract.
A minimal template you can copy straight in
- →# Acme Corp.
- →> Acme is a B2B warehouse-automation provider serving 50 to 500-person manufacturers with WMS implementation and consulting; clients include… (one to three sentences, concrete and verifiable)
- →## Services, then each line: - [WMS Implementation](https://acme.com/services/wms): 30-day rollout, pricing tiers, and deliverables
- →## Docs: - [API Reference](https://acme.com/docs/api): complete REST API reference, including auth and error codes
- →## Contact: - [Book a consult](https://acme.com/contact): a free 30-minute assessment
- →## Optional: put secondary links here: blog index, press coverage, and so on
Use factual descriptions that explain what a link contains. Select links for the intended reader or consuming workflow; the proposal does not impose a universal 10–30-link limit. geo.tenten.co/llms.txt is a reference URL, not proof of a current four-section or under-40-line layout. Check the response you actually receive.
CH.03
llms.txt, robots.txt, sitemap.xml: Three Files, Three Jobs
robots.txt communicates crawl instructions to agents that honor it; sitemap.xml lists URLs for discovery; llms.txt proposes a curated overview for agents that use it. None is enforced access control, and llms.txt does not replace ordinary Search eligibility requirements.
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| Question it answers | Where can you crawl? | Which pages do I have? | Which pages matter, and what do they say? |
| Format | Dedicated directive syntax | XML | Markdown |
| Primary audience | Automated crawlers that honor the protocol; user-triggered exceptions vary by provider | Search-engine indexers | Agents and tools that explicitly use the proposal |
| Nature of content | Allow / block rules | Full URL set + last-modified | Curated links + semantic descriptions |
| Standard status | Formal standard (RFC 9309) | Industry standard (sitemaps.org) | Community proposal (llmstxt.org) |
| Cost of getting it wrong | Restricting a search crawler can limit its access; training opt-out is a separate choice | A sitemap can help discovery; omission does not prove a page is undiscoverable or guarantee indexing | Agents that use the file receive less useful context; other discovery paths may still work |
Access instructions, discovery hints, and optional curated context have different purposes; none replaces readable pages or proves a hidden engine pipeline.
In a 30-day audit, first check the access intended for search crawlers and user-triggered fetchers, then ordinary page discovery, and finally the optional llms.txt file. Blocking GPTBot or ClaudeBot for training does not by itself block OAI-SearchBot or Claude-SearchBot. Check the applicable robots group and CDN rule for each agent.
Chapter takeaway
Separate access, discovery, and curated context. Check robots.txt and CDN controls for the agents you intend to serve; an optional llms.txt file cannot undo a blocked request and is not required for Google’s AI features.
CH.04
Advanced Techniques: llms-full.txt and .md Mirror Pages
For a workflow that benefits from Markdown, consider a consolidated export or selected .md mirrors. Neither is needed for Google Search. HTML is not inherently unreadable, and the best format depends on the consuming system.
llms-full.txt: all your content in a single file
llms-full.txt can consolidate exported content for a consuming tool. File size, context limits, privacy, and update cost depend on what you include. Decide using the tool’s documented needs and a synchronized source; this guide does not establish a superior return for full exports or mirrors.
.md mirror pages: append .md to the original URL
A .md mirror can expose selected page content as Markdown. /pricing.md is a reference URL: compare its actual response with the HTML page’s current prices, terms, and currency before relying on it. A mirror is an optional format, not proof that all agents need it or read it.
- →Choose mirror candidates by actual reader or tool needs, such as specifications or pricing; do not assume these are universally the most-requested AI pages.
- →If you maintain mirrors, link them with accurate descriptions in llms.txt where useful. The presence of a link does not prove automatic use.
- →Keep dates, prices, and terms accurate across formats. Stale copies can mislead readers or tools; a date label alone does not establish freshness.
- →Maintenance discipline: when the HTML page changes, update the .md in lockstep: put it on the same release checklist, don't rely on memory
CH.05
Six Deployment Checks
Use these six checks to review descriptions, updates, link selection, rendered content, crawler access, and response format. They are practical checks, not a published estimate of how often client sites fail.
- →Check 1 · Descriptions: explain the information each linked page contains, with concrete facts rather than slogans.
- →Check 2 · Updates: review changed names, URLs, and facts when publishing. A quarterly review can be a Tenten planning reminder, not a proven universal effort estimate.
- →Check 3 · Selection: keep links useful for the chosen scope and consumer. There is no universal link-count threshold or proof that a long list destroys all context value.
- →Check 4 · Rendering: inspect linked content in raw HTML and the target system’s rendered view. Google runs JavaScript; other agents vary. Consider server output or mirrors only for an identified access need.
- →Check 5 · Agent access: review search crawlers such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot against robots.txt, WAF rules, and verified logs. A deliberate GPTBot or ClaudeBot training opt-out is not an llms.txt failure. Apply user-triggered fetcher exceptions separately.
- →Check 6 · Response: verify status, suitable text Content-Type, and actual Markdown body. A 200 error page or header-only test can hide a broken export; do not require one MIME label for every consumer.
Chapter takeaway
Fix accessible responses and useful content before polishing the file. A blocked search crawler or an empty page can limit that agent’s access; neither establishes that every AI system loses all knowledge of the site. Keep deployment checks separate from ranking or citation claims.
CH.06
The Four-Step Deployment and Validation Checklist
If you choose to deploy, use these four steps to write, publish, validate, and maintain the file. Estimate effort for the site and workflow; no fixed afternoon or minute-by-minute completion time is promised.
STEP 1
Inventory and write
Choose the pages needed for your scope and consumer. Follow CH.02 with a truthful summary, grouped links, and useful descriptions. Ask whether the file accurately states who you are and what the selected pages contain; there is no required page count.
STEP 2
Publish at the appropriate path
Use https://yourdomain/llms.txt for a site-wide overview, or a scoped path such as /docs/llms.txt for that section under the v2 proposal. On a static Next.js site, public/llms.txt is one option; a CMS can serve a static route. Check a 200 response, suitable text Content-Type, UTF-8, and the actual Markdown body.
STEP 3
Validate responses and content
Check transport and the actual body, then follow links and compare facts with their source. If a model is given the file, a summary test can reveal issues to investigate, but an error may arise from the model, access, or context as well as your description. It does not prove autonomous discovery or citation.
STEP 4
Assign maintenance
Assign ownership and synchronize updates when pages change. Choose a review cadence appropriate to change frequency; a quarterly check or a 3-week rewrite-loop review can be Tenten planning options, not verified client results or ten-minute guarantees. Inspect verified request logs if available.
- →Transport reference: curl -sI https://yourdomain/llms.txt checks headers. Also fetch and inspect the body and relevant linked responses.
- →Treat a model-summary error as an investigation item. Giving a file to ChatGPT or Claude is not proof that either discovers or prioritizes it unaided.
- →Review intended access for OAI-SearchBot, Claude-SearchBot, and PerplexityBot separately from GPTBot, ClaudeBot, and Google-Extended training/grounding controls. Check user-triggered fetcher exceptions; llms.txt requires no universal training opt-in.
One last thing
llms.txt is an optional engineering choice, not the cheapest required GEO foundation. Maintain accurate content and intended access first; evaluate the file for a documented consumer rather than presumed ranking value or urgency.
Sources and Methodology
Specification: Jeremy Howard’s llmstxt.org proposal, published September 2024 and updated to v2 on 10 August 2026: https://llmstxt.org/ . Google’s AI optimization guide, updated 10 July 2026, says Google Search ignores llms.txt and it neither helps nor harms visibility or rankings: https://developers.google.com/search/docs/fundamentals/ai-optimization-guide . The 15 June clarification is recorded at https://developers.google.com/search/updates . This does not establish other providers’ behavior. Prior training/search/user-trigger distinctions follow official documentation including https://developers.openai.com/api/docs/bots . Deployment checks and review cadences are Tenten recommendations, not a client-frequency dataset, measured ROI, fixed completion time, or universal first-stop rule. Reference URLs geo.tenten.co/llms.txt and /pricing.md need current response and source comparison; exact live counts are not asserted. Substantive review: 5 October 2026.
© 2026 Tenten GEO · Feel free to cite this guide; please credit the source.
Discuss your AI search priorities
Request a 30-minute benchmark interpretationSend a booking request. We’ll email you to confirm a time within 24 hours.