When the AI engine reads your web page, the first step is to tear down the HTML: throw away the navigation bar, sidebar, cookie prompts, and tracking scripts, leaving only the text it determines as the main body. This teardown will go wrong, and you can't see where it goes wrong. Instead of perfecting the HTML semantic markup and betting on the model to guess the layout correctly every time, it’s better to just give it a copy of the Markdown you’ve compiled. For the same content, if layout noise is removed, the model does not need to guess, and the accuracy of the extracted quotes will be stabilized.
Why AI would rather read Markdown than your HTML
The key is the noise ratio. For a typical SaaS product page, the HTML source code may have six to seven thousand tokens, of which the real content only accounts for a small part. The rest are class names, embedded styles, SVG paths, and analysis scripts. After the model reads in, it must first filter these to find sentences that can be used to answer questions. The Markdown version reduces the content of the same paragraph to a fraction of the original size. The title is the title, the list is the list, and the key points are the key points. The structure itself carries semantic meaning, and the model leaves a lot of room for guessing.
This is especially important for answer engines. ChatGPT, Perplexity, and Google AI Overviews will extract quotable paragraphs from candidate pages when generating answers. The cleaner the page and the more self-sufficient the paragraphs, the higher the chance of being fully quoted. We have helped clients deal with a common situation: the article itself is well written, but it is wrapped in layers of divs and client-rendered components, and the model only captures half of the sentence. After switching to the Markdown endpoint, the same paragraph of content can be accessed in its entirety, and the reference will not be broken at strange locations.
/md endpoint: Give each page a machine-readable clone
The approach is to provide a corresponding Markdown version for each public page, and the URL rules must be predictable. There are two common ways: add .md after the original URL (for example, /blog/aeo-basics corresponds to /blog/aeo-basics.md), or add /md after the path. Both are fine, the key is that the rules are consistent and can be speculated. Next.js's App Router can be implemented with a Route Handler: create a route file, read the same content data as the HTML page, and output the Content-Type as text/markdown.
- Single source of data: The HTML page and the Markdown endpoint read the same content (MDX, database or content object). Do not maintain two copies, otherwise they will be out of sync sooner or later.
- Put link rel=alternate type=text/markdown in the head of HTML to clearly tell the crawler where the machine-readable version is.
- Markdown retains only the main text: titles, paragraphs, lists, tables, code blocks and links, and removes navigation, footers, related articles and marketing banners.
- Keep necessary source information: Use the front-matter at the beginning of the file or the first line H1 to bring out the title, original URL, and update date to facilitate model labeling of sources.
- Return the correct cache header so that the CDN can also serve this endpoint without having to go back to the origin every time.
Use content negotiation to make format switching happen automatically
In addition to fixed URLs, the same URL can also be returned in different formats depending on the identity of the requester. There are two basis for judgment: Accept header and User-Agent. When the request carries Accept: text/markdown, or the User-Agent belongs to a known AI crawler (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.), it is rewritten to the Markdown route in the middleware layer; general browsers still get HTML. People see the full page, the model gets clean text, and the URL remains unchanged from beginning to end.
Plain text backup: when even Markdown is too much
Some agents or crawlers do not parse Markdown syntax and only eat plain text; other situations, such as voice assistant source summaries, only require the most concise content. This is where it makes sense to have a plain-text backup. The method is the same as the Markdown endpoint, the difference is that it outputs text/plain: the title and paragraph are separated by newlines, and the link attaches the URL to the text in brackets. The advantage of layer-by-layer backup is that no matter what level the other party's parsing capabilities fall on, you can always get a readable version.
- llms.txt: Place /llms.txt in the root directory of the website. Use Markdown to list important pages and brief descriptions, which is equivalent to giving the model a navigation map.
- llms-full.txt: Concatenate the entire core content into a single file, making it easier for the model to read it all at once without having to crawl page by page.
- Single-page .md endpoint: a machine-readable clone of each article, each product page.
- Plain text fallback: minimal text/plain, for tools that don't parse Markdown.
How to confirm that AI can really read
Verify it after it goes live, don’t assume it will take effect automatically. The fastest way is to use curl to simulate a crawler request: bring Accept: text/markdown or set the User-Agent of the AI crawler to see if the returned Markdown is clean, the status code is 200, and the cache header is correct. Then check the server logs to confirm that crawlers such as GPTBot and ClaudeBot are indeed hitting your Markdown endpoint, instead of continuing to grab the noisy HTML. Tools like isitagentready can also quickly help you check the readability of a page for agents.
Whether it can be cited by AI depends on whether it can read your content cleanly. Giving it a copy of Markdown is equivalent to completely removing the step of "guessing the layout" from the process.— Tenten GEO
Start with one page instead of revamping the entire site at once
First select the pages with the highest traffic and those most wanted to be referenced by AI, add .md endpoints and plain text backup to them, measure the crawling behavior of the AI crawler for two to four weeks, and then decide whether to roll it out to the entire site. In this way, risks can be controlled and real logs can be used to convince the team. If you want to know first how readable your pages are for AI and where the gaps are, Tenten GEO’s 30-day GEO audit will take stock of these machine readability issues page by page. If you’d like to chat about your situation, go to /contact to schedule a 30-minute GEO diagnosis.
