Many people write a line of Disallow in robots.txt to block GPTBot, thinking that this will prevent it from being referenced by ChatGPT. In the end, neither thing happens. GPTBot is only responsible for grabbing data to train the model. What really determines whether you will appear in the ChatGPT answers is another crawler called OAI-SearchBot. Two crawlers with different tasks from the same company use different User-Agents. First recognize what each name on the list is doing so that the permissions and blocks you set in robots.txt will not be reversed.
Let’s make a clear distinction first: there are actually three types of AI crawlers
- Training crawler: grabs your content and incorporates it into the training corpus of the model. It is usually processed offline and is not directly related to the answer that the user is currently seeing. Representatives include GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Meta-ExternalAgent, Bytespider (ByteDance).
- Search/index crawler: Establish an instant searchable index for the AI answer engine, which directly determines whether your page will be used as a reference source. Representatives include OAI-SearchBot (ChatGPT search), PerplexityBot, Googlebot (AI Overviews follows its existing index), Applebot (Siri and Apple Intelligence).
- User-triggered crawler: crawls when the user pastes your URL in the conversation or requests instant verification. It usually only reads one page at a time and does not enter the training set. Representatives include ChatGPT-User, Perplexity-User, and Claude-User.
If you want to be cited by AI, the last two categories you should not block are the latter two categories. If the search type cannot enter, you are not on the candidate source list; if the user-triggered type is blocked, others will actively throw your link to the AI, and it will not be able to read the content. The training type is the target that can be considered according to the brand strategy: if you are worried that the content will be used for free for training, block it; if you want to increase the probability of appearing in the underlying corpus of the model, let it go. Confusing these three layers is the root cause of most robots.txt being written in the wrong direction.
OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User
- GPTBot: For training purposes, the User-Agent contains "GPTBot/1.2", and the robots.txt token is "GPTBot". OpenAI also publishes the crawler's IP segment (JSON file), which can be used to verify whether the visit is genuine.
- OAI-SearchBot: Responsible for including your page into ChatGPT's search and citation sources, the scepter is "OAI-SearchBot". Being blocked is equivalent to voluntarily withdrawing from the visibility in ChatGPT, but it is the one that most people mistakenly block.
- ChatGPT-User: Triggered when the user requests to browse or verify your URL in the conversation. The token is "ChatGPT-User". It is a real-time, single-page crawl and is not included in the training.
Google-Extended is not a crawler, it is a switch
Google-Extended is often included in the "AI crawler list", but it is not a crawler that will appear in your server log at all, but a product token in robots.txt. Releasing or blocking it only affects whether Google uses your content for Gemini training and partial generation, and does not affect Googlebot's general indexing at all. Therefore, blocking Google-Extended will not make you drop out of Google search, and it usually will not make you disappear from AI Overviews, because AI Overviews mainly uses Googlebot's existing index data. Apple's Applebot-Extended is the same design. It is the switch for Apple Intelligence training and is different from the Applebot responsible for general indexing.
PerplexityBot, and those unruly crawlers
To be referenced by Perplexity, both PerplexityBot (index) and Perplexity-User (user real-time query) must be allowed. But here is a reminder of reality: robots.txt is a gentleman’s agreement and has no force. In 2025, Cloudflare publicly named Perplexity, saying that after being blocked by robots.txt, it switched to using an undeclared User-Agent disguised as a general browser to continue crawling; Bytespider was also known for its high-frequency crawling method with almost no regard for speed in the past. This means that two things are true at the same time: the crawler you want to block may not be able to be blocked, and the source you want to release also needs a method to confirm whether it is the original deity.
Amazonbot, ClaudeBot, and other names you'll encounter in the log
- ClaudeBot: Anthropic’s training crawler. There are also Claude-User (user-triggered) and the older anthropotic-ai and Claude-Web scepters, which may appear in the log at the same time.
- Amazonbot: Used by Alexa and Amazon's AI services, User-Agent contains "Amazonbot/0.1".
- CCBot: Common Crawl. It is not an AI company itself, but the public corpus it captures is used by a large number of models, which is an indirect training pipeline and is most easily missed during training.
- Other common names: Meta-ExternalAgent (Meta training) and Meta-ExternalFetcher (user triggered), YouBot (You.com), DuckAssistBot (DuckDuckGo), cohere-ai (Cohere), MistralAI-User, Diffbot, Timpibot.
How to write robots.txt in 2026
There is no one-size-fits-all answer, but for most B2B websites that want to do GEO, a reasonable default is: release search and user-triggered types so that the AI engine can read and quote them; training types are determined individually based on content value and brand strategy. Below is an example that favors visibility.
- User-agent: OAI-SearchBot → Allow: / (preserve ChatGPT visibility)
- User-agent: PerplexityBot → Allow: / (enter Perplexity’s reference pool)
- User-agent: ChatGPT-User and Perplexity-User → Allow: / (don’t block the user’s active verification)
- User-agent: Googlebot → Allow: / (AI Overviews and general search share the same index, be sure to allow it)
- User-agent: GPTBot, CCBot, Google-Extended → Determined by policy (disallow: / if you don’t want to be used for training, this will not affect the visibility above)
Don’t bet everything on robots.txt
robots.txt can stop crawlers that obey the rules, but cannot stop those who are determined to bypass it. If you really want to control, you have to go to the next level: use reverse DNS plus forward confirmation, and compare the IP blocks officially announced by each company to verify the authenticity of the crawler; when facing disguise or high-frequency crawling, use WAF, Cloudflare's AI crawler management or rate limit instead of just writing an extra line in robots.txt. Equally important is monitoring: regularly check the server's access log to see which AI crawlers have actually visited and how frequently they have visited. This is your first-hand signal to judge your GEO exposure, which is more accurate than any prediction.
The list of AI crawlers changes every season, with new names popping up and scepters changing. Instead of memorizing each one, it is better to remember the classification logic: training, retrieval, and user triggering, each corresponding to different strategic consequences. If you are not sure which AI engines are currently crawling your site and which pages are missing that should be referenced, Tenten's GEO audit will spread out the crawler access records and reference status at once. If you want to find out your gaps first, you can make an appointment for a 30-minute GEO diagnosis (/contact).
