When most people block AI crawlers in robots.txt, what they are actually doing is not protecting the content, but removing themselves from the answers to ChatGPT, Perplexity, and Claude. The reason is that OpenAI, Anthropic, and Perplexity do not send out a single crawler, but two or three each: one crawls your content to train the model, and the other retrieves your page in real time and marks the source link in the answer. What you usually want to block is the former, and what you definitely want to keep is the latter. But just writing a set of User-agent: * and Disallow: / will shut out both.
Training crawlers and retrieval crawlers are two different things
Taking OpenAI as an example, GPTBot is responsible for capturing content into the corpus for future model training; OAI-SearchBot is responsible for putting your page into the search index of ChatGPT, allowing you to be quoted and attached with links in real-time answers; ChatGPT-User is only activated when the user requests to visit a certain URL during the conversation. This means that when you block GPTBot, it only affects "whether the content will be used for training" and does not affect at all whether ChatGPT can reference you at the moment. Anthropic and Perplexity have the same division of labor logic, but have different names. If you confuse this layer, you will block what should be allowed and miss what you want to block.
- "GPTBot" (OpenAI): Fetches content for training base models; blocking it will not make you disappear from ChatGPT searches.
- "OAI-SearchBot" (OpenAI): Indexes ChatGPT search and instant answers, and brings source citations; if you want to be cited, you must let it go.
- "ChatGPT-User" (OpenAI): Triggered when the user actively requests to visit a certain URL during the conversation.
- "ClaudeBot" (Anthropic): Capture content for training and model use.
- "Claude-SearchBot" and "Claude-User" (Anthropic): are respectively responsible for the search index and the retrieval triggered by the user.
- "PerplexityBot" (Perplexity): Indexes the Perplexity answer engine; blocking it is equivalent to exiting the Perplexity reference.
- "Perplexity-User" (Perplexity): Fetched immediately when the user asks the question. It has been accused of not fully complying with robots.txt.
- Other common objects: "Google-Extended" (not an actual crawler, but a control item that determines whether the content is used for Gemini training and grounding), "CCBot" (Common Crawl, the source of training corpus for many models), "Bytespider" (ByteDance, fierce crawling).
The basic rules of robots.txt, don’t rush to post the settings yet
robots.txt must be placed in the root directory of the domain. The URL is fixed to your domain followed by /robots.txt; and each host has its own copy - blog.example.com and www.example.com are two different hosts. The rules of the former cannot control the latter. http and https and different ports are also considered different sources. The syntax has only three keywords: User-agent specifies the object, Disallow blocks the path, and Allow makes exceptions. When Allow and Disallow conflict, mainstream engines adopt "longest path first", so you can use Allow to open a small door in a large-scale blockade.
- robots.txt is a "request" not a "firewall": well-behaved crawlers (GPTBot, ClaudeBot, Googlebot) will comply, but malicious or unruly crawlers can ignore it. If you really want to force blocking, you have to rely on WAF, firewall rules or verification mechanisms at the server layer.
- Blocking crawling does not mean deleting the existing data: robots.txt only affects "whether it can be crawled later", and the content that has been included in the training data will not disappear.
- The user-agent name must be precise: the comparison is not case-sensitive, but if it is spelled incorrectly (for example, written as GPT-Bot or Perplexity Bot), it means that it is not set, and each set of rules will only take effect against the matched User-agent.
Scenario 1: Allow to be quoted, refuse to be trained (default for most B2B SaaS)
If you want to appear in AI answers and be marked as a source, and do not want the content to be used for free to train next-generation models, then allow search crawlers and block training crawlers. This is the default configuration we use for most of our B2B clients: visibility first, training authorization optional. In practice, each crawler is written as an independent User-agent group, each with its own rules; writing a clear Allow: / to the search crawler can clearly express the intention and override any old settings of site-wide blocking.
- User-agent: GPTBot → Disallow: / (train crawler, block)
- User-agent: ClaudeBot → Disallow: / (train crawler, block)
- User-agent: CCBot → Disallow: / (Common Crawl training corpus, blocked)
- User-agent: OAI-SearchBot → Allow: / (ChatGPT search reference, release)
- User-agent: PerplexityBot → Allow: / (Perplexity reference, release)
- User-agent: Claude-SearchBot → Allow: / (Claude searches for references and allows)
- If you don’t want to enter Gemini training, you can add User-agent: Google-Extended → Disallow: / (does not affect Google search ranking)
Scenario 2: Not even a reference is left, and AI access is completely blocked.
Content behind paywalls, compliance-sensitive information, or materials still in the licensing negotiation stage may need to be blocked from even retrieval. The method is to list all known AI User-agents one by one and set Disallow: / for each. But be aware of the cost: after blocking, your references in AI answers will return to zero, which is equivalent to voluntarily withdrawing from this growing traffic source. This is usually a defensive, temporary setting and should not be considered the default for B2B marketing websites.
- OpenAI: Set Disallow: / for GPTBot, OAI-SearchBot, and ChatGPT-User respectively.
- Anthropic: Set Disallow: / for ClaudeBot, Claude-SearchBot, and Claude-User respectively.
- Perplexity: Set Disallow: / for each of PerplexityBot and Perplexity-User
- Optional additional gearing: Google-Extended, Applebot-Extended, Bytespider, CCBot respectively set Disallow: /
Don’t let the “one-click block all AI” boilerplate harm your GEO
There are many templates circulating on the Internet that "paste this paragraph into robots.txt to block AI". The problem is that they all set Disallow: / to all AI User-agents, blocking the search crawlers responsible for referencing you. It turns out that you thought you were protecting content, but you were actually actively opting out of ChatGPT search, Perplexity, and Claude's source list. For B2B brands who want to be referenced by AI, this is almost always counterproductive. Before posting any template, make sure it blocks training or retrieval.
After setting, how to confirm that it really takes effect?
- Direct check: open your domain in the browser and add /robots.txt, confirm that the file exists, is placed in the root directory, and the group is not misspelled.
- Read the server log: Search for GPTBot, ClaudeBot, and PerplexityBot in the User-agent field to see if the blocked ones return 403, and if the allowed ones capture the page normally.
- Authenticity verification: The User-agent of an AI crawler can be forged. OpenAI and Anthropic both publish official IP segments and use reverse DNS and forward DNS cross-confirmation to avoid being deceived by fake crawlers.
- Simulate a request: Use curl to pull the page with the corresponding User-agent, and check whether the robots.txt rules are allowed or blocked as expected.
- Review regularly: Each company will add or rename User-agent (for example, OAI-SearchBot was later separated from GPTBot), check it every quarter, and do not let the settings expire.
robots.txt is not a safe, it is a sign at the door. It determines which kind of AI you welcome in and which kind you keep out; and in the GEO world, keeping search crawlers out is equivalent to keeping business out.— Tenten GEO Consulting Team
Setting up robots.txt is just the first step in GEO's technical foundation. What really determines whether you can be cited by AI includes structured information, llms.txt, whether the content can be cleanly extracted, and the actual visibility of the brand in various engines. If you want to know whether you are being referenced by AI or blocked by your own robots.txt, you can make an appointment for a 30-minute GEO diagnosis. We will use your actual domain to run the crawler access and reference status, and tell you where the gap is on the spot.
