Blocking all AI crawlers with one click sounds great, but it actually kicks yourself out of AI answers. Because the crawler that gets you cited by ChatGPT and Perplexity, and the one that uses your content to train the model, are often two different robots — and Cloudflare’s “Block AI Bots” switch will turn off both of them together.
The first time many teams realize that AI is capturing their own sites is when they see an inexplicable increase in server traffic, or when legal counsel jumps out and asks, "Has our content been used for training?" The reflexive action is to open Cloudflare and look for an option that blocks all AI crawlers. Cloudflare does provide this switch, and it only takes three seconds to operate. The problem is, if you are running B2B SaaS and want to be recommended by the AI engine, this block is equivalent to cutting off the entrance to the cash flow. This article talks about how to block accurately - block what you don't want and let go of what you want to be quoted.
Let’s make a clear distinction first: there are three types of AI crawlers, and they cannot be treated equally.
All robots called "AI crawlers" are actually doing three different things. Only by understanding these three categories can you know which one should be blocked and which one should not be touched.
- Training crawler: crawl your content back and feed it to model training. Representatives include OpenAI’s GPTBot, Anthropic’s ClaudeBot, byte-hopping Bytespider, and Common Crawl’s CCBot. Blocking them will not affect your citations.
- Search crawler: When users ask questions in ChatGPT or Perplexity, they will immediately crawl your page, and then write the content into the answer and attach the source. Representatives include OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, and Claude-User. This category is the key to sending you to the AI answer. If you block it, it will disappear.
- Search index crawlers: Googlebot vs. Bingbot. They don't just do traditional searches — Google AI Overviews relies on Googlebot, and ChatGPT and Copilot rely on Bing for some results. Block them by mistake, and SEO and AI visibility will drop together.
The most counter-intuitive point here: blocking the training crawler will not make you disappear from the AI answers. Whether the model refers to you depends on the content currently captured by the retrieval and search crawlers. It is different from whether it used you for training a few months ago. So you can refuse to be used for training while striving to be cited immediately - provided you can tell who is who.
Step 1: Find out who’s after you at Cloudflare
Before you start blocking, read the information first. The "AI Crawl Control" (formerly AI Audit) in the Cloudflare backend will list the number of recent crawls for each AI crawler, which paths have been crawled, and whether it is currently allowed or blocked. It’s usually a surprise when you open it for the first time—the request volume of Bytespider or GPTBot may be even worse than that of Googlebot. This list is the basis for all your subsequent decisions: first see clearly which robots are really consuming your server resources, and which retrieval crawlers have not actually been here at all, which means that you have not been seen in that engine.
Step 2: Classify release and blockade, don’t use that universal switch
Cloudflare gives you three tools with different strengths, and a mix of them works best. The thickest thing is the one-click switch of "Block AI Bots" in the security settings, which is convenient, but it is a nuclear bomb and will blow up the retrieval crawler together. Unless you don't care about AI visibility at all, don't touch it. Moderately powerful is hosted robots.txt, which is suitable for dealing with trained crawlers that are polite and follow the rules. The most accurate one is WAF custom rules, which can release or block a single robot. This is the main force.
- In the WAF custom rules, create a blocking rule, compare the user agent of training crawlers such as Bytespider, CCBot, GPTBot, ClaudeBot, etc., and set the action to block or return 403.
- Create another Skip rule to clearly allow OAI-SearchBot, PerplexityBot, Perplexity-User, ChatGPT-User, Googlebot, and Bingbot, and rank them in front of the blocking rules to ensure that retrieval is always smooth.
- Use Cloudflare's "verified bot" field to compare, rather than just comparing the user agent string, because user agents can be forged and someone can impersonate Googlebot. Verified bots are verified by Cloudflare using reverse DNS and signatures to block imposters.
- If you want to go one step further, Cloudflare's Pay Per Crawl can return HTTP 402 to the trained crawler and require payment before being caught, turning being prostituted into an income or a bargaining chip.
What should be allowed and what should be blocked?
If your goal is to be recommended by an AI engine, and most B2B SaaS is, the default policy is clear: allow all retrieval and search crawlers, and selectively block training crawlers. Special reminder of two pitfalls that are easy to step on. First, Google-Extended and Googlebot are two different things. The former only controls whether your content should be used for Gemini training and grounding. Blocking it will not make you disappear from AI Overviews; but if you block even Googlebot, then the entire Google ecosystem will really not be able to see you. Second, don’t block Bing, because some of the results of ChatGPT and Copilot eat into Bing’s index, and blocking Bingbot will also affect them.
Step 3: Verify to confirm there is no accidental injury
Once a rule is live, don’t assume it will work the way you want it to. Go back to the AI Crawl Control and WAF event logs and confirm one by one: whether the training crawler got 403, and whether the retrieval type and Googlebot still got 200 stably. The most common accident is that a blocking rule that is too broad sweeps Googlebot in. You think it is just blocking AI, but the natural ranking starts to collapse after two weeks. Use Google Search Console to check the URL, or directly look at the server response code, and compare it for a week to make sure that the release list is really smooth before calling it a day.
Blocking AI crawlers is easy, but the hard part is to block only the ones that should be blocked, and at the same time, roll out the red carpet to welcome in the ones that send you answers.— Tenten GEO
Blocking out crawlers is just defense, being cited is the purpose.
The purpose of managing crawler traffic well is to prevent the server from being used for nothing or bringing down the server. But allowing retrieval crawlers does not mean that AI will cite you. They must be able to catch you, understand it, and think you are worthy of citation. This is the real battleground for GEO: content structure, entity markup, brand consistency across platforms. Managing Cloudflare well is opening the door. Whether you can be invited into the answer is another battle. If you want to know whether the crawlers that are not cited or released in the six major AI engines are in vain, you can make an appointment for a 30-minute GEO diagnosis. We will use your actual questions to show you where the gaps are.
