Block Training, Allow Retrieval

CH.01

The Cost of Blocking Everything: You're Not Blocking Crawlers, You're Blocking Referrals

A blanket block can restrict search crawlers along with training crawlers. Identify each provider’s training, search, and user-triggered fetchers before choosing what to allow. Their controls and exceptions differ.

Most people block AI crawlers because they 'don't want to be used for training.' The reasoning is sound; the execution is where it falls apart. They write User-agent: * / Disallow: / in robots.txt, or flip on Cloudflare's one-click 'Block AI Bots,' thinking it only stops training. In reality, you've also blocked the 'retrieval crawlers' that power the cited answers in ChatGPT, Perplexity, and Claude - the very ones that send traffic back to your site.

Lay out the proportions and you'll see why a blanket block is pure waste. Cloudflare's May 2026 observations show that 51.8% of AI crawler requests are pure training, 35.7% are mixed-purpose, and only 9.3% are pure retrieval. The training traffic you actually want to block is more than half; the pure retrieval you must never block is under a tenth. Block everything with one rule and you're locking out the only guests who bring money back - that sub-10% - just to shoo away the half you didn't want.

Training and search access have separate controls. OpenAI separates GPTBot from OAI-SearchBot, and Anthropic separates ClaudeBot from Claude-SearchBot. Google-Extended controls Gemini training and some grounding uses without affecting Google Search inclusion or rankings. These distinctions do not establish a universal ranking or citation guarantee.

CH.02

Crawler Map: Check Each Purpose and Control

One provider can use different agents for training, search indexing, and user-triggered retrieval. Check the documented purpose and control for each name; a shared company name does not make their rules interchangeable.

GPTBot collects content that may be used for training; OAI-SearchBot supports ChatGPT search. ChatGPT-User handles certain user-triggered visits and does not determine Search inclusion. Anthropic distinguishes ClaudeBot, Claude-SearchBot, and Claude-User for training, search, and user requests. Search access supports discovery; it does not guarantee a source link in every answer.

CH.03

Precise Split-Bot Setup: Line-by-Line robots.txt Rules

Match the crawler’s product token to its robots.txt group. Under RFC 9309, matching groups are combined; the * group is the fallback when no product-token group matches. Within the applicable group, the longest matching path rule applies; an equivalent Allow takes precedence over Disallow.

A named group with Allow: / can allow a search crawler even when the fallback User-agent: * group has Disallow: /. Omitting a named group does not mean access is allowed: that crawler may inherit the wildcard restriction. Check the applicable group and path instead of assuming file order controls the result.

• →Training policy: use separate User-agent groups for GPTBot, ClaudeBot, and CCBot with Disallow: / if you want to restrict those crawlers. Treat Google-Extended separately: its Disallow also restricts covered grounding uses, not just training.
• →Search access: use User-agent: OAI-SearchBot with Allow: /, and assess Claude-SearchBot and PerplexityBot separately. ChatGPT-User is user-triggered and robots.txt may not apply; Perplexity-User generally ignores it. Anthropic states that its three agents honor robots.txt.
• →Search engines - allow them, no exceptions: keep Googlebot and Bingbot fully crawlable; don't lump them in with AI training crawlers
• →Check the wildcard fallback: User-agent: * with Disallow: / can block agents without a matching named group. Explicit matching groups are not automatically overridden by that wildcard. robots.txt is a crawl instruction, not enforced access control.

CH.04

CDN-Layer Control: The Wall That Actually Holds

Robots.txt is a request, the CDN is enforcement. If what you need is a 'guaranteed block' rather than a 'polite ask,' access control belongs at the host or CDN layer - returning a 403 on a user-agent rule, not just a line in robots.txt.

Why? Because robots.txt has no enforcement power. A crawler that spoofs its user-agent or simply ignores the rules can still scrape the directories you Disallowed. When your goal is something with real stakes, like 'don't get used for training,' a sign on the door isn't enough - you need the CDN/WAF to intercept by user-agent before the request ever reaches your content.

CH.05

Verify It Took Effect: Confirm Retrieval Crawlers Can Still Get In

Configured doesn't mean correct. The most dangerous failure of a split is 'silently blocking the wrong one' - you think you only blocked training, but you actually blocked the retrieval crawlers too, with no error anywhere. You have to verify actively, not assume.

Verification has only one goal: confirm that 'the training crawlers you wanted blocked are blocked, and the retrieval crawlers you wanted allowed flow freely.' Check both sides - checking only that training was blocked, without checking that retrieval was allowed, is verifying just half. The five steps below run in under half an hour for one pass.

CH.06

The Split-Bot Template: Copy, Paste, Done

You don't have to write it from scratch. The robots.txt template below already blocks training and allows retrieval for you - read it line by line, swap in your domain, and it's ready to ship.

How to use it: write the directives in the table below, in order, into the /robots.txt at your site's root. Every training crawler is its own block (one User-agent line, one Disallow line); retrieval crawlers get an explicit Allow; keep the Sitemap at the end. Remember to pair this with Chapter 4 - add the same 'block' crawlers to your CDN/WAF's forced interception rules.