Breakdown · Trends & Insight

AI Crawl and Referral Economics: Measure Before Choosing Controls

Compare verified bot purposes, actual crawl costs and attributed referrals under the same period and filters. HTML requests are not unique pages or visitors. Training, search and user-triggered access differ; neither allowing nor blocking guarantees an outcome.

v1.1 · October 2026 · Tenten GEO Research

CH.01

What Crawl-to-Refer Measures

A platform-level request ratio can provide context, but does not settle the value or access policy of your site.

On a warm dark background, a broad luminous cream and lavender field at left contains many rectangular strips and small human silhouettes; the strips narrow toward an isolated glowing violet orb at far right. — Conceptual comparison of content access and human traffic, not an observed 10,000-to-one result.

Crawling, showing a link and a person visiting are separate events. Count each with its actual unit. A request total can include repeated requests; a displayed citation does not prove a visit or sale.

Cloudflare’s August 28, 2025 article defines the ratio as platform-associated crawler HTML requests divided by human HTML requests referred by that platform across its customer base. It is not unique pages per unique visitor, a per-bot referral attribution experiment or your site’s ROI.

Chapter takeaway

Use the ratio as context, not a verdict. For a site decision, gather its own costs and observed referrals with explicit scope.

CH.02

Compare Like with Like

A meaningful trend needs the same time window, units and filters. The earlier Q1/June ratios lack verified matching snapshots here and are not presented as established 2026 comparisons.

Comparison check	Record	Limit	Evidence
Time window	Exact dates and data source	Publication date is not collection period	Original snapshot or export
Platform and bot	Provider, verified user-agent and declared purpose	Platform referrals cannot automatically be assigned to one training bot	Provider docs and identified request logs
Numerator and denominator	Crawler HTML requests / attributed human HTML requests	Not unique pages, people, citation probability or revenue	Request definitions and attribution coverage
Filters and coverage	Same site/industry, purpose and inclusion rules	Changed filters or missing referrals can change the ratio	Saved settings and scope

Four evidence checks, not a vendor ranking or an unverified improvement trend.

Cloudflare’s 2025 article separates Training, Search, User action and Undeclared. These categories describe known purposes, not every request’s proven downstream use. This guide does not reuse the earlier unverified breakdown or infer an improvement from unmatched periods.

Chapter takeaway

Save the source and settings before comparing periods. Leave unmatched values unverified rather than treating them as current policy evidence.

CH.03

Training, Search and User-Triggered Access

Choose controls for an identified purpose. Search indexing is not limited to the moment a user asks; user-triggered access has separate rules.

GPTBot supports model training; OAI-SearchBot supports search; ChatGPT-User acts for user requests and is not an automatic search crawler. Provider descriptions and robots exceptions differ. A role does not guarantee a citation, click, model inclusion or complete exclusion.

Dimension	Training access	Search / user-triggered access
Examples	GPTBot; ClaudeBot under current provider docs	Search: OAI-SearchBot; user action: ChatGPT-User
Purpose	Collect content for possible model training	Build search access or act on a user request; separate roles
Links and referrals	Training collection alone does not establish referrals	Search access can support discovery, not guaranteed links or visits
Controls and limits	Check provider identity, robots scope and enforcement	Check search eligibility and user-action exceptions separately

Purpose-specific controls; not a two-bot causal referral experiment.

robots.txt directives apply within their User-agent groups. A GPTBot-only Disallow does not automatically block OAI-SearchBot. Broad wildcard rules, CDN rules and user-triggered exceptions need separate review. robots.txt is not authentication or proof all clients complied.

Chapter takeaway

Separate purposes before making an access decision. Allowing a bot does not promise traffic; restricting one bot does not prove invisibility across a whole platform.

CH.04

Measure Costs and Business Evidence Separately

Rights, privacy, access policy and operational cost matter alongside referrals. There is no verified universal cost-versus-value result for B2B here.

Use your server/CDN logs and actual billing to estimate request, bandwidth and compute cost. Shared infrastructure allocation and bot identity uncertainty should be stated; a global request ratio does not prove pure loss on your site.

Measure attributed visits and business outcomes using your own definitions. A citation or brand mention can be observed, but its monetary value cannot be inferred simply from average deal size. Unobserved influence and missing referrals remain unknown.

Step one

Record actual access cost

Choose a fixed period, identify requests and estimate actual incremental cost. State uncertainty; do not assume every B2B bill is negligible.

Step two

Separate citations from business outcomes

Use fixed questions and record product, mode, region, language, date, repetitions and denominator. Keep citations, mentions, visits and attributable qualified outcomes separate.

Step three

Review policy with the evidence

Compare actual cost and attributable value where available, together with rights and business policy. Use bounded scenarios for uncertainty, not a universal allow/block formula.

Chapter takeaway

Use actual observations and costs. Unknown citation value is not zero, and an observed citation is not an attributed sale.

CH.05

Four Controls to Evaluate

Open access, selective restrictions, negotiated paid access and broad restrictions have different requirements. None is a universal B2B winner.

Path	How	Who it fits	The math
Open access	Keep selected public paths accessible	Sites whose rights and policy permit that access	Monitor actual cost and outcomes; access does not promise referrals
Selective restrictions	Purpose-specific robots and appropriate enforced controls	Sites with a documented purpose-specific policy	Verify identity, scope and compliance; not a guaranteed cost saving
Paid access / paywall	Use a supported product or negotiated agreement if available	Content owners with a feasible access contract	Check availability, counterparties and terms; no assumed payment or revenue
Broad restrictions	Define wildcard or enforced access restrictions deliberately	Sites whose rights, privacy or policy require this scope	Review discovery/access implications; not proof of universal invisibility

Four existing controls with conditions, not ranked returns.

Selective and broad controls affect different requests, depending on identity and enforcement. Check the actual rule groups and product paths. A one-line difference is not evidence of guaranteed referrals versus complete disappearance.

Chapter takeaway

Choose a documented, reversible policy and monitor it. Paid access requires an actual supported arrangement; no option guarantees income.

CH.06

Tenten’s Policy Review Worklist

This is an authored worklist, not a default robots file to copy blindly. Confirm your current policy and the provider’s purpose before adjusting controls.

Setting one

Verify rules by purpose

Identify training, search and user action; inspect actual robots groups and enforced controls. Check provider exceptions rather than treating every bot identically. This guide changes no live policy.

Setting two

Keep useful public information accessible

Publish accurate readable content on intended public paths. Supported schema must match the page; llms.txt is optional and not required by Google Search. Neither format guarantees citations.

Setting three

Review cost and observed outcomes

Tenten recommends a periodic review with fixed conditions, request/cost logs and separate referral/business measures. Record unverified or not-applicable items; do not infer success from five green checks.

Check item	The state it should be in
Is robots.txt blocking everything in one shot?	Document exact User-agent groups, wildcard scope and enforced controls
OAI-SearchBot / PerplexityBot	Decide search access under your policy; not a referral guarantee
GPTBot training / CCBot	Decide training access from rights/policy and evidence, not an automatic block
Citation-rate tracking	Fixed observations with product/mode/region/date/repetitions and denominator
Crawl cost vs. citation value	Actual costs and attributed outcomes, with missing data and uncertainty

Five review items; unknown or not applicable is not automatic failure.

One last thing

Make access decisions with purpose, rights, measured costs and observed outcomes. Keep uncertainties visible; no generic ratio or crawler setting guarantees business value.

Sources & Methodology

Substantively revised October 5, 2026 in Taipei. Cloudflare’s August 28, 2025 request-ratio definition and purpose categories are scoped; unverified 2026 Q1/June ratios and May percentages are retired, not declared false. Provider bot roles use official docs. Cost/measurement/control worklists are Tenten recommendations, not client outcomes or proven ROI. https://blog.cloudflare.com/ai-crawler-traffic-by-purpose-and-industry/ https://developers.openai.com/api/docs/bots https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler https://developers.google.com/search/docs/fundamentals/ai-optimization-guide https://www.rfc-editor.org/rfc/rfc9309

Discuss your AI search priorities

Request a 30-minute benchmark interpretation

Send a booking request. We’ll email you to confirm a time within 24 hours.