AI crawlers
AI crawler access and referral traffic for documentation
Published crawl-to-referral ratios vary by reporting period, industry filter and provider. Compare those methods before using a benchmark to set crawler access rules for your documentation. The calculator below helps you measure your own site.
Takeaways
- Use matching provider totals and reporting periods from your own site before changing crawler access. Investigate request units, industry filters and referral attribution; a crawl-to-referral ratio does not measure answer accuracy or customers acquired.
- Cloudflare defines crawl-to-refer as HTML crawl requests divided by referred HTML page requests for the same provider.[1]
- Cloudflare's first-week-of-August 2025 snapshot reports Anthropic near 50,000 to 1 across industries and 8,800 to 1 for Computer & Electronics.[1]
- Those are provider-level ratios. They should not be described as referrals sent by an individual training bot.
- Search indexing, training and user-requested retrieval have different access controls. Restricting search crawling can reduce visibility.[2][3]
What is the crawl-to-referral ratio?
It compares a provider's crawl requests with referred page requests over the same reporting period.
Cloudflare introduced the metric on Radar in July 2025. It compares HTML page requests from a platform's crawler against HTML page requests referred by that platform, which Cloudflare treats as human traffic.[1] A ratio of 5 to 1 means five crawl requests for each referred page request. Keep those units consistent when comparing sites or reporting periods.
Published crawl-to-referral benchmarks
Cloudflare's August 2025 analysis provides a comparison using one reporting window. Across the first week of August, its unfiltered view reports Anthropic near 50,000 crawl requests per referral, OpenAI at 887 and Perplexity at 118.[1]
Provider crawl-to-referral ratios, August 2025
The figure uses a logarithmic scale. A provider's ratio combines crawl and referral measurements; it does not show that its training crawler sends visits.
Comparing reporting periods and industry filters
In Cloudflare's Computer & Electronics category for the same period, the ratios were lower: Anthropic 8,800, OpenAI 401.7 and Perplexity 88.[1] That category includes many sites besides developer documentation. Neither aggregate predicts your own ratio.
Do not combine these figures with another month's data as if the sites, filters and measurement rules were unchanged. Referral headers can also be absent, so analytics can miss some attributable visits. Record the period, selected industries and definition of a referral alongside the result.
Training, search and user-requested crawlers
OpenAI and Anthropic each split their crawling into three jobs: training, search indexing and user-triggered fetches.
| Job | OpenAI | Anthropic | Effect of restricting access |
|---|---|---|---|
| Model training | GPTBot | ClaudeBot | Signals exclusion from future training datasets |
| Search index | OAI-SearchBot | Claude-SearchBot | Search indexing or retrieval may be limited |
| User request | ChatGPT-User | Claude-User | User-requested retrieval may be limited |
OpenAI says sites that opt out of OAI-SearchBot will not appear in ChatGPT search answers, although navigational links can still appear. It also says ChatGPT-User does not decide what appears in search, and robots.txt rules may not apply to user-initiated requests.[2] Anthropic says blocking Claude-SearchBot prevents indexing for search and may reduce visibility in search results. Blocking Claude-User may reduce visibility in user-directed web search.[3] Anthropic also supports the non-standard Crawl-delay directive.[3]
For a deeper look at each vendor's bots, read ChatGPT Search crawler access: a publisher's guide and Claude web crawlers: a site owner's guide.
Cloudflare controls that shipped in September 2026
Cloudflare's September 15 release distinguishes blocking a crawler from expressing a preference against training. Its Disallow AI Training setting allows supporting mixed-use crawlers to continue search indexing. Blocking those crawlers can block search access too.[4]
For new ad-supported domains, Cloudflare recommends allowing Search, disallowing Training and blocking Agent requests on pages with ads. Existing domains migrate according to their earlier settings. This is more specific than a blanket default block on every unconfigured site.[4]
If your documentation uses Cloudflare, review the actual domain settings alongside robots.txt and firewall rules. Confirm the effect of a change with request logs.
Calculate your own crawl-to-referral ratio
Use server logs or content delivery network (CDN) analytics for crawl requests, and your analytics platform for referred page requests. Group both by provider and use the same reporting period. The calculator processes entries locally in your browser.
Free tool
Crawl-to-referral calculator
Enter provider-level HTML crawl requests and referred HTML page requests for the same date range. The first rows are example numbers; replace them with yours. Nothing you type leaves your browser.
| Provider | Crawl requests | Referred page requests | Ratio | Remove |
|---|---|---|---|---|
| 4.7 : 1 | ||||
| 284 : 1 | ||||
| 1,689 : 1 |
The example rows are illustrative, not benchmark data. Aggregate crawl requests by provider, then compare them with that provider's referred page requests. Sessions and page requests are different units; a requests-per-session result is not comparable with Cloudflare's HTML-request ratio. Training bots do not themselves refer visitors.
Where to find the numbers
- Crawls: filter access logs or CDN bot analytics by verified crawler, then aggregate requests by provider. Count HTML page requests only.
- Referrals: count HTML page requests attributed to chatgpt.com, claude.ai, perplexity.ai or google.com for the same period. If you use sessions instead, label the result requests per session; it is not the same metric as Cloudflare's HTML-request ratio.
- Verify the bots. Anyone can fake a user agent. Use a vendor's published verification method where available, or your CDN's verified-bot classification. A user-agent string alone is not proof of identity.
Setting crawler access rules for documentation
Set crawler policies by purpose: model training, search indexing or user-requested retrieval. Access to documentation can support accurate answers about your application programming interface (API), but allowing a crawler does not guarantee retrieval, citation or referral traffic. Training access is a separate licensing and infrastructure decision.
Evaluate these controls for your documentation:
- Review search-bot access. OAI-SearchBot, Claude-SearchBot, Googlebot and PerplexityBot.
- Review user-requested retrieval. ChatGPT-User and Claude-User. These fire when a person asks about your page.
- Document the training policy. Consider licensing, content rights and server load separately from search visibility.
- Manage request volume. Use supported crawl-delay settings or tested CDN rate limits where appropriate. Confirm which crawlers honor the setting and check that legitimate retrieval still works.
- Measure quarterly. Recompute your ratios. Check whether citations and referrals move after each change.
A citation without a referral is a visibility observation, not a measured visit or conversion. For a separate analysis of citations and brand mentions, read Sources cited in AI answers about developer tools.
Sources
- AI crawler traffic by purpose and industry. Cloudflare Blog.
- Overview of OpenAI Crawlers. OpenAI Developer Documentation.
- Does Anthropic crawl data from the web, and how can site owners block the crawler?. Anthropic Help Center.
- Have it both ways: stay discoverable in search while disallowing AI training. Cloudflare, September 15, 2026.
Last verified October 22, 2026.