Claude
Claude web crawlers: a site owner’s guide
Learn what ClaudeBot, Claude-SearchBot, and Claude-User do, how Anthropic documents robots.txt controls, and how to check crawler access.
Takeaways
- Keep access checks separate from indexing, citation frequency and business results. An accessible page does not establish that an engine retrieved or cited it.
- Choose a policy for each crawler.
Claude-SearchBot,Claude-UserandClaudeBotserve different purposes. Record which uses you allow before changing their rules.[1] - Check every relevant hostname. Fetch the live robots.txt file at each subdomain root and confirm the intended crawler groups. A rule on one hostname is not a check of another.[1]
- Protect private content with access controls. Robots.txt expresses crawling preferences; it does not prevent someone from requesting a confidential page.[2]
- Investigate retrieval failures through the delivery path. Compare robots.txt, challenge behavior, response status and returned HTML for the affected URL. Anthropic says its bots do not bypass CAPTCHAs, so an allowed rule cannot solve a human challenge.[1]
The three crawler roles
| User agent | Anthropic’s documented use | Effect of blocking it, according to Anthropic |
|---|---|---|
Claude-SearchBot | Navigates the web to improve search result quality | May reduce the site’s visibility and accuracy in Claude search results. |
Claude-User | Retrieves web content in response to a user’s request | Prevents Claude from retrieving the site in response to a user query. |
ClaudeBot | Collects web content that may contribute to model development | Signals that future materials should be excluded from Anthropic’s model-training datasets. |
These descriptions come from Anthropic’s current crawler guidance. The wording describes intended roles and the stated effects of blocking; it does not promise that any page will be cited or that every fetch will use a listed bot.[1]
Set a clear robots.txt policy
To allow search crawling and user-directed retrieval while expressing a training opt-out, a site might use:
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: ClaudeBot
Disallow: /This example permits search crawling and user-directed retrieval. Review each permission against your legal, licensing, privacy and product requirements. Anthropic says its bots honor robots.txt and that site owners should place the file at the root of each subdomain they want to control.[1] Robots.txt communicates crawler preferences and does not provide access control for private content.[2]
Anthropic documents support for the non-standard Crawl-delay directive. If you choose to use it, add it to the relevant crawler group without removing that group's access rules. Anthropic says IP-based blocking is not a persistent substitute for a robots.txt opt-out, and it currently does not publish IP ranges for its bots.[1]
Check the whole delivery path
After reviewing robots.txt, inspect the infrastructure that can affect a request:
- Content delivery network (CDN) and web application firewall (WAF) rules.
- Bot mitigation, CAPTCHA, and JavaScript challenges.
- Authentication and region restrictions.
- Rate limits and responses such as 403 or 429.
- The actual HTML returned for important pages.
Anthropic says its bots honor anti-circumvention technologies and do not attempt to bypass CAPTCHAs. If the site requires a human challenge, an allowed robots.txt rule will not make the protected content available to the crawler.[1]
A test request that sets a Claude crawler user-agent string is a useful first check of server behavior. It does not verify Anthropic’s network path or prove that every edge rule allows the crawler. Treat it as a local diagnostic and confirm with your hosting and security logs.
A safe validation sequence
- Decide separately whether to permit search, user-directed retrieval, and training-related crawling.
- Edit the top-level robots.txt file for each relevant hostname.
- Fetch the live file and check the exact crawler group.
- Review CDN, WAF, challenge, and rate-limit behavior.
- Inspect server logs after a crawler request, where available.
- Recheck the policy when Anthropic updates its crawler documentation.
For each hostname, record the intended permissions, live robots.txt rules and delivery failures. Label manually supplied user-agent requests as tests and crawler identities in logs as unverified unless independently authenticated. A successful test does not confirm that Anthropic retrieved the page.
For ChatGPT Search access, see ChatGPT Search crawler access: a publisher’s guide. For Google Search eligibility, see Google AI Overviews and AI Mode: a site owner’s guide.
Sources
- Does Anthropic crawl data from the web, and how can site owners block the crawler?. Anthropic Help Center.
- Robots Exclusion Protocol. Internet Engineering Task Force.
Last verified October 17, 2026.