ChatGPT Search
ChatGPT Search crawler access: a publisher’s guide
Understand OpenAI’s published crawler roles and check the robots.txt, CDN, firewall, and analytics settings that affect ChatGPT Search access.
Takeaways
- Separate search access from training permission. Set
OAI-SearchBotandGPTBotrules independently. You can allow search crawling without allowing the training crawler.[1] - Check more than robots.txt. Review firewall challenges, denied responses and rate limits for important public pages. An allowed path can still be blocked by the site's delivery settings.[3]
- Label your test requests. A request with an
OAI-SearchBotuser-agent string tests the response to that string; it does not authenticate an OpenAI crawler. Check provider rules and logs against OpenAI's published crawler information.[1] - Track visits without calling them citations. Filter analytics for visits carrying
utm_source=chatgpt.com, then examine engagement and conversions. These visits are a different measure from appearances in answers.[4]
Know which crawler does what
| User agent | OpenAI’s documented purpose | Site owner consideration |
|---|---|---|
OAI-SearchBot | Crawls to help surface websites in ChatGPT Search | Allow it if you want content to be eligible for search summaries and links. |
GPTBot | Crawls content that may be used to improve OpenAI foundation models | Set its robots.txt rule according to your training-use preference. |
ChatGPT-User | Fetches pages for some user-initiated actions | It is not the crawler OpenAI documents for controlling automatic Search discovery. |
OpenAI says the controls for OAI-SearchBot and GPTBot are independent. A publisher can allow search crawling while disallowing training-related crawling. OpenAI also says ChatGPT-User is not used for automatic web crawling or to determine Search inclusion.[1]
Review robots.txt
If the intended policy is to permit ChatGPT Search crawling while disallowing GPTBot, the relevant groups can look like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /Review the full file for more specific path rules, duplicate groups, or generated directives from a content management system. OpenAI’s crawler documentation includes each bot’s name and describes these controls as separate settings.[1]
Robots.txt communicates crawler preferences. It is not an access-control system for private information. Keep confidential pages protected by authentication and authorization.[5]
Check the infrastructure around the site
Check the content delivery network (CDN), web application firewall (WAF), rate limits, bot challenges and application access controls alongside robots.txt. OpenAI’s publisher guidance recommends checking web protection layers and publishes IP ranges for its search crawler.[3]
A request sent with an OAI-SearchBot user-agent string can show how a server responds to that string. It cannot prove that the production firewall permits requests from OpenAI’s documented IP ranges or that the crawler can retrieve every important page. Verify the relevant edge logs and provider rules as well.[1][3]
Verify a change
- Fetch the live robots.txt file from the site root.
- Confirm that the intended
OAI-SearchBotpaths are allowed. - Inspect CDN and WAF rules for bot challenges, 403 responses, and 429 rate limits.
- Check access logs for requests matching OpenAI’s published crawler information.
- Retest key public pages after changing rules.
OpenAI notes that Search systems may take about 24 hours to reflect a robots.txt change. Record what each test establishes: the live robots.txt policy, the response to a test request, or a logged crawler request. None establishes Search inclusion.[1][2]
Measure visits from ChatGPT
OpenAI says referral URLs from ChatGPT Search include the campaign-tracking parameter utm_source=chatgpt.com, which publishers can use in analytics reporting. Filter for the recorded campaign source chatgpt.com and check how your analytics system classifies those visits. Then evaluate engaged visits and conversions. The parameter is a traffic attribution signal, not a count of every time a site appeared in an answer.[4]
Record access and referrals separately
After changing crawler rules, save the URLs tested, test time, response statuses and remaining access failures. Report visits attributed to utm_source=chatgpt.com separately from crawler requests. Neither measure counts every appearance in an answer.
OpenAI says Search uses multiple factors to select results, and placement is not guaranteed.[2]
For Google’s separate eligibility rules, see Google AI Overviews and AI Mode: a site owner’s guide. For Anthropic’s crawler settings, see Claude web crawlers: a site owner’s guide.
Sources
- Overview of OpenAI crawlers. OpenAI Developer Documentation.
- Searching the web with ChatGPT. OpenAI Help Center.
- Publishers and Developers FAQ. OpenAI Help Center.
- Publishers and Developers FAQ. OpenAI Help Center.
- Robots Exclusion Protocol. Internet Engineering Task Force.
Last verified October 17, 2026.