Technical
How to allow ChatGPT search in robots.txt
Check whether OAI-SearchBot can reach your public pages and choose how GPTBot may use your content for training.
A crawler is software that visits web pages. Website owners can publish instructions in a file called robots.txt. The first decision is which provider you want to reach which pages, and for what purpose.
Read your current rules
Add /robots.txt after the hostname of the site you want to check. Each User-agent line names a crawler or uses an asterisk to match other crawlers. Allow and Disallow lines describe paths. Save a copy of the current rules before making changes.
Choose a policy for each purpose
OpenAI names its search crawler OAI-SearchBot and its training crawler GPTBot. You can allow search crawling and decline training crawling separately. For other providers, check the crawler names and purposes in their own documentation.
Check one public page end to end
Ask your host to compare the rules for that page with the traffic reaching it. A firewall can challenge even an allowed crawler. OpenAI publishes IP ranges that your host can use to verify requests. Keep private customer information behind authentication.
Which crawlers do search and AI services use?
Find the service you want to reach, then check its crawler name against the rules in your robots.txt. Each name links to the provider’s guidance.
| Search experience | Crawler name |
|---|---|
| ChatGPT search | OAI-SearchBot |
| Google Search, AI Overviews and AI Mode | Googlebot |
| Bing Search | bingbot |
| Perplexity search | PerplexityBot |
| Claude web search | Claude-SearchBot |
Example
A shop wants to appear in ChatGPT search
Its owner asks the hosting team to allow OAI-SearchBot on public product pages and disallow GPTBot. The team checks the existing path restrictions before editing the file, then confirms the public pages remain reachable.
Check your work
Ask your site manager to show the matching robots.txt rule for a specific public URL and a verified crawler request in the server logs when one arrives. A successful visit establishes access. Inclusion in a particular answer is a separate decision.