Show My Businessto AI Agents
All guides

Technical

How to allow ChatGPT search in robots.txt

Check whether OAI-SearchBot can reach your public pages and choose how GPTBot may use your content for training.

A crawler is software that visits web pages. Website owners can publish instructions in a file called robots.txt. The first decision is which provider you want to reach which pages, and for what purpose.

  1. Read your current rules

    Add /robots.txt after the hostname of the site you want to check. Each User-agent line names a crawler or uses an asterisk to match other crawlers. Allow and Disallow lines describe paths. Save a copy of the current rules before making changes.

  2. Choose a policy for each purpose

    OpenAI names its search crawler OAI-SearchBot and its training crawler GPTBot. You can allow search crawling and decline training crawling separately. For other providers, check the crawler names and purposes in their own documentation.

  3. Check one public page end to end

    Ask your host to compare the rules for that page with the traffic reaching it. A firewall can challenge even an allowed crawler. OpenAI publishes IP ranges that your host can use to verify requests. Keep private customer information behind authentication.

Which crawlers do search and AI services use?

Find the service you want to reach, then check its crawler name against the rules in your robots.txt. Each name links to the provider’s guidance.

Search experienceCrawler name
ChatGPT searchOAI-SearchBot
Google Search, AI Overviews and AI ModeGooglebot
Bing Searchbingbot
Perplexity searchPerplexityBot
Claude web searchClaude-SearchBot

Example

A shop wants to appear in ChatGPT search

Its owner asks the hosting team to allow OAI-SearchBot on public product pages and disallow GPTBot. The team checks the existing path restrictions before editing the file, then confirms the public pages remain reachable.

Check your work

Ask your site manager to show the matching robots.txt rule for a specific public URL and a verified crawler request in the server logs when one arrives. A successful visit establishes access. Inclusion in a particular answer is a separate decision.

Sources