GPTBot in robots.txt: allow or disallow, training vs search
To block GPTBot, add User-agent: GPTBot and Disallow: / to robots.txt. That opts out of training only. ChatGPT search uses OAI-SearchBot, a separate rule.
By CiteGraph · 15 Sept 2026 · updated 2 Oct 2026 · every number below has a source
To disallow GPTBot, add two lines to robots.txt: User-agent: GPTBot, then Disallow: /. That tells OpenAI your pages should not be used to train its models. It does not take you out of ChatGPT search, which OpenAI controls through a separate crawler, OAI-SearchBot, so leave that one allowed if you want to stay eligible for search answers. For search results, OpenAI says a robots.txt change can take about 24 hours to show.
User-agent: GPTBot
Disallow: /
GPTBot is OpenAI’s crawler for content that may be used to train its foundation models. In our readiness check, 116 of the 121 sites allow AI crawlers, but the check does not distinguish which agent each site intended to allow.
What does GPTBot do on your site?
GPTBot crawls content that may be used in training OpenAI’s generative AI foundation models. According to OpenAI’s bot documentation, GPTBot is “used to make our generative AI foundation models more useful and safe” and is used to crawl content that may be used in training. That makes it a training crawler, not the crawler OpenAI names for search.
The same page says OpenAI uses web crawlers and user agents for product actions, either automatically or after a user request. It also identifies OAI-SearchBot and GPTBot as separate robots.txt controls, so you have separate choices rather than one large OpenAI switch. In practice, a GPTBot disallow is a training opt-out, nothing else.
GPTBot access does not tell you whether a page can appear as a cited source in a ChatGPT search answer. For that, the relevant agent is OAI-SearchBot.
Can you rely on a GPTBot disallow rule?
Yes, OpenAI publishes GPTBot as a robots.txt user agent and states that disallowing it indicates that a site’s content should not be used in training its generative AI foundation models. If you get the agent name wrong, the rule may express a different policy from the one your team intended. Check the live file after deployment, then test the exact user-agent name rather than relying on a CMS preview.
The two lines at the top of this page are a GPTBot disallow rule. They do not, by themselves, disallow OAI-SearchBot. You can instead allow GPTBot and disallow selected paths, depending on the policy you want your site to express. The syntax is familiar, but the agent name matters, because a rule aimed at GPTBot is not a rule aimed at every OpenAI product.
OpenAI’s documentation also says that each setting is independent. Its example allows OAI-SearchBot so a site can appear in search results while disallowing GPTBot so crawled content should not be used for training. That is the useful pattern when you want ChatGPT search visibility while opting out of training use.
What happens to your visibility when you disallow GPTBot?
A GPTBot disallow does not remove your site from ChatGPT search answers when OAI-SearchBot remains allowed. The page can still be eligible for retrieval, although eligibility does not mean the system will select or cite it. OpenAI is explicit on the search agent: “OAI-SearchBot is for search.”
The same documentation says, “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.” That makes OAI-SearchBot the relevant control if your aim is to stop ChatGPT search answers from using your pages. GPTBot access is a training decision, not a search decision.
This is the practical risk of blocking GPTBot by mistake. You may state a training preference without changing search access, while still believing you have blocked ChatGPT citations. Before changing the file, decide whether you are managing training, search visibility, or both.
Do you need to block OAI-SearchBot?
You need to block OAI-SearchBot if you want to opt out of ChatGPT search answers. OpenAI recommends allowing OAI-SearchBot and allowing requests from its published IP ranges when you want the site to appear in search results. The recommendation concerns search, while GPTBot remains a separate training control.
These two rules express a policy that keeps search access while opting out of GPTBot training use:
The first rule disallows the training crawler. The second allows the search crawler. Together, they keep the two decisions separate instead of treating an OpenAI bot rule as a bundle. You can reverse the policy if you are comfortable with training use but do not want the site shown in ChatGPT search answers.
ChatGPT-User adds another source of confusion. OpenAI states, “ChatGPT-User is not used to determine whether content may appear in Search.” Changing a ChatGPT-User rule is therefore not the same as allowing OAI-SearchBot. We compare all three OpenAI agents, and which to allow, in GPTBot vs OAI-SearchBot vs ChatGPT-User.
How do you write and verify a GPTBot robots.txt rule?
Start by writing the policy in words before writing the directive. “We do not want GPTBot to crawl content for possible training use” leads to a GPTBot disallow. “We want the site available to ChatGPT search” leads to an OAI-SearchBot allow. The two statements can sit together in the same robots.txt file.
Here is a worked example using the readiness check’s actual figures. The check for AI crawlers allowed in robots.txt passed on 116 of 121 sites and failed on 5 of 121 sites.
Read it line by line:
Baseline: the sample contains 116 of 121 passing sites and 5 of 121 failing sites for this check.
Your site: assume it is one of the 5 of 121 sites that failed the robots check.
Training policy: add a GPTBot rule that disallows the content you do not want crawled for possible training use.
Search policy: keep OAI-SearchBot allowed if you want to remain eligible for ChatGPT search.
Verification: fetch the live robots.txt file, test the exact user-agent names, then rerun the check.
The figures describe the measured result, not why each site passed or failed. A passing access check means the crawler can reach permitted content; it does not promise a citation. Check more than the homepage if your server, CDN or CMS generates separate rules for paths.
For search results, OpenAI’s documentation says, “it can take ~24 hours from a site’s robots.txt update for our systems to adjust.” Do not judge the change immediately after deployment. After that, see which OpenAI agents actually visit your pages. Robots.txt is a gate, not a recommendation engine.
How does this affect your ChatGPT visibility?
The first effect is access. If OAI-SearchBot cannot fetch the site under the policy OpenAI describes, the site will not be shown in ChatGPT search answers. If OAI-SearchBot is allowed, the site remains eligible for that search path, but eligibility is not selection.
The figure below plots the fifteen readiness checks across the audited sample. AI crawler access passes on 116 of 121 sites, while the check for a plain statement of what the product is, who it is for, its price and its alternative passes on 20 of 121 sites.
Readiness checks across 121 audited sites. AI crawler access passes more often than the page checks that help an answer system use the content.
The wider pattern is that access can be configured while a page still lacks information an answer system can use. A clear page structure, direct answers and commercial context matter after the crawler can reach the content. These are readiness signals, not a ranking formula.
The readiness sample covers 121 sites; a robots.txt result in that sample says nothing about the sites we have not checked. A robots.txt pass says the crawler is allowed. It does not say that ChatGPT cited your page, named your business or preferred it over another source. The free AI readiness check shows the access and page checks, while the free AI visibility checker is the more direct place to inspect visibility outcomes.
Inspect the live robots.txt file, then test the exact user-agent names. Do not infer the policy from a CMS setting or a template comment.
After access is settled, improve one page that answers a buyer’s question in plain text.
Questions people ask
What does GPTBot do with content on my website?+
GPTBot crawls content that may be used in training OpenAI’s generative AI foundation models. It is a training crawler, not the crawler OpenAI names for search, so allowing or disallowing it expresses a training preference rather than a search visibility decision.
Does GPTBot stop ChatGPT search visibility?+
No. A GPTBot disallow does not remove your site from ChatGPT search answers when OpenAI’s search crawler remains allowed. The search crawler controls that search path, while GPTBot controls content that may be used for training, so the settings are independent.
Which robots.txt rule controls whether ChatGPT can show my site?+
OpenAI’s search crawler controls whether a site can appear in ChatGPT search answers. GPTBot controls content that may be used for training, so a rule for GPTBot does not by itself control search visibility.
How long does a robots.txt change take to affect search visibility?+
For search results, OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust. After that window, fetch the live file and test the exact user agent names rather than relying on a CMS preview.