GPTBot vs OAI-SearchBot vs ChatGPT-User: which to allow

A user agent is the name a web crawler or browser-like visitor sends to your server. That name matters this year because OpenAI uses separate agents for search, model training and user-triggered page visits.

If you are checking the gptbot user agent, do not treat every OpenAI visitor as the same thing. If you block the wrong agent, you can lose search visibility without meaning to. Getting this wrong can quietly cost you search visibility, since neither ChatGPT nor you will notice right away.

What each user agent does

OpenAI says its crawlers and user agents “perform actions for its products, either automatically or triggered by user request” in its bot documentation. The permissions are independent, which is the useful part for a site owner.

A single rule for all OpenAI agents can therefore produce a result you did not intend.

In CiteGraph’s readiness checks, the robots check passed on 116 of 121 audited sites. That is a useful sign that access is often already in place, but it does not tell you whether the right agent has access for the right purpose.

The choice behind the block

The default we recommend for a site that wants to be found in ChatGPT is simple: allow OAI-SearchBot and ChatGPT-User, and disallow GPTBot if you do not want your pages used for training.

There is a fair objection. If ChatGPT-User does not crawl the web automatically and does not affect Search, allowing it cannot improve your search visibility. That is true. It permits a user-triggered page visit, rather than helping your pages enter the search index.

If your policy is to allow only the crawler that affects search, allow OAI-SearchBot and disallow GPTBot and ChatGPT-User. If your policy is to make your pages available when a user’s question causes a visit, allow ChatGPT-User as well. The important mistake is calling all three settings a search setting.

The robots.txt block

For the usual visibility-first choice, use this block:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: GPTBot
Disallow: /

This block spells out the decision in plain form: OAI-SearchBot is allowed for ChatGPT search, ChatGPT-User is allowed for a page visit after a user request, and GPTBot is disallowed for its stated training purpose.

If you want to allow GPTBot too, remove its separate rule or replace the disallow rule with an allow rule. Do that only when your content policy accepts training access. Search visibility does not require it.

Keep the rules separate. A single wildcard rule can make your intent less clear, particularly when you later add another crawler or change one permission. The block above also makes a useful record for your team because someone reading the file can see which OpenAI purpose you chose.

The documentation notes that it can take about 24 hours from a robots.txt update for OpenAI’s systems to adjust search handling. Do not change the file and expect an immediate answer from a test query. Check the deployed file, then give the systems time to respond.

Robots.txt controls access. It does not guarantee a citation, a mention or a recommendation.

What access cannot solve

A page that can be fetched can still be a poor source. It may never state what the company sells, who it is for, what it costs or how it compares with the alternatives a buyer is considering.

That gap shows up in our free AI readiness check across 121 sites. The plain statement of what the product is, who it is for, what it costs and what it is against passed on 20 sites, warned on 56 and failed on 43.

Comparison or alternatives pages passed on 54 sites, warned on 46 and failed on 21. Questions answered on the page passed on 56 sites and warned on 63.

Those pages give an assistant material it can use when a buyer asks for a shortlist or a comparison. Access alone does not create that material.

Technical access was less often the problem in this sample. Content was present without JavaScript on 92 sites, while 18 failed and 9 warned.

That pattern does not prove that content beats access in every case. It does show why an allowed crawler is only the first check.

A clear page also needs a clear identity. Organization schema passed on 72 of 121 sites, while Product or SoftwareApplication schema passed on 60.

These checks are not substitutes for readable copy, but they can expose whether the site has stated what the business and product are.

This is a sample of 121 sites we audited, not a prediction of how OpenAI treats every page or category.

A short check before you publish

Use this checklist after changing the file:

• Open the live robots.txt file and confirm that the rules you edited are deployed on the host you care about.
• Check that OAI-SearchBot is allowed if you want the site to appear in ChatGPT search answers.
• Check that ChatGPT-User is allowed if you want to support a page visit triggered by a user’s question.
• Keep the GPTBot rule aligned with the training policy you chose above.
• Make sure the important answer is present in HTML without JavaScript, rather than relying on a visitor to assemble it in the browser.
• State what you sell, who it is for, the price and the relevant alternative on the page a buyer needs.
• Recheck after OpenAI’s documented adjustment period instead of treating an immediate test as final.

You can pair the robots check with the free AI visibility checker, then inspect the page that an answer cites rather than stopping at an allow rule. Our methodology page explains how we measure the checks, which matters when a pass looks more certain than the evidence allows.

Allow OAI-SearchBot and ChatGPT-User so ChatGPT can find and visit you, and disallow GPTBot only if you don't want your pages used for training.