AI crawler access checker: test eight bots in one run

A common mistake is testing GPTBot against robots.txt, seeing it allowed, and assuming OAI-SearchBot and PerplexityBot are also getting through. That can leave a search crawler blocked while a training crawler remains open, or show access for a bot that never supplies search citations. The cost is a false sense of coverage, followed by pages that do not appear in answers when buyers look for them. You then spend time changing page content when the first problem is crawler policy.

An ai crawler access checker needs to test bot identities separately. It also needs to apply robots.txt rules as each crawler does, rather than looking for a general word such as GPT or Claude and returning one site-level result. Our free check tests eight agents in one pass and keeps search, user retrieval and training access distinct. In the free AI readiness check, the robots check passed on 116 of 121 audited sites, while a plain statement of what the business is, for whom, at what price, against what passed on 20 of 121.

The mistake is testing one bot

A single allow or block assumption hides the jobs behind each user agent. OpenAI says that OAI-SearchBot is used to surface websites in ChatGPT search features, while GPTBot crawls content that may be used in its foundation-model training. Those are different decisions, and OpenAI says each setting is independent. The OpenAI bot documentation gives the example of allowing OAI-SearchBot for search while disallowing GPTBot for training.

Perplexity separates its search crawler from its user-initiated fetcher. According to its bot documentation, PerplexityBot is designed to surface and link websites in search results, while Perplexity-User supports user actions. That distinction matters because a search crawl and a fetch triggered by a person's request are different access paths.

Anthropic has three relevant identities too. Its crawler guidance says ClaudeBot collects content that could contribute to model training, Claude-SearchBot improves search result quality, and Claude-User can retrieve content in response to a user query. Anthropic says disabling Claude-User may reduce visibility for user-directed web search, while disabling Claude-SearchBot may reduce visibility in user search results. That gives you separate visibility and training controls rather than one Claude decision.

This is why an ai crawler checker should not collapse the result into “AI allowed” or “AI blocked”. Those labels lose the decision you need to make. You need to know which agent is allowed, which path that agent supports, and whether the rule affects search, user retrieval or training.

Each agent represents a different decision

The eight agents in the run are Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, GPTBot and Google-Extended. The check treats the first six as critical: if any of them is shut out of the site, the check fails. Five of them feed live answers: Googlebot for the index behind Gemini and AI Overviews, Bingbot for Bing’s index, OAI-SearchBot and ChatGPT-User for ChatGPT, and PerplexityBot for Perplexity. The sixth, ClaudeBot, is Anthropic’s training crawler by Anthropic’s own description (more on that below). GPTBot and Google-Extended are training controls. If only GPTBot or Google-Extended is shut out, the check warns, because that is a training decision rather than a search one.

A pass does not mean that allowing every bot is right for you. It means none of the eight is shut out of the whole site. For example, allowing OAI-SearchBot while disallowing GPTBot makes a different choice from allowing both: the first keeps ChatGPT search eligible while opting out of training, and the second also permits training use. The agents therefore need separate results even when they belong to the same provider.

That separation gives the check a practical output. If OAI-SearchBot is disallowed, the concern is ChatGPT search visibility, because OpenAI says sites opted out of that agent will not be shown in ChatGPT search answers, though they may still appear as navigational links. If ChatGPT-User is disallowed, the concern is ChatGPT fetching your page after a user asks a question. If GPTBot is disallowed, the decision concerns training rather than a search result.

The result should therefore be read as a policy map, not a score. A site can deliberately block training agents while allowing search agents, or accidentally block a search agent through a broad group intended for something else. The checker exposes that difference before you interpret an answer-visibility problem as a content problem.

The wider readiness sample shows why access belongs beside page-quality checks:

Access is close to universal in this sample; usefulness is not. The checker keeps those two facts separate instead of treating an open robots.txt policy as evidence that a page is ready to support an answer.

The checker evaluates identities separately

The check does not ask whether the file contains a friendly-looking line. It reads the published robots.txt file, identifies the rules that apply to the agent being tested, and evaluates that agent’s access separately. It uses the agent’s own group of rules, or the catch-all group when the file has none for that agent, and looks for a rule that shuts the agent out of the whole site: a Disallow of / with no Allow of /. A rule for one user agent is not silently treated as a rule for another. This matters when a file names a product family, a specific crawler or a broad catch-all group.

For OpenAI, the test uses OAI-SearchBot, ChatGPT-User and GPTBot, so search, a fetch for an answer and training each get their own result.

For Perplexity, the test checks PerplexityBot, the published search crawler. It does not test Perplexity-User: Perplexity documents it as a user-action agent and says that this fetch generally ignores robots.txt because it was requested by a person, so a robots.txt result would say little about it.

For Anthropic, the test checks ClaudeBot. Anthropic documents ClaudeBot as its training crawler, but the check counts a ClaudeBot block as a failure alongside the answer crawlers, so read that result with Anthropic’s own description in mind. Claude-SearchBot and Claude-User are not tested separately, so check their rules by hand if your file names them.

For Google, the test checks Googlebot, whose index feeds Gemini and AI Overviews, and Google-Extended, the control Google offers for Gemini training.

Here is a worked example using the aggregate readiness result. First, the sample contains 121 audited sites. Second, the AI crawler access check passed on 116 sites. Third, it failed on 5 sites. Fourth, it warned on 0 sites, so the outcomes account for all 121 sites. Finally, compare that access result with the check for the site’s own pages getting cited, which passed on only 5 of 121 sites. The point is not to turn the figures into a conversion rate. It is that access passed on 116 sites while the citation check passed on 5, so an open robots.txt policy did not amount to evidence of citation.

That limitation matters. The ledger contains no per-bot breakdown for those 121 sites, so the aggregate result does not reveal which of the eight agents caused any individual failure. A site could pass the aggregate policy check while making different decisions for search, training and user retrieval, depending on how the check defines the aggregate result.

That is the difference between an ai crawler access checker and a text search over robots.txt. Text search can tell you that GPTBot appears in the file. It cannot, by itself, tell you whether OAI-SearchBot, ChatGPT-User, PerplexityBot or ClaudeBot receives the same treatment. To check AI bot access, you need the result attached to the agent and its documented purpose.

The method also keeps provider timing separate from file contents. OpenAI says robots.txt changes can take about 24 hours to adjust for search results. A corrected file can therefore be right while the product has not yet changed its behaviour. The checker tells you what the current file says for each identity, but it does not claim that every provider has already applied the change.

What access cannot tell you

An access pass cannot tell you that an assistant will cite your page. It tells you that a crawler or retrieval agent is not blocked by the robots.txt policy as tested. Selection still depends on the page, the query, the provider’s systems and the other sources available to the answer. Reachability is a prerequisite for some paths, not a citation result.

The readiness sample shows the gap clearly. Of 121 sites, 92 passed the check for canonical and Open Graph tags, 101 passed the check for one clear H1, and 119 passed the check that the homepage answers a plain request. Those are useful signals, but they do not establish citation or recommendation. The check for the site’s own pages getting cited passed on only 5 of 121 sites, with 11 failing and 5 warning.

We also cannot tell you from robots.txt alone whether an assistant has already fetched a page, whether the page was selected for a particular answer, or whether a provider will use a training crawler in the way you expect. The 121-site sample describes those sites, not every site, and an access pass is not evidence that any assistant has actually used the page. We measure samples, so the limits belong to the result.

That limit changes what you do next. If a search or retrieval agent is blocked, fix the policy first and allow time for the provider to process the change. If access is open, inspect the page that the crawler needs to use: the plain product statement, pricing, questions answered on the page, and comparison or alternatives pages. The plain product statement, comparison pages, and questions answered on the page are the three checks the sample shows failing or warning most often. Pricing itself held up better, passing on 90 of 121 sites.

Use the weekly research on who owns AI answers to inspect visibility and citations rather than treating a robots result as the finish line. The access checker answers one narrow question: which of these eight agents can use the site under the current robots.txt rules? The next question is whether the pages they can reach give them a reason to use you.

Of the pages these agents can reach, which ones are actually being cited? The answer lives in your citation and recommendation checks, and in the methodology behind how CiteGraph measures them.