ClaudeBot and robots.txt

ClaudeBot is Anthropic’s crawler for collecting public web content that may help train its models. That matters this year because robots.txt can control several Anthropic access paths, and blocking ClaudeBot does not automatically block the others.

Anthropic gives each crawler a different job

Anthropic’s support page explains the three agents and their controls, describing robots used to gather public web data, search the web and retrieve content at a user’s direction. ClaudeBot collects content that could contribute to model training. Claude-User can retrieve pages after someone asks Claude a question. Claude-SearchBot supports search-related access by analysing online content.

That split matters because the same page can be useful in different ways. The support page notes that blocking ClaudeBot signals that future material should be excluded from Anthropic’s training datasets. Blocking Claude-User affects retrieval for a user request, while blocking Claude-SearchBot affects the search path. These are separate robots, so there is no single Claude crawler switch hiding behind one rule.

If you’ve already blocked ClaudeBot and are worried that you damaged your visibility, the data does not support that conclusion on its own. The distinction is about access, not a promise of visibility. Per Anthropic’s guidance, disabling Claude-User may reduce visibility for user-directed web search, and disabling Claude-SearchBot may reduce visibility and accuracy in search results. Allowing either agent leaves that route open, but does not guarantee that Claude will use, cite or recommend your page.

What each robots.txt rule changes

Robots.txt is where you state which crawlers may access your site. Anthropic says it uses different robots “to enable website owner transparency and choice”, so the useful decision is narrower than whether to allow AI in general. Decide whether a page should be available for training, user-directed retrieval, search access, or some combination. A public pricing page and a customer-only page may reasonably have different policies.

Blocking ClaudeBot is the control for future training use. Blocking Claude-User prevents Anthropic retrieving your content in response to a user query, according to its support guidance. Blocking Claude-SearchBot prevents indexing for search optimisation, which the same guidance says may reduce visibility and accuracy in search results.

The policy can therefore vary by page. You may want a public product page available when a buyer asks Claude for software options, while keeping an unpublished research page away from every relevant path. Robots.txt is a request, not a lock, so most site owners pair it with access controls for restricted areas. The free AI readiness check can tell you whether AI crawlers are allowed in the current file, but it cannot decide which access policy fits your business.

What the Claude sample actually shows

CiteGraph sampled 1,467 Claude answers. Claude cited 4.8 sources per answer on average, and the scanned product appeared by name in 6% of those answers, according to our methodology page. Those measures describe citations and naming separately. A product can be named without its own page appearing as a cited source.

Here is the worked example, using the ledger without inventing a site-level forecast. First, take the 1,467 sampled Claude answers. Second, record the average of 4.8 cited sources per answer, which describes how many sources appeared across the sample. Third, record the 6% naming figure, which describes how often the scanned product appeared by name. The two figures sit beside each other, but they do different jobs: 4.8 is a citation average, while 6% is an aggregate naming rate across the sampled answers and sites.

That means you cannot turn 4.8 and 6% into the odds that your site will be cited, and you cannot treat 6% as a baseline for one site before or after a robots.txt change. We also cannot isolate the effect of one robots.txt change from that naming figure. It does not identify the responsible agent, the source page, the query type or the result for a particular domain. The chart below shows the citation average for Claude alongside the other three engines, so you can see the comparison behind 4.8 rather than treating it as a site forecast.

The wider readiness sample gives a separate technical baseline. Across 121 audited sites, the check for AI crawlers allowed in robots.txt passed on 116 sites and failed on 5. The same audit found that the site’s own pages get cited passed on 5 of 121 sites, failed on 11 and warned on 5. A crawler pass is therefore an access signal, not evidence that the site will be cited.

Where the evidence stops

We cannot tell you which individual pages Anthropic has crawled, which agent requested them, or whether a blocked request would have produced a citation. The readiness sample covers 121 sites and the Claude sample covers 1,467 answers. Neither tells you about a site or answer outside those samples, and the data does not isolate the effect of one site’s robots.txt change.

Anthropic’s public support page gives its stated purpose for each agent and the stated consequence of disabling each one. It does not give us a page-level ranking model, a promised crawl schedule or a guarantee that allowing an agent will produce a citation. We should not fill those gaps with confident guesses, especially when a founder is trying to explain a sudden change in visibility.

The readiness robots check has a narrower limit too. It reports whether AI crawlers are allowed, not whether every rule matches your intended separation between training, retrieval and search access. A passing check can coexist with a policy you have not made explicit, so read the file and Anthropic’s agent guidance together.

Set the rule by page purpose

Start with the pages, not the bot name, and mark the content you want available for model development, the content you want Claude to retrieve when a buyer asks a question, and the content you want discoverable through Claude search. Some public product facts may fit all three groups. Private, regulated or unfinished material may fit none. That page-level decision is easier to review than a blanket allow or block rule.

Then inspect robots.txt for the named agents and compare each rule with that page policy. Do not assume that blocking ClaudeBot also blocks Claude-User or Claude-SearchBot. If you make a change, record which agent changed and which pages it covers, so a later visibility result has some context.

Afterwards, measure answer outcomes rather than relying on crawler logs alone. Test buyer questions that should require your product category, record whether Claude names the product and record whether it cites your pages. The AI visibility checker can help with answer and citation checks, while the weekly research on who owns AI answers provides broader context for source selection.

A technical pass remains worthwhile. In the 121-site readiness sample, the homepage answered a plain request on 119 sites, a sitemap was published on 100, and content was present without JavaScript on 92. Those checks do not replace crawler policy, but they reduce other reasons for a page to be hard to use once an agent can reach it.

The caveat that matters most is simple: allowing an Anthropic agent leaves a route to your pages open, but it does not make Claude cite or recommend your business.