Google-Extended: what it blocks and what it does not

116 of the 121 sites in our readiness sample allow AI crawlers in robots.txt. The setting is common, but common does not mean understood.

The phrase google-extended robots.txt usually appears when a site owner wants to limit Gemini training use. The useful question is what the rule changes, and what it leaves alone.

Google-Extended is a training control

Google-Extended is a robots.txt control token. It tells Google how to respect a site's preference about training use when Google crawls the site.

Robots.txt is the crawl-instruction file at the site root; Google-Extended is one control token used inside it; grounding is the separate job of using available content to support an answer, rather than train a model.

Google's own documentation on Google's common crawlers puts the search boundary plainly: “Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.”

If your concern is training use, Google-Extended is relevant. If your concern is whether Gemini cites a product page, look beyond this token.

What Google-Extended blocks

A Google-Extended rule controls training use of site content covered by the rule. It gives you a direct way to express a preference if you do not want Google to use those pages for that purpose.

It is a policy choice, not an SEO trick. Decide it at the level of whoever owns content risk at your company, rather than as a side effect of a crawl-file edit.

Google says its automated crawlers obey robots.txt rules. It also says Google-Extended does not have a separate HTTP request user agent string. You should not expect a clean, separate stream of requests that makes the setting obvious in server logs.

The technical detail matters during a review. Google says crawling uses existing Google user agent strings, while the Google-Extended token is used “in a control capacity”. The token is therefore a policy instruction, not necessarily the name you will see in each request.

What it does not block

Google-Extended does not block your site from Google Search. It does not act as a Google Search ranking signal. Those points come directly from Google's public documentation, rather than from an inference about how the file might work.

It is also not a switch for Google AI Overviews. Blocking the token does not tell Google to ignore a page when it assembles a search answer.

A site owner who wants to limit an AI search feature therefore needs to inspect the control that governs that feature, rather than assume every Google AI setting has the same reach.

How robots.txt works here, with an example

For Google-Extended, the relevant pattern is a user-agent line naming the token followed by a disallow instruction. A broad example looks like this:

User-agent: Google-Extended
Disallow: /

That example expresses a site-wide preference for the content covered by the token. A narrower rule can name a path rather than the whole site. Use that only if your content policy distinguishes between sections.

If the rule is meant to protect private or sensitive material, do not rely on robots.txt as the only access control. Robots instructions are not a replacement for authentication.

Google's wording also explains why a robots.txt review needs care. Its crawlers “always obey robots.txt rules when crawling automatically”, but Google-Extended “doesn't have a separate HTTP request user agent string”. Check the syntax, publish the file where your site expects it, and test the result against the policy you intended to express.

Seeing Googlebot fetch a page does not by itself prove that Google ignored the Google-Extended preference. The public documentation says existing Google user agent strings do the crawling while the token controls the relevant use.

Decide what the rule is meant to control

Start with the business decision in plain language: “We do not want this content used for Google-Extended training.” Then check whether the selected paths match that decision.

Only 5 of the 121 sites in our sample passed the check that their own pages get cited. A robots.txt preference will not tell you whether buyers can find your company in an AI answer.

Keep the scope clear. Is the policy for the whole site, only editorial content, or only a proprietary section? The robots file should reflect that decision rather than becoming a blanket response to a vague fear.

Google-Extended is not a ranking lever. Blocking it will not improve the position of your pages, and allowing it will not guarantee that Gemini cites them.

The rest of the site still affects AI visibility

If you allow AI crawlers, or if your main concern is citations, the rest of the site still matters. In our 121-site sample, the weakest readiness checks concerned plain product statements, comparison or alternatives pages, and questions answered on the page.

Only 20 of the 121 sites passed the check for a plain statement of what they are, for whom, at what price, and against what. The check for questions answered on the page passed on 56 sites, while the comparison or alternatives check passed on 54.

Those are page and content problems. A robots setting cannot explain what your product does, who it suits, or how it compares with another option.

The chart below compares the readiness checks by the number of sites that passed them.

Five sites failed the check for AI crawlers allowed in robots.txt. The other readiness checks show why crawl access is only one part of making a site usable to an assistant.

You can use the free AI readiness check to inspect these signals together. If you want to see whether assistants actually name your company or cite its pages, use the free AI visibility checker. Readiness is about whether the site gives an assistant usable material. Visibility is about what the assistant did with it.

Our data has a limit. We measure samples, not every website or every Gemini response, and we print the margins of what we have measured rather than pretending to know what we have not.

Decide the policy, then measure the outcome

Treat Google-Extended as a content-use decision. Write down whether you want to allow or block training use, identify the paths covered, and check the published robots.txt rule against that decision.

Then measure the outcome you actually care about. For search visibility, review crawl access, page clarity and the facts a system needs to use an answer. For citations, track the pages and prompts that matter to buyers.

Start with pages buyers can quote. Make the product statement plain. Answer the questions people ask. Publish comparisons where a buyer needs to choose between options. Technical access gives those pages a chance, but it does not supply the missing explanation.

If you want to block training use, do not use citation counts as proof that the policy worked. Check the result against your written policy, then use the weekly research on who owns AI answers or your own tracked prompts for the visibility question.

The opening number was 116 of 121 sites allowing AI crawlers in robots.txt. The number has not changed in this sample. What should change is the decision behind it: use Google-Extended for the training preference it covers, and measure search or AI visibility on its own terms.