Can AI search bots crawl my website?

When we ran our readiness check across 121 sites, the robots.txt result was mostly sound: 116 sites passed and 5 failed. Other checks found weaker spots in JavaScript rendering and page content. So yes, AI search bots can crawl your website when your access rules let them fetch the page and the returned HTML contains something useful.

Robots rules

Robots.txt is the clearest way to say no. A rule can block a named AI crawler, a path or the whole site. It can also look harmless while preventing the pages you need an assistant to read.

In CiteGraph's AI readiness checks, the robots check passed on 116 of 121 audited sites and failed on 5. That means 5 sites had an explicit crawl access problem in this check. The other 116 did not fail this particular gate.

A pass proves permission, not access. The check says the robots rules did not block the AI crawlers covered by it. It does not test every bot or confirm that every important page returns useful content.

Bot protection

Bot protection is another way AI bots can reach your website and still get nowhere. A firewall may return a challenge, ask for a browser check, require a cookie or deny a request based on its user agent or network reputation.

The 121-site readiness sample does not contain a separate bot protection check. We therefore do not know how many audited sites returned a challenge or blocked an AI crawler at this layer. The ledger gives no share for that failure mode, so we won't invent one.

You can spot the problem by requesting a public page with a crawler user agent and comparing the result with a normal browser request. Look for a challenge page, a redirect to a verification service, an access denied response or a page that contains less text than the browser view.

Browsers carry JavaScript, cookies and other signals. A crawler usually arrives with none of that, and the comparison tells you more than either request alone.

The source ledger contains no vendor quotation, so we have not added a pull quote here. That is a sourcing gap, not a reason to put words in a vendor's mouth.

JavaScript rendering

A page can hide its main content behind JavaScript. The initial HTML contains a shell. The browser runs scripts, calls an API and fills in the product description, prices or answers afterwards.

The readiness check for content present without JavaScript passed on 92 of 121 sites, failed on 18 and warned on 9, according to the published check data. Pass, fail and warn together cover 119 of the 121 sites, so two sites fall outside this check entirely.

The 18 failures are the clearest measured share for this access problem in our sample. That does not mean every failed page is invisible to every AI system. Some systems render JavaScript, while others may use a cached version, a search index or a different fetch process.

Put the basic answer in the HTML response where possible. Product name, what it does, who it serves, important limits and the route to pricing should not appear only after a script completes. Structured data can support those facts, but it cannot rescue an empty page.

Login walls

A login wall is a direct no. If the useful text appears only after sign-in, an unauthenticated crawler cannot read it unless the site offers another public version.

The audited readiness data does not include a dedicated login wall check. We do not know how many of the 121 sites put relevant pages behind authentication, so there is no measured share to report for this failure mode.

Check more than the homepage. A public homepage may explain the product while the comparison page, documentation, pricing detail or customer evidence requires an account. Those are often the pages a buyer needs before making a decision.

If the information is meant to influence a public buyer, publish a public version. Keep private features private, but do not make every useful explanation depend on a session that a crawler cannot create.

The audit

You can test the main access conditions in under an hour. Use a page that matters, such as a product page, pricing page or comparison page, rather than relying on the homepage alone.

1. Open robots.txt and check whether AI crawlers or the relevant page path are disallowed.
2. Request the page without a logged-in browser session. Record whether you receive the page, a redirect, a challenge or an access denied response.
3. Compare the raw HTML with the rendered browser view. Search the raw response for the product name, core description, audience and price or pricing link.
4. Open the page in a private window with JavaScript disabled. Note which buyer-facing facts disappear.
5. Repeat the test on a pricing or comparison page. A clean homepage does not certify the rest of the site.
6. Save the response status, final URL and a copy of the returned HTML. Ask your developer or hosting provider about any difference between the crawler request and the browser request.

For a quicker first pass, use the free AI readiness check. If you need to see whether assistants actually mention and cite your pages, use the free AI visibility checker. These answer different questions: one checks conditions on your site, while the other checks observed answers.

The measured checks

The access evidence is uneven in our sample. Robots access and JavaScript rendering have dedicated checks. Bot protection and login walls do not, so their shares remain unknown rather than quietly becoming zero.

The strongest measured result is robots access: 116 of 121 sites passed, leaving 5 failures. JavaScript-free content passed on 92 sites, failed on 18 and warned on 9. Two sites fell outside that JavaScript check.

The wider readiness picture gives useful context. The check for a sitemap passed on 100 of 121 sites. The check for a homepage that answers a plain request passed on 119. Those are helpful signs, but neither one proves that an AI crawler can reach and read every important page.

The checks that failed or warned most often were a plain statement of what it is, for whom, at what price, against what; comparison or alternatives pages; and questions answered on the page. Crawlability makes the page reachable. Clear content makes it usable once reached.

A different, narrower check found the site's own pages were cited on only 5 of 121 sites, with 11 failing and 5 warning. That is a citation result, not a rendering result, and the totals do not cover all 121 sites.

The limits of the sample

A readiness check is not a guarantee of citation or recommendation. This is a sample of conditions at one point in time, not a guarantee about any future answer. The checks are described on the methodology page.

Our separate citation data, 45,309 citations across 7,260 AI answers, can show which pages appeared in answers, but it cannot confirm that a crawler ever fetched a given page. The figures come from our research data.

It also cannot prove that a future answer will use a page that passed a technical check. AI systems use different fetching and retrieval processes, and access rules can change outside your page template.

The practical order is simple: make the important page publicly fetchable, keep its core facts in the returned HTML, remove accidental challenges and test without a login. Then measure citations separately, because crawl access and being chosen are different events.

AI search bots can crawl your website when your robots rules, access controls, rendering and login requirements let them fetch and read the page.