PerplexityBot and robots.txt

The mistake is treating every Perplexity request as the same crawler. You block a user agent called PerplexityBot, then discover that a user-triggered fetch uses a different agent and follows different rules. Or you allow the name in robots.txt but reject the request at the WAF because the source IP is not on your allowlist. Either way, you have changed the wrong control.

For perplexitybot, the useful distinction is between discovery and a fetch requested by a person. Perplexity publishes a crawler for search results and a separate fetcher for user actions. Its documentation also publishes the IP source you should use when checking requests. Across the 121 sites in our readiness checks, 116 passed the check for AI crawlers allowed in robots.txt, while 5 failed. That is a high pass rate, but it leaves a small group with a direct access problem.

Are you blocking PerplexityBot by mistake?

Perplexity says, “PerplexityBot is designed to surface and link websites in search results on Perplexity.” It also says the bot is not used to crawl content for AI foundation models. That gives the bot a narrower published job than the phrase “AI crawler” often suggests: it helps Perplexity find pages that can appear as sources in search results.

The published user agent is:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

A robots.txt rule that disallows PerplexityBot tells this fetcher not to use the page, assuming the request respects the file and reaches the site. A WAF rule can create the same outcome even when robots.txt allows it. Check both layers. A green line in the file does not prove that the server accepts the request.

The check did not identify which of the 5 failing sites had blocked Perplexity specifically, so do not read the result as a Perplexity-only pass rate. It measures the broader AI crawler rule.

The figure below lines up all fifteen readiness checks side by side. Robots.txt access is one of the strongest results in the set, second only to homepage reachability, and well ahead of the statement and comparison checks.

The wider readiness picture matters because crawl access is only one gate. In the same sample, 119 homepages answered a plain request, 100 sites published a sitemap, and 92 had content present without JavaScript. Access gets a page into the possible source set. It does not make the page a useful source once it is there.

Which Perplexity user agent should you allow?

You should separate PerplexityBot/1.0 from Perplexity-User/1.0 in your logs and policy. Perplexity describes the second agent as supporting user actions within Perplexity. Its published user agent is:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)

The difference is operational. Perplexity says that when a user asks a question, it might visit a web page to help provide an accurate answer and include a link to that page in the response. It also says this fetcher is not used for web crawling or to collect content for training AI foundation models. A request triggered by a user is treated differently, so your disallow rules do not govern it in the usual way.

That does not make robots.txt irrelevant. It still controls the published crawler, and it remains a clear record of whether you want PerplexityBot to discover and surface your pages. But a denied rule is not a reliable way to stop every Perplexity request. If your policy needs to stop user-initiated fetches, inspect the request at the server or WAF layer and apply a policy based on the documented user agent and verified source IP.

Do not treat the user agent string alone as proof. Perplexity’s documentation says to use the most current IP ranges from its official JSON endpoints, and that those addresses are updated regularly and should be the source of truth for WAF configuration. The current IP values are not in our ledger, so we do not reproduce or infer them here.

What does PerplexityBot fetch from your site?

The published description answers the narrow question: PerplexityBot fetches websites so they can be surfaced and linked in Perplexity search results. It is not described as a training crawler. Perplexity-User is different again: it fetches a page in response to a user action, and that fetch generally ignores your robots.txt disallow rules, per Perplexity's own documentation.

That distinction changes what you should inspect. For PerplexityBot, look at pages that answer the questions you want Perplexity to source. For Perplexity-User, look at the page a user would need after asking a question, including whether the important answer is available to the fetcher and whether the response contains a stable, linkable URL.

The page itself still has to do useful work. In our 121 site checks, only 20 passed the check for a plain statement of what the product is, for whom, at what price, and against what. By contrast, 101 passed the check for one clear H1, and 90 passed the check for a visible price or linked pricing page. Clear headings and pricing help, but the low pass count for the complete statement shows where many pages remain vague.

The same sample found 54 sites passing the check for comparison or alternatives pages, while 56 passed the check for questions answered on the page. Those are not Perplexity requirements published in the bot documentation. They are signals from our site audit about whether a page gives an answer engine material it can quote or link. Keep the categories separate: the official documentation tells you how Perplexity identifies its fetchers, while the audit shows common readiness gaps.

Perplexity cited an average of 5.3 sources per sampled answer in our scan data, and the scanned product was named in 8.2% of those answers. Those figures describe answer samples, not a ranking position and not a promise that allowing a crawler will produce a citation. They show why source access and source quality belong in the same review.

How do you check PerplexityBot robots.txt access?

Use this procedure before changing a blocking rule. Most of it fits inside an hour if you already have log access. The full WAF and CDN investigation can be a follow-up task.

1. Read the current robots.txt file and search for PerplexityBot, Perplexity-User, wildcard rules, and rules that apply to the target path. Record the result before editing it.
2. Compare the user agent in your access logs with the two published strings. Separate PerplexityBot/1.0 from Perplexity-User/1.0; do not combine their visits into one crawler total.
3. Check the request source against the current IP ranges in Perplexity’s official JSON endpoint. Do not use an old copied list in a WAF rule.
4. Check the WAF or CDN logs for a deny, challenge, timeout, redirect loop, or server error. Leave a full request trace through the application and origin for a later investigation if the first checks do not explain the result.
5. Test a representative page and inspect the response body. Confirm that the useful text is present without relying on a client-side render.

If the request is allowed in robots.txt but absent from the origin logs, the block may sit at the CDN or WAF. If it reaches the origin but returns a thin shell, the problem is not the crawler rule. The rendering check is a separate technical check, not a footnote to robots access. Recheck the logs after your normal log cycle, especially when you change WAF rules or IP ranges.

You can run the free AI readiness check for the broader set of controls, or use the free AI visibility checker when you need to compare whether your product is named and whether your pages appear as sources. Neither replaces log analysis. The check is a site sample; the logs show what happened to your requests.

Should you block Perplexity if you do not want citations?

You can block PerplexityBot in robots.txt if you do not want that published crawler to use your pages for Perplexity search results. That choice has a clear trade-off: you remove one documented path by which your pages can be surfaced and linked. It does not necessarily stop a user-initiated Perplexity-User fetch, because that request is handled outside the normal robots.txt restriction.

A broad WAF block needs more care. If you block by user agent only, another client can claim the same string. If you block by IP only, you need the current published ranges and an update process. If you block both, test that the rule does not catch unrelated traffic or prevent ordinary users from reaching the page. The official guidance says the IP addresses are updated regularly, so a static allowlist is maintenance work, not a one-time setting.

The decision should follow your access policy, not a hope that a blocked crawler will prevent every appearance. Perplexity can cite pages from other sites, and our scan measures answers rather than the full web. Our sample covers 1,787 Perplexity answers; it cannot tell you what an unmeasured query or a future IP range will do.

If your aim is inclusion, allow the crawler you intend to support, verify the current IP ranges, and make the target pages plain enough to use. If your aim is exclusion, treat robots.txt as one control and enforce the rest at the edge. The methodology page explains how we measure answer samples and citations, which is useful when you need to keep crawler access separate from observed visibility.

PerplexityBot can discover pages for Perplexity search, while user-triggered fetches generally require server or edge controls beyond robots.txt.

What should you ask before changing PerplexityBot access?

Does PerplexityBot use my content to train an AI model?

Perplexity says PerplexityBot is not used to crawl content for AI foundation models. Its published purpose is to surface and link websites in Perplexity search results.

Can you stop Perplexity-User from fetching a private page?

A robots.txt disallow is not a dependable control for a user-triggered fetch. Use authentication, server rules, CDN controls, or a WAF policy if the page must not be fetched.

How can you verify that a request really came from PerplexityBot?

Compare the user agent with the published PerplexityBot/1.0 string, then check the source against the current IP ranges in Perplexity’s official JSON endpoint. Do not rely on the user agent alone because it can be copied.

Will allowing PerplexityBot guarantee that your page gets cited?

No. Allowing access removes one technical barrier, but it does not determine which pages Perplexity selects or cites. The answer-sample data above shows observed visibility, not a guarantee for an individual page.

Should you remove PerplexityBot from robots.txt if you want fewer AI visits?

You can disallow PerplexityBot if you do not want its search crawler to use your pages. That rule may not stop user-triggered Perplexity fetches, so enforce a wider policy at the edge if required.