See the steps before a citation, as they happen
Getting cited by AI takes months, and most of that time looks like nothing. It isn’t. Google indexes the page, ChatGPT’s search crawler picks it up, then one day an assistant opens it to answer someone. CiteGraph records each of those requests from your own server, checks every one against the addresses the company publishes, and shows them beside the scans that tell you when you’re named.
Endpoint https://www.citegraph.app/api/crawls · server-side · included with every plan
What gets recorded
| Kind | What it means | Crawlers |
|---|---|---|
| AI answers | An assistant opened the page while answering someone's question. The step right before a citation. | ChatGPT-User, Claude-User, Perplexity-User, Google-Agent, MistralAI-User, DuckAssistBot |
| Indexing | Search indexes that AI answers are built from, including the assistants' own. | Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| Training | Crawlers collecting pages for future models. | GPTBot, ClaudeBot, GoogleOther, Applebot, CCBot, Bytespider, meta-externalagent |
| Other AI | AI crawlers that don't say what they fetch for. | GrokBot, FacebookBot, TongyiBot, OAI-AdsBot |
How it works
A few lines on your server look at each request’s user agent. When it looks like an AI or search crawler asking for a page (not an image, script or stylesheet), they send CiteGraph the URL, the user agent and the address, in the background. Human visitors never match and are never sent.
On our side the crawler list decides what the request is, so a newly named crawler is recognised without you changing anything. The address is checked against the ranges the operator publishes; impersonators are left out. Your site ID is in the app, on the AI crawlers page of your project, with the code already filled in.
Next.js
For Vercel, Netlify or any Next.js host.
- Create citegraph-crawl.ts at the root of your project (inside src/ if you use one).
- Create or update proxy.ts in the same folder. On Next.js 15 or older the file is middleware.ts and the function is named middleware.
- Deploy, then press Check installation on the AI crawlers page.
citegraph-crawl.ts
// citegraph-crawl.ts: tells CiteGraph when an AI crawler requests a page.
// Humans and static files are skipped here, and nothing is awaited, so no page waits on it.
import type { NextFetchEvent, NextRequest } from "next/server";
const SITE_ID = "cgs_YOUR_SITE_ID";
const BOTS = /chatgpt-|oai-|gptbot|claude-|claudebot|anthropic-ai|perplexity-|perplexitybot|googlebot|googleother|google-|googleagent|bingbot|msnbot|copilot|applebot|amazonbot|amzn-|duckassist|xai-|grokbot|grok-|meta-external|facebookbot|mistralai-|kimi-|kimibot|bytespider|doubaobot|tiktokspider|baiduspider|erniebot|yiyanbot|qwen-|qwenbot|tongyibot|aliyunbot|chatglm|deepseekbot|cohere-|ai2bot|youbot|ccbot|citegraph-verify/i;
const SKIP = /\.(avif|bmp|br|cjs|css|csv|eot|gif|gz|ico|jpe?g|js|json|map|mjs|mov|mp3|mp4|otf|pdf|png|svg|ttf|wasm|wav|webm|webmanifest|webp|woff2?|zip)$/i;
export function trackAICrawl(request: NextRequest, event: NextFetchEvent) {
const userAgent = request.headers.get("user-agent") || "";
if (!BOTS.test(userAgent) || (request.method !== "GET" && request.method !== "HEAD")) return;
if (SKIP.test(request.nextUrl.pathname)) return;
const h = request.headers;
const ip = (h.get("cf-connecting-ip") || h.get("x-real-ip") || h.get("x-forwarded-for") || "").split(",")[0].trim();
const token = process.env.CITEGRAPH_CRAWL_TOKEN;
event.waitUntil(
fetch("https://www.citegraph.app/api/crawls", {
method: "POST",
headers: { "Content-Type": "application/json", ...(token ? { Authorization: `Bearer ${token}` } : {}) },
body: JSON.stringify({ siteId: SITE_ID, href: request.url, userAgent, ip, referrer: h.get("referer"), source: "nextjs" }),
signal: AbortSignal.timeout(1500),
}).catch(() => {}),
);
}
proxy.ts
// proxy.ts, next to your app folder (middleware.ts with "export function middleware" before Next.js 16).
// Already have one? Add the trackAICrawl line to the top of your function and keep the rest.
import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server";
import { trackAICrawl } from "./citegraph-crawl";
export function proxy(request: NextRequest, event: NextFetchEvent) {
trackAICrawl(request, event);
return NextResponse.next();
}
export const config = {
matcher: ["/((?!api|_next/static|_next/image|favicon.ico).*)"],
};
Cloudflare
For Framer, Webflow, WordPress, Shopify or any host, when your DNS is on Cloudflare.
- Your domain's DNS must be on Cloudflare with the proxy switched on (the orange cloud) for example.com.
- In Cloudflare: Workers & Pages, Create, Start with Hello World, Deploy. Then Edit code, replace everything with the code below, and Deploy.
- In the Worker's Settings, Domains & Routes, add two routes on the example.com zone: example.com/* and www.example.com/*.
- Press Check installation on the AI crawlers page.
worker.js
// A Cloudflare Worker on your site's routes. Every request passes through untouched;
// requests from AI crawlers are also reported to CiteGraph in the background.
const SITE_ID = "cgs_YOUR_SITE_ID";
const BOTS = /chatgpt-|oai-|gptbot|claude-|claudebot|anthropic-ai|perplexity-|perplexitybot|googlebot|googleother|google-|googleagent|bingbot|msnbot|copilot|applebot|amazonbot|amzn-|duckassist|xai-|grokbot|grok-|meta-external|facebookbot|mistralai-|kimi-|kimibot|bytespider|doubaobot|tiktokspider|baiduspider|erniebot|yiyanbot|qwen-|qwenbot|tongyibot|aliyunbot|chatglm|deepseekbot|cohere-|ai2bot|youbot|ccbot|citegraph-verify/i;
const SKIP = /\.(avif|bmp|br|cjs|css|csv|eot|gif|gz|ico|jpe?g|js|json|map|mjs|mov|mp3|mp4|otf|pdf|png|svg|ttf|wasm|wav|webm|webmanifest|webp|woff2?|zip)$/i;
export default {
async fetch(request, env, ctx) {
const response = await fetch(request);
const userAgent = request.headers.get("user-agent") || "";
const { pathname } = new URL(request.url);
if (BOTS.test(userAgent) && (request.method === "GET" || request.method === "HEAD") && !SKIP.test(pathname)) {
ctx.waitUntil(
fetch("https://www.citegraph.app/api/crawls", {
method: "POST",
headers: { "Content-Type": "application/json", ...(env.CITEGRAPH_CRAWL_TOKEN ? { Authorization: "Bearer " + env.CITEGRAPH_CRAWL_TOKEN } : {}) },
body: JSON.stringify({ siteId: SITE_ID, href: request.url, userAgent, ip: request.headers.get("cf-connecting-ip"), statusCode: response.status, source: "cloudflare" }),
}).catch(() => {}),
);
}
return response;
},
};
Node / Express
For Express, or anything that takes Express middleware.
- Save the file next to your server entry.
- Register it with app.use before your routes. Node 18 or newer (it uses the built-in fetch).
- Deploy, then press Check installation on the AI crawlers page.
citegraph-crawl.js
// citegraph-crawl.js: Express middleware. Register it before your routes:
// app.use(require("./citegraph-crawl"));
const SITE_ID = "cgs_YOUR_SITE_ID";
const BOTS = /chatgpt-|oai-|gptbot|claude-|claudebot|anthropic-ai|perplexity-|perplexitybot|googlebot|googleother|google-|googleagent|bingbot|msnbot|copilot|applebot|amazonbot|amzn-|duckassist|xai-|grokbot|grok-|meta-external|facebookbot|mistralai-|kimi-|kimibot|bytespider|doubaobot|tiktokspider|baiduspider|erniebot|yiyanbot|qwen-|qwenbot|tongyibot|aliyunbot|chatglm|deepseekbot|cohere-|ai2bot|youbot|ccbot|citegraph-verify/i;
const SKIP = /\.(avif|bmp|br|cjs|css|csv|eot|gif|gz|ico|jpe?g|js|json|map|mjs|mov|mp3|mp4|otf|pdf|png|svg|ttf|wasm|wav|webm|webmanifest|webp|woff2?|zip)$/i;
module.exports = function citegraphCrawl(req, res, next) {
const userAgent = req.headers["user-agent"] || "";
if (BOTS.test(userAgent) && (req.method === "GET" || req.method === "HEAD") && !SKIP.test(req.path)) {
res.once("finish", () => {
const ip = String(req.headers["cf-connecting-ip"] || req.headers["x-forwarded-for"] || req.socket.remoteAddress || "").split(",")[0].trim();
const token = process.env.CITEGRAPH_CRAWL_TOKEN;
fetch("https://www.citegraph.app/api/crawls", {
method: "POST",
headers: { "Content-Type": "application/json", ...(token ? { Authorization: "Bearer " + token } : {}) },
body: JSON.stringify({ siteId: SITE_ID, href: req.protocol + "://" + req.get("host") + req.originalUrl, userAgent, ip, statusCode: res.statusCode, source: "express" }),
signal: AbortSignal.timeout(1500),
}).catch(() => {});
});
}
next();
};
WordPress / PHP
For WordPress, Laravel or any PHP site with cURL.
- WordPress: paste into functions.php, or save as a must-use plugin so a theme change keeps it.
- Other PHP: require the file at the top of index.php.
- Press Check installation on the AI crawlers page.
citegraph-crawl.php
<?php
// WordPress: add to your theme's functions.php, or save as wp-content/mu-plugins/citegraph-crawl.php.
// Any other PHP site: require this file at the top of index.php.
function citegraph_track_ai_crawl() {
$ua = $_SERVER['HTTP_USER_AGENT'] ?? '';
$method = $_SERVER['REQUEST_METHOD'] ?? 'GET';
if (($method !== 'GET' && $method !== 'HEAD') || !preg_match('/chatgpt-|oai-|gptbot|claude-|claudebot|anthropic-ai|perplexity-|perplexitybot|googlebot|googleother|google-|googleagent|bingbot|msnbot|copilot|applebot|amazonbot|amzn-|duckassist|xai-|grokbot|grok-|meta-external|facebookbot|mistralai-|kimi-|kimibot|bytespider|doubaobot|tiktokspider|baiduspider|erniebot|yiyanbot|qwen-|qwenbot|tongyibot|aliyunbot|chatglm|deepseekbot|cohere-|ai2bot|youbot|ccbot|citegraph-verify/i', $ua)) return;
$path = parse_url($_SERVER['REQUEST_URI'] ?? '/', PHP_URL_PATH) ?: '/';
if (preg_match('/\.(avif|bmp|br|cjs|css|csv|eot|gif|gz|ico|jpe?g|js|json|map|mjs|mov|mp3|mp4|otf|pdf|png|svg|ttf|wasm|wav|webm|webmanifest|webp|woff2?|zip)$/i', $path)) return;
$https = !empty($_SERVER['HTTPS']) && $_SERVER['HTTPS'] !== 'off';
$payload = json_encode([
'siteId' => 'cgs_YOUR_SITE_ID',
'href' => ($https ? 'https' : 'http') . '://' . ($_SERVER['HTTP_HOST'] ?? '') . ($_SERVER['REQUEST_URI'] ?? '/'),
'userAgent' => $ua,
'ip' => $_SERVER['HTTP_CF_CONNECTING_IP'] ?? $_SERVER['REMOTE_ADDR'] ?? null,
'statusCode' => http_response_code() ?: null,
'source' => 'php',
]);
$headers = ['Content-Type: application/json'];
$token = getenv('CITEGRAPH_CRAWL_TOKEN');
if ($token) $headers[] = 'Authorization: Bearer ' . $token;
$ch = curl_init('https://www.citegraph.app/api/crawls');
curl_setopt_array($ch, [
CURLOPT_POST => true,
CURLOPT_HTTPHEADER => $headers,
CURLOPT_POSTFIELDS => $payload,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_CONNECTTIMEOUT_MS => 800,
CURLOPT_TIMEOUT_MS => 1500,
]);
curl_exec($ch);
curl_close($ch);
}
if (function_exists('add_action')) add_action('shutdown', 'citegraph_track_ai_crawl');
else register_shutdown_function('citegraph_track_ai_crawl');
Any server
For Rails, Django, Go, Laravel, a reverse proxy: anything that can send a POST.
- From your server, send one POST per AI crawler request, in the background.
- The fields and the two patterns are below. Everything else is decided on our side.
- Press Check installation on the AI crawlers page.
The request
POST https://www.citegraph.app/api/crawls
Content-Type: application/json
Authorization: Bearer cgbot_... (only if you created a token)
{
"siteId": "cgs_YOUR_SITE_ID",
"href": "https://example.com/pricing",
"userAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot",
"ip": "203.0.113.7",
"statusCode": 200,
"referrer": null,
"source": "custom"
}
Send it only when the user agent matches:
/chatgpt-|oai-|gptbot|claude-|claudebot|anthropic-ai|perplexity-|perplexitybot|googlebot|googleother|google-|googleagent|bingbot|msnbot|copilot|applebot|amazonbot|amzn-|duckassist|xai-|grokbot|grok-|meta-external|facebookbot|mistralai-|kimi-|kimibot|bytespider|doubaobot|tiktokspider|baiduspider|erniebot|yiyanbot|qwen-|qwenbot|tongyibot|aliyunbot|chatglm|deepseekbot|cohere-|ai2bot|youbot|ccbot|citegraph-verify/i
and the path is not a static file:
/\.(avif|bmp|br|cjs|css|csv|eot|gif|gz|ico|jpe?g|js|json|map|mjs|mov|mp3|mp4|otf|pdf|png|svg|ttf|wasm|wav|webm|webmanifest|webp|woff2?|zip)$/i
Send it in the background with a short timeout (1.5s). Never make the page wait for it.
Optional: a request token
Create a token on the AI crawlers page and set it on your server as CITEGRAPH_CRAWL_TOKEN; every snippet above sends it when it is set. Once it is deployed, switch on Reject reports without it, and reports that don’t carry the token are refused. Rotating the token stops the old one at once; deleting it switches rejection off.
Questions
Why can't a normal analytics script see AI crawlers?+
Crawlers request the HTML and leave. They don't run JavaScript, so a script in the page never loads for them. The request only exists on your server, which is why tracking runs there: in your Next.js proxy, a Cloudflare Worker, Express middleware, or a PHP hook.
Does it slow my site down?+
No. The code checks the user agent first, so human visitors are never sent anywhere. For crawler requests it sends one small report in the background, with a 1.5 second timeout, after or beside the response. A failure is swallowed; the page is never held for it.
How do you know a visit really is ChatGPT?+
OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, DuckDuckGo, Mistral and Common Crawl publish the IP addresses their crawlers use. Every report is checked against those lists. A request that claims to be one of them from another address is an impersonator: it is kept out of every number. Crawlers whose operators publish nothing are shown as user agent only.
Is a crawler visit the same as being cited?+
No. It shows a page was indexed or read, which has to happen before a citation can. Whether an assistant names or cites you comes from CiteGraph's scans, which ask the buyer questions on four engines. The crawler view shows the steps before that: indexed by Google, crawled by AI search, read by an assistant for an answer.
What do you store?+
The page path, the crawler's user agent, which company and kind of crawler it is, whether its address checked out, the status code when your server passes it, and the time. IP addresses are used for the check and not stored. Human visitors are never sent to us.
Can CiteGraph's own scans show up as visits?+
They can: when we ask an engine about your category, it may open your pages to answer. Visits from those engines during a scan of your site are marked and left out of the numbers, and shown as during our scan in the log.
Using a coding agent? Connect CiteGraph over MCP and it installs this for you.
MCP server →