← ClickRadius Institute

AI Crawlers Explained: GPTBot, ClaudeBot and More

Published May 7, 2026 · ClickRadius Institute

Open your server logs and you will find a crowd that did not exist a few years ago: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, Amazonbot, CCBot. Content-delivery networks now report AI crawlers among the most active automated visitors on the web — Cloudflare's published analyses have repeatedly placed GPTBot, ClaudeBot and Bytespider near the top of AI-bot traffic by request volume. Most site owners react with a blunt instrument: block them all, or ignore them all. Both are mistakes, because these bots do fundamentally different jobs, and the job determines whether blocking protects you or erases you from AI answers. Here is the taxonomy that makes the decisions easy.

The three kinds of AI crawler — and why the difference is everything

Every AI-associated bot falls into one of three functional categories:

1. Training crawlers

These collect content to train future models. Your content becomes part of a training corpus; there is no citation, no link, no traffic — the model may later reproduce knowledge derived from your pages without attribution. Examples: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (a robots.txt control token, not a separate crawler, governing use of your content for Google's AI training and grounding), Meta-ExternalAgent (Meta), Bytespider (ByteDance), CCBot (Common Crawl, whose corpus many labs train on), Amazonbot (Amazon, mixed-purpose).

2. Search-index crawlers

These build the retrieval indexes that AI answer engines search at question time. Content indexed here can surface as a live, linked citation. Examples: OAI-SearchBot (OpenAI's search index for ChatGPT's browsing/search answers), PerplexityBot (Perplexity's index), Bingbot (Microsoft's index, which also feeds Copilot), and Googlebot itself — the same crawler that powers classic Google Search also feeds Google's generative surfaces.

3. User-triggered fetchers

These fetch a specific page in real time because a human (or their agent) asked a question that made the assistant open your URL right now. Examples: ChatGPT-User (OpenAI), Claude-User (Anthropic), Perplexity-User (Perplexity). These are the closest thing AI has to a human visitor: a request on behalf of an identified user action, typically followed — when your page is useful — by a citation the user can click.

"Should I block AI bots?" is a malformed question. Training crawlers take without attribution; search-index crawlers and user-triggered fetchers are how you get cited at all. One robots.txt decision cannot be right for both.— ClickRadius Institute analysis

The major crawlers, one by one

OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User

OpenAI is the cleanest example of the three-way split, because it documents three distinct user agents for three distinct purposes. According to OpenAI's bot documentation: GPTBot gathers content that "may be used to train" its models; OAI-SearchBot builds the index behind ChatGPT's search features and is explicitly not used for training; ChatGPT-User fires when a user's action in ChatGPT requires fetching a page. OpenAI publishes IP ranges for verification. The practical consequence: a publisher can block GPTBot (no training) while allowing OAI-SearchBot and ChatGPT-User (yes citations) — and many major publishers do exactly that.

Anthropic: ClaudeBot and Claude-User

Anthropic documents ClaudeBot as its web crawler and Claude-User for user-initiated fetches, with robots.txt compliance and published guidance in its help center. The legacy anthropic-ai token also appears in many robots.txt files. As with OpenAI, the crawl/fetch distinction lets you choose training exposure and live-answer presence independently.

Google: Googlebot and Google-Extended

The most misunderstood pair. Google-Extended is a robots.txt token controlling whether your content is used for Gemini training and grounding. It does not control classic Search indexing — that is Googlebot — and per Google's crawler documentation, blocking Google-Extended does not remove you from Search results or from generative features built on the Search index, such as AI Overviews. The uncomfortable corollary: you cannot stay in Google's classic index while fully opting out of Google's generative surfaces; they drink from the same crawl.

Perplexity: PerplexityBot and Perplexity-User

Perplexity is citation-native — every answer links its sources — so its crawler and fetcher are, for most businesses, among the most straightforwardly beneficial to allow. Note that Perplexity has been publicly accused (notably in Cloudflare's 2024–2025 analyses and by publishers) of fetching via undeclared user agents in some circumstances; the dispute is unresolved, but it is a useful reminder that robots.txt is a compliance convention, not an enforcement mechanism.

Microsoft: Bingbot

There is no separate "Copilot bot" to manage: Microsoft Copilot's web answers draw on the Bing index. Blocking Bingbot removes you from Bing, Copilot answers, and the downstream products that license Bing results. Few businesses should ever do this. Getting into that index faster is a separate topic — see IndexNow and Instant AI Indexing.

The rest: Bytespider, CCBot, Meta-ExternalAgent, Amazonbot

Bytespider (ByteDance) has drawn criticism in CDN reports for aggressive crawling and inconsistent robots.txt behavior, and offers no citation upside for Western businesses today. CCBot feeds Common Crawl's open corpus — blocking it opts you out of many research datasets and smaller labs' training data in one line, but also out of whatever downstream visibility those corpora confer. Meta-ExternalAgent collects for Meta's model training. Amazonbot supports Alexa and Amazon's AI features. None of these currently drives meaningful cited-answer traffic for a typical business; they are candidates for blocking if training exposure concerns you.

What AI crawlers can't do: the JavaScript blind spot

A technical fact with large strategic consequences: Googlebot renders pages in an evergreen headless Chromium, but the documented AI crawlers from OpenAI, Anthropic and Perplexity fetch raw HTML and do not reliably execute JavaScript. If your prices, FAQs or service descriptions only exist in the DOM after a client-side framework hydrates, those pages are — to GPTBot, ClaudeBot and PerplexityBot — substantially blank. Industry crawl studies since 2024 have consistently flagged this gap. The fixes are server-side rendering or static generation of citation-critical content, and honest raw-HTML fallbacks for anything interactive. This is also why performance and rendering architecture appear in a technical GEO audit at all — the connection we detail in Core Web Vitals in AI Search.

Verification: user agents lie

Anything can put GPTBot in its user-agent header — scrapers routinely impersonate reputable bots to slip through allowlists. Before drawing conclusions from logs or building rules:

  1. Verify by IP. OpenAI publishes JSON files of official IP ranges for each of its bots; Anthropic and Microsoft publish ranges; Google supports reverse-DNS verification (host the IP, confirm it resolves to googlebot.com/google.com, then forward-confirm).
  2. Never allowlist by user-agent string alone. Match string and verified network.
  3. Watch your logs quarterly. The roster changes fast — new agents appear, purposes get re-documented. A quarterly log review of top automated user agents is cheap situational awareness. Rate-limiting misbehaving verified bots at the CDN is legitimate; silent 403s for bots you want citations from is self-harm you will never see reported anywhere.

Reading AI crawlers in your own logs

Ten minutes with your access logs turns this taxonomy from theory into a picture of your actual situation. On any Linux server:

grep -iE "gptbot|oai-searchbot|chatgpt-user|claudebot|claude-user|perplexity|bingbot|google-extended|bytespider|ccbot|amazonbot|meta-external" \
  access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn

What the output tells you, pattern by pattern:

Sites behind Cloudflare or a similar CDN should pull the same report from the CDN's bot analytics instead, for two reasons: the CDN sees requests it blocks before your origin ever logs them, and its verified-bot classification has already done the IP-validation work for the major crawlers. Either way, the deliverable is the same — a one-page quarterly summary of who came, who was turned away, and whether that matches the policy you think you have.

A sane default policy

For a business that wants AI visibility (rather than a publisher licensing content):

The full syntax, edge cases and a copy-ready file are in the companion piece, robots.txt for the AI Era.

Your robots.txt has quietly become a business-development document. It decides which answer engines are allowed to discover that you exist — and industry estimates suggest a large majority of brands currently appear in none of them.— ClickRadius Institute analysis

Frequently asked questions

If I block GPTBot, will my site disappear from ChatGPT answers?

Blocking GPTBot opts your content out of OpenAI's model-training corpus. ChatGPT's live search answers rely on the search-index and user-triggered fetchers — OAI-SearchBot and ChatGPT-User — which OpenAI documents separately. Blocking those is what removes you from live answers and citations, so decide per bot, not per company.

Do AI crawlers execute JavaScript?

Generally no. Unlike Googlebot, which renders pages in a headless Chromium, the documented AI crawlers from OpenAI, Anthropic and Perplexity fetch raw HTML and do not reliably execute JavaScript. Content that only exists after client-side rendering is effectively invisible to them — a strong argument for server-side rendering of anything you want cited.

How do I know a request claiming to be GPTBot is genuine?

User-agent strings are trivially spoofed. The major operators publish verification mechanisms — OpenAI and Anthropic publish official IP ranges, and Google supports reverse-DNS verification for its crawlers. Verify by IP against published lists before making decisions from log data, and never whitelist by user-agent string alone.

ClickRadius checks your crawl configuration against the current AI-bot roster — who you are blocking, who you are inviting, and what that means for citations across the five live engines it monitors. Start with your free AI Readiness Score, or see plans on the pricing page.

Related from the Institute