AI Crawlers Explained: GPTBot, ClaudeBot and More

Published May 7, 2026 · ClickRadius Institute

AI-associated requests can represent different provider purposes: potential model training, automatic search indexing, or access initiated by a user. Those purposes do not share one control surface or one evidentiary meaning. Anthropic alone currently documents ClaudeBot, Claude-SearchBot, and Claude-User separately. Before changing policy or interpreting a log, identify the exact agent, read the provider’s current purpose statement, verify the request source, and record only what the request proves.

Three common roles — with provider-specific controls

These roles are a useful map, not a claim that every agent fits one immutable category. Use the provider’s current documentation for the exact token:

1. Training crawlers

These request content that may contribute to future model development. A training-policy request does not itself establish later model use, attribution, a link, or an answer outcome. Examples include GPTBot (OpenAI) and ClaudeBot (Anthropic). Google-Extended is a control token rather than a separate crawler and governs specified training and grounding uses without controlling Google Search inclusion.

2. Search-index crawlers

Providers use these agents for automatic search discovery, indexing, or result quality. Examples include OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity), Bingbot (Microsoft), and Googlebot (Google). Allowing a search crawler removes one documented access barrier; it does not prove indexing, answer use, or citation.

3. User-triggered fetchers

These may request content in response to a user action. Examples include ChatGPT-User (OpenAI), Claude-User (Anthropic), and Perplexity-User (Perplexity). A verified request establishes transport to the recorded URL; a user can supply a URL directly, so the event does not prove independent discovery, favorable ranking, citation, recommendation, or a later click.

A user-agent name identifies a documented request role. It does not identify the answer outcome, the provider’s private candidate set, or the human’s next action.— ClickRadius Institute

The major crawlers, one by one

OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User

According to OpenAI’s crawler documentation, GPTBot requests content that may be used for training, while OAI-SearchBot is the automatic crawler governing appearance in ChatGPT search answers. ChatGPT-User is used for certain user actions, is not an automatic crawler, and does not determine Search appearance; OpenAI says robots.txt may not apply to those user-initiated requests. The GPTBot and OAI-SearchBot settings are independent.

Anthropic: ClaudeBot, Claude-SearchBot, Claude-User

Anthropic documents three current agents: ClaudeBot collects content that could potentially contribute to future model training; Claude-SearchBot indexes content for search optimization and result quality; and Claude-User accesses websites for user-initiated requests. Anthropic describes independent robots.txt controls for all three. A policy or audit that checks only ClaudeBot and Claude-User cannot establish Claude search access.

Google: Googlebot and Google-Extended

Google-Extended is a robots.txt token controlling specified Gemini training and grounding uses. It does not control Google Search indexing — that is Googlebot — and Google says a Google-Extended preference does not affect Search inclusion or ranking. Keep the consent choice separate from Search eligibility.

Perplexity: PerplexityBot and Perplexity-User

Perplexity documents PerplexityBot for automatic search surfacing and linking and Perplexity-User for possible access in response to a user’s question. PerplexityBot follows robots.txt; Perplexity says its user-requested fetcher generally ignores it. Blocking the automatic crawler limits that route but does not prove the user-requested route or every possible citation is unavailable.

Microsoft: Bingbot

Microsoft documents Bingbot as Bing’s crawler for discovering and refreshing content in its searchable index. Evaluate its current purpose and controls before changing policy. A verified Bingbot request establishes access to the requested URL, not that a downstream Copilot answer used or cited the page.

The rest: Bytespider, CCBot, Meta-ExternalAgent, Amazonbot

Other agents require the same provider-specific review. Do not infer purpose, compliance, or business value from a familiar name or from aggregate CDN volume. Record the current documentation version, stated purpose, supported control, verified network, and observed response before deciding whether to allow, limit, or block a request class.

Do not infer rendering behavior from a user-agent purpose

The cited provider pages document why an agent requests content; they do not provide one universal JavaScript-execution contract for every product and request mode. Inspect both the origin response and, where a provider supplies one, the rendered or extracted result. Keeping material facts in an accessible server response is a robust publishing practice, but a raw-HTML check alone does not prove what an external system parsed, indexed, or used in an answer.

Verification: user agents lie

A user-agent header can be copied by an unrelated requester. Before drawing conclusions from logs or building rules:

1. Use the provider’s current verification method. OpenAI links JSON network ranges by agent, and Anthropic links a current source-IP list. Google documents IP-range matching or reverse-then-forward DNS verification. Bing documents reverse-then-forward DNS under search.msn.com and expressly does not publish a fixed crawler range.
2. Never allowlist by user-agent string alone. Pair the token with the applicable provider verification result.
3. Review the roster on a recorded cadence. New agents appear and documented purposes change. Preserve the documentation URL and review date with each policy decision.

Reading AI crawlers in your own logs

A first-pass access-log inventory can identify request tokens for later provider verification. On a typical Linux server:

grep -iE "gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|claude-user|perplexity|bingbot|googlebot|bytespider|ccbot|amazonbot|meta-external" \
  access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn

What the output tells you, pattern by pattern:

• Training-agent requests (such as GPTBot or ClaudeBot) show that the verified agent requested the recorded URLs. They do not prove the content entered a training dataset or affected a model.
• Automatic search-agent requests (such as OAI-SearchBot, Claude-SearchBot, or PerplexityBot) show that the documented search agent reached the recorded URLs. They do not prove indexing, ranking, answer use, or citation.
• User-requested agents (such as ChatGPT-User, Claude-User, or Perplexity-User) show an identified request associated with a user-action pathway. A user may have supplied the URL directly; the log does not prove independent discovery, that the page was considered a best answer, or that it appeared in the response.
• Requests that conflict with the recorded policy require investigation. Verify the network, effective policy version, propagation delay, intermediary caches, and response before labeling the request spoofing or non-compliance.

Compare origin logs with CDN or WAF evidence because an intermediary may see or block requests that never reach the origin. Preserve the classifier and verification method rather than assuming every vendor label is authoritative. The result should show who requested what, which layer responded, and whether the observed response matches the recorded policy.

A purpose-by-purpose policy

There is no universal allowlist. Make and document each decision separately:

• Automatic search access: decide whether each provider’s documented search crawler may request the public site. Record that access is only one prerequisite, not a citation promise.
• User-requested access: decide whether the provider offers an effective site-owner control and state its exact limit. OpenAI says robots.txt may not apply to ChatGPT-User; Perplexity says Perplexity-User generally ignores it; Anthropic documents a robots.txt control for Claude-User.
• Training and grounding: make a separate consent decision for each documented token. Do not describe a training opt-out as Search de-indexing.
• Infrastructure protection: rate-limit or block abusive traffic based on verified behavior and operational impact, while preserving why the control was applied.

The companion guide covers the syntax, edge cases, provider-specific restriction examples, and why no universal named-agent allow template is safe: robots.txt for AI crawlers.

A robots policy records which identified automated agents are permitted to request a site. Interpret it with the provider's current agent-purpose documentation and server logs; it does not prove discovery, indexing, answer use, citation, or market-wide visibility.— ClickRadius Institute

Frequently asked questions

If I block GPTBot, will my site disappear from ChatGPT search answers?

GPTBot is OpenAI’s training crawler; its setting is independent of Search. OAI-SearchBot controls automatic crawling for appearance in ChatGPT search answers. ChatGPT-User is used for certain user-initiated actions, is not automatic crawling, does not determine Search appearance, and robots.txt rules may not apply. None of these request types proves or guarantees a citation.

Which Anthropic agents should a site owner evaluate?

Anthropic documents three separate agents: ClaudeBot for content that may contribute to future model training, Claude-SearchBot for search indexing and optimization, and Claude-User for access initiated by a user’s request. Evaluate and record each policy independently.

What does an AI-associated request prove?

After the user agent and network are verified, the request proves that the identified agent requested that URL at that time and received the recorded response. It does not by itself prove discovery, ranking, favorable evaluation, answer use, citation, recommendation, referral, or a human visit.

The ClickRadius AI Readiness assessment can report crawl-policy and response evidence visible at the submitted URL. Interpret each token against current provider documentation; the assessment does not prove indexing, answer use, or citation. Start with the free assessment, or see plans on the pricing page.