← ClickRadius Institute

Should I Block AI Crawlers?

Published June 16, 2026 · ClickRadius Institute

For most businesses, the answer is no. If your goal is to be found, recommended and cited by AI assistants, blocking crawlers like GPTBot, ClaudeBot, PerplexityBot and Google-Extended is self-defeating — an AI engine cannot cite a page it was never allowed to read. There are legitimate exceptions, and they are worth knowing precisely: paywalled or licensed content, proprietary data, and pages where privacy or regulation argues against wide reuse. But for the ordinary marketing, service and educational pages that make up most of a business website, a blanket block trades away visibility to solve a problem you probably do not have. This article explains what each crawler actually does, how the robots.txt mechanics work, and how to make the call for your own site with clear eyes.

Why this question is suddenly everywhere

Two years ago almost nobody thought about AI crawlers. Then, through 2024 and 2025, a wave of publishers and site owners added Disallow rules for GPTBot and its peers, driven by a reasonable instinct: "these companies are using my content to train models I do not control, and I am not being paid for it." That instinct is legitimate for some businesses. The problem is that the block-everything reflex spread far beyond the businesses it fit, carried along by headlines about newspapers suing AI companies.

The context changed decisively at Google I/O in May 2026. VP of Search Elizabeth Reid called the new experience "the biggest upgrade to our Search box in over 25 years," and Sundar Pichai called it "our biggest upgrade to Search ever." AI Mode — powered by Gemini — became the default search experience rather than an experiment, and AI Overviews now appear on roughly 48% of queries, up from about 15% earlier in the year. Increasingly, the answer a searcher sees is composed on the results surface itself, assembled from sources the engine has read and chosen to trust.

In that world, the decision to allow or block AI crawlers stops being a philosophical stance about training data and becomes a concrete visibility decision. The question is no longer "do I want my content used?" It is "do I want to be one of the sources the answer is built from, or do I want the answer built without me?"

The old fear was that AI would copy your content. The newer, larger risk for most businesses is that AI will describe you without ever having read you — using competitors' pages, directories and reviews as its only sources.

— ClickRadius Institute analysis

What "blocking AI crawlers" actually means

An AI crawler is an automated agent that fetches web pages on behalf of an AI company. Different crawlers do different jobs, and conflating them is the single most common mistake in this whole topic. Broadly there are three functions:

You control these through your site's robots.txt file — a plain-text file at the root of your domain that names user-agents and tells them which paths they may or may not fetch. The reputable AI crawlers document their user-agent tokens and state that they honor robots.txt. A block is therefore a request the well-behaved bots comply with; it is not a technical wall. We cover the syntax in depth in Robots.txt for the AI Era.

The major AI crawlers, and what each one does

Get the specifics right, because the correct decision differs by bot. The details below reflect how these operators have publicly documented their crawlers; always confirm against each company's current published documentation, since tokens and behaviors change.

GPTBot (OpenAI)

GPTBot is OpenAI's crawler for gathering web content, associated primarily with training data collection. OpenAI publishes its user-agent and IP ranges and states GPTBot obeys robots.txt. Separately, OpenAI operates user-triggered fetchers for ChatGPT's browsing and search features, which retrieve pages in response to a live user request. Blocking GPTBot limits OpenAI's ability to ingest your site broadly; it does not necessarily stop every live-retrieval fetch, and it does not stop ChatGPT describing you from third-party sources.

ClaudeBot (Anthropic)

ClaudeBot is Anthropic's web crawler. Anthropic documents its crawler user-agents and states they respect robots.txt directives, and it publishes guidance for site owners who want to allow or disallow it. As with OpenAI, there is a distinction between broad crawling and fetches made when a user asks Claude to look something up.

PerplexityBot (Perplexity)

Perplexity is fundamentally an answer engine that cites live sources, so its crawler is closely tied to whether your pages can appear as cited references in Perplexity answers. Because Perplexity's core product surfaces source links prominently, being crawlable here is unusually directly connected to visible, clickable citations.

Google-Extended (Google)

This is the one most often misunderstood. Google-Extended is not Googlebot. Google-Extended is a separate robots.txt token that governs whether your content is used to train Gemini models and to power certain generative features. Crucially, blocking Google-Extended does not remove you from Google Search, and it does not remove you from AI Overviews or AI Mode, because those features are grounded in the normal search index that Googlebot builds. If you block Googlebot itself, you disappear from Search entirely — almost never what a business wants.

Google-Extended lets a publisher decline Gemini training while remaining fully indexed and eligible for AI Overviews. Confusing it with Googlebot has led site owners to think they must choose between "AI" and "Search." They do not.

— ClickRadius Institute analysis

Others worth knowing

Bing's crawler (Bingbot) feeds Microsoft Copilot as well as Bing search, so the same allow/block logic that governs your search presence governs Copilot exposure. Meta and Amazon operate their own AI crawlers with their own tokens. The reputable operators publish their user-agents; a few crawlers in the wild ignore robots.txt entirely, which is a reason to rely on server-side controls rather than robots.txt alone when a block genuinely matters.

Why blocking usually backfires for visibility-seeking businesses

Here is the mechanism, plainly. When someone asks an assistant "who does commercial roofing in Tampa" or "what's the best CRM for a small law firm," the engine assembles an answer from the sources it can access and trust. If your site is blocked, your carefully written pages are not in that pool. But the answer still gets written — from directory listings, review sites, aggregators, press mentions and your competitors' pages, all of which describe your category and often describe you.

The result is the outcome nobody wants: the AI talks about your market, maybe even names you, using everyone's framing except your own. You have surrendered the ability to be the primary, authoritative source for your own business while gaining essentially nothing, because the third-party sources you cannot control were never covered by your robots.txt in the first place.

The stakes here are set by how search behaves now. Industry trackers put zero-click searches — those that end without a click to any website — at roughly 60% overall, up from about 45%, and near 93% within AI Mode. Position-#1 click-through rates have fallen from roughly 27% to about 11% as answers move onto the results surface. When the majority of searches never produce a click, being the cited source inside the answer is the visibility. Refusing to be crawlable in that environment is refusing to be in the running.

This is also why on-site access is necessary but not sufficient. The majority of what drives AI citations, by third-party estimates, is off-site — entity building, directory presence, reviews and multi-platform signals. Allowing crawlers simply keeps you eligible to compete; it does not by itself win the citation. We treat the off-site side in How Large Language Models Use Training Data and the retrieval side in Grounding and Why AI Cites Live Sources.

The legitimate exceptions

Blocking AI crawlers is the right call in specific, identifiable situations. If any of these describe your content, a targeted block is defensible — and often prudent.

  1. Paywalled or subscription content. If people pay for access to your articles, courses or reports, letting a crawler ingest them undermines the product. Publishers with a subscription model have a straightforward commercial reason to disallow broad AI crawling of the paid material.
  2. Proprietary data and licensed research. Original datasets, priced research reports, and unique reference material you monetize or license are assets. You may prefer to license them to AI companies on your terms rather than give them away through an open crawl.
  3. Personal, legal, health or regulated information. Pages containing personal data, protected health information, client details or content under regulatory constraints can warrant blocking on privacy and compliance grounds, independent of any visibility calculus. Here the question is not "will this help my marketing" but "should this be freely reusable at all."
  4. Staging, internal, and utility paths. Login pages, admin areas, cart and checkout flows, search-result pages and staging sites should not be in any answer engine. These are already standard robots.txt disallows and simply extend to AI crawlers.

Notice the common thread: in every legitimate case, the content is not meant to be a public, freely answerable resource. That is the actual decision criterion — not "is it AI?" but "do I want this content to be an openly reusable answer?"

How to make the decision — and implement it cleanly

Rather than an all-or-nothing switch, treat crawl access as something you tune by path and by bot. A pragmatic approach:

  1. Default to allow for public marketing and educational content. Service pages, product pages, articles, guides, FAQs, location pages — the content whose entire purpose is to be found — should be crawlable by the reputable AI engines.
  2. Block specific bots on specific paths where an exception applies. Disallow the AI user-agents on your paywalled directory, your licensed-data endpoints, or your regulated content — not across the whole domain.
  3. Separate training consent from search presence. If your concern is model training rather than visibility, use the training-scoped control (for example Google-Extended) so you can decline training while remaining fully indexed and citable. This is the nuance most blanket blocks miss.
  4. Never disallow the search-index crawlers you depend on. Blocking Googlebot or Bingbot removes you from search and, by extension, from AI Overviews, AI Mode and Copilot. That is a different and far more costly decision than blocking a training crawler.
  5. Use server-side controls for blocks that truly matter. robots.txt is a request, not enforcement. For content you must protect, authentication and server-level rules — not a polite line in a text file — are what actually stop access.

A minimal robots.txt that allows reputable AI crawlers on public content while protecting a paid section might disallow the AI user-agents only on /members/ and /reports/, leave the rest open, and keep Googlebot and Bingbot fully allowed sitewide. The point is precision: match the rule to the content, not to a mood about AI in general.

A note on control you do not have

It is worth being honest about the limits of robots.txt as a lever. It governs only the crawlers that choose to obey it, and it governs only your own site. It does nothing about the third-party sources — directories, review platforms, news, Wikipedia and social — that AI engines also read and that often carry more weight than your own pages. That is why blocking your site rarely achieves what people hope: the information ecosystem around your brand is far larger than the pages you control.

The constructive version of this realization is that your energy is better spent making your crawlable content the most accurate, complete and citation-worthy source about your business, and making the off-site signals consistent, than on trying to wall yourself off. Being the best available source is a stronger position than being an absent one. ClickRadius exists partly to make that first step measurable — every scan checks whether the reputable AI crawlers can actually reach and read your key pages, because access is the precondition for everything downstream.

robots.txt is a fence around your own yard. It does nothing about what the neighborhood says about you — and in AI search, the neighborhood is doing most of the talking.

— ClickRadius Institute analysis

Frequently asked questions

Does blocking GPTBot stop my brand from appearing in ChatGPT answers?

Not entirely, but it changes how. GPTBot controls whether OpenAI can crawl your site for training and, in some products, for retrieval. Blocking it removes one path by which your own words reach the model. But ChatGPT with browsing and other engines can still describe you from third-party sources — directories, reviews, news and Wikipedia — so a block often means AI describes your business using everyone's words except yours, which is the worst of both worlds if visibility is your goal.

What is the difference between Google-Extended and Googlebot?

They are separate controls. Googlebot crawls for Google Search indexing, including AI Overviews and AI Mode, which are grounded in the normal search index. Google-Extended is a separate token that governs whether your content trains Gemini models and feeds certain generative features. Blocking Google-Extended does not remove you from Search or from AI Overviews. Blocking Googlebot would remove you from Search entirely, which almost no business wants.

When does it actually make sense to block AI crawlers?

When the content is not meant to be publicly answerable: paywalled or subscription articles you sell access to, proprietary research or data you license, and pages carrying personal, legal or health information where privacy or regulation argues against wide reuse. In those cases blocking specific AI user-agents on specific paths is reasonable. For ordinary marketing pages, service descriptions and educational content where the goal is to be found and cited, blocking works against you.

Not sure whether AI engines can even reach your most important pages? A free AI Readiness Score checks crawl access alongside the other signals that decide whether you get cited, and the pricing page shows how ongoing monitoring keeps that access — and your visibility — intact.

Related from the Institute