Before you block AI crawlers, name the token

Before you block AI crawlers, name the token you mean to refuse and the search crawler you still want to allow. A robots.txt file is a public request to automated clients, not a lock, and a blanket disallow aimed at every bot can also stop Googlebot from crawling pages you want in ordinary search [2][3]. The decision is which tokens control training, which control search, and how you will read the live file after it is published.

What the file actually is

A robots.txt file governs one host, one protocol, and one port. Rules on the apex host do not cover www, a subdomain, or another port [2]. The September 2022 Robots Exclusion Protocol, RFC 9309, requires the name `/robots.txt` in lowercase, UTF-8, and plain text [3]. Google's crawlers read that file before an automatic crawl. The protocol does not apply to some user-controlled Google fetches, such as feed subscriptions, or to some safety checks, such as malware analysis [2].

The file is a set of groups. A group starts with one or more `User-agent` lines, then `Allow` or `Disallow` rules. Field names are not case-sensitive. Path values are case-sensitive and must start with `/` [2]. For any one crawler, Google uses only the group with the most specific matching user agent and ignores the rest. A named group is not merged with `User-agent: `, and group order does not pick the winner [2]. The 2022 standard states the fallback: if nothing matches the product token, the crawler obeys `` when that group exists, and if neither exists, no rules apply [3].

A line written for "all bots" does not quietly add itself to a named group. A token you forgot does not get a special exception. It falls through to `*`, or to no rule at all.

Two limits come before any token. These rules are not access authorization [3]. Google's useful-rules page says not to use robots.txt to hide private content, and to use real authentication instead, because a disallowed URL can still be indexed without a crawl and because anyone can read the file [8]. Listing a private path publishes that path [3]. A disallow also does not, by itself, guarantee that a URL vanishes from search. Google cannot index the content of a page it is disallowed from crawling, but it may still index the URL and show it without a snippet [2]. Keeping a page out of results takes Google's indexing controls, not a robots.txt rule alone [2].

Googlebot and Google-Extended are different choices

`Googlebot` is the token for ordinary Google Search. Preferences addressed to it affect Google Search, including Discover and all Google Search features, plus products such as Google Images, Google Video, and Google News [1]. `User-agent: Googlebot` with `Disallow: /` is the wrong line if the site should stay findable there. Our search visibility work treats that search crawl as its own question, separate from a training opt-out.

Google-Extended is not a second Googlebot. It has no separate HTTP user-agent string. Crawling uses existing Google user-agent strings, and the robots.txt token is only a control [1]. You will not confirm this block by searching logs for the name Google-Extended.

Google calls it a standalone product token for a narrower choice: whether content Google already crawls may train future generations of Gemini models that power Gemini Apps and the Vertex AI API for Gemini, and whether that content may be used for grounding in Gemini Apps and in Grounding with Google Search on Vertex AI [1]. Grounding, in that definition, means providing content from the Google Search index to the model at prompt time [1]. The same page says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal [1]. Blocking it is not how you leave Google Search. Leaving it open is not a ranking trick.

The control can be path-scoped. Google's example allows one archived URL and disallows the rest of that directory [1]:

``` user-agent: Google-Extended allow: /archive/1Q84 disallow: /archive/ ```

When two path rules match, Google uses the longest path, and in a conflict it uses the least restrictive rule [2]. A short `Disallow: /` does not automatically beat a longer `Allow` for the same agent. Google also does not support `crawl-delay`. Supported fields are `user-agent`, `allow`, `disallow`, and `sitemap` [2].

OpenAI splits search from training

OpenAI documents two robots.txt tags and says each setting is independent. A site can allow `OAI-SearchBot` so it may appear in search results, and disallow `GPTBot` to indicate that crawled content should not train OpenAI's generative AI foundation models [4].

`OAI-SearchBot` is how sites get surfaced in ChatGPT search. Opting out means the site will not be shown in ChatGPT search answers, though OpenAI says it can still appear as a navigational link [4]. OpenAI recommends allowing that token, and requests from its published IP ranges, if you want those search results [4].

`GPTBot` crawls content that may be used to train foundation models. Disallowing it is the training signal [4]. The HTTP string includes a version OpenAI says may change, so the robots.txt token is `GPTBot`, not a frozen version [4].

If you allow both, OpenAI may use one crawl for both uses, to avoid fetching twice [4]. A training opt-out has to be the `GPTBot` line itself. OpenAI also says its systems can take about 24 hours to adjust after a robots.txt change [4]. The file being public is not the same minute as the new policy taking effect.

`ChatGPT-User` is not the search switch. When a person asks ChatGPT or a custom GPT something, the product may visit a page with that agent. OpenAI says this is not an automatic crawl, that robots.txt rules may not apply because a person started the action, and that this agent does not decide whether content appears in search [4]. Search opt-outs belong on `OAI-SearchBot` [4]. A `GPTBot` disallow does not, on OpenAI's page, promise that a user-triggered fetch will never happen.

Anthropic uses three robots

Anthropic's help center describes three robots that honor robots.txt, with different results [5].

`ClaudeBot` collects content that could contribute to training. Restricting it signals that the site's future materials should be excluded from Anthropic's training datasets [5]. The documented block is a group in the top-level file on every subdomain you want to cover [5]:

``` User-agent: ClaudeBot Disallow: / ```

A rule on the apex host does not travel to `www` or to a campaign subdomain [2][3][5].

`Claude-SearchBot` crawls to improve search result quality. Disabling it stops that indexing, which Anthropic says may reduce visibility and accuracy in user search results [5]. If people should be able to find the site through Claude's search, this token stays open. It is not the training token.

`Claude-User` fetches when a person asks a question. Anthropic says disabling it prevents retrieval in response to that query, which may reduce visibility for user-directed web search [5]. That is a different claim from OpenAI's note that robots.txt may not apply to `ChatGPT-User` [4]. Copy the rule from the operator you mean, not from the other one.

Anthropic also says blocking its IP addresses may not persistently guarantee an opt-out, because the block can stop the crawler from reading robots.txt [5]. The documented opt-out is the robots.txt group. The help article links a JSON address list so you can tell an Anthropic crawler from a stranger using the name [5]. Anthropic supports a non-standard `Crawl-delay` under `ClaudeBot` [5]. That still is not a Google rule [2].

Common Crawl is an archive choice

CCBot is the crawler for Common Crawl, a nonprofit that publishes an open repository of web data for anyone to analyze [6]. A `CCBot` rule does not decide Google Search. The CCBot page says the crawler identifies itself as `CCBot/2.0`, and that this group prevents a crawl [6]:

``` User-agent: CCBot Disallow: / ```

That is a choice about future collection into an open archive. The page does not say Common Crawl trains a model. It says the repository is there to be analyzed [6]. If a public page should stay out of that archive, name `CCBot`. If the page is ordinary public material, many sites leave the token alone. Write the choice down so a later star disallow does not make it by accident.

Common Crawl warns that other clients falsely identify themselves as CCBot [6]. The real crawler uses published IP ranges, with reverse DNS on IPv4. IPv6 reverse DNS is not supported yet on that page, and the ranges are also published as JSON [6]. A log line that merely contains the letters CCBot is not proof.

What a blanket disallow catches

This is the short file many templates suggest:

``` User-agent: * Disallow: / ```

Google's create guide says an asterisk matches all crawlers except the various AdsBot crawlers, which must be named on their own [7]. The useful-rules page repeats that AdsBot exception on a full-site disallow, and it warns that URLs may still be indexed even when they were not crawled [8].

For everyone else, `*` is the default. Googlebot follows it when no more specific Googlebot group exists, because the named group and the star group are not combined [2]. `GPTBot`, `OAI-SearchBot`, `ClaudeBot`, `Claude-SearchBot`, and `CCBot` follow it too, unless each has a group of its own. A star disallow is how a "block the AI bots" edit also blocks ordinary search crawling, and how a search bot you meant to keep gets refused because you never wrote its name.

If ordinary search should continue, give `Googlebot` its own group that allows the public paths. If ChatGPT search or Claude search should continue, give those search tokens their own allow groups. Put `Disallow: /` only on training tokens you have decided to refuse, such as `GPTBot`, `ClaudeBot`, or `Google-Extended`. Leave `CCBot` as its own yes or no.

The other extreme is an empty `Disallow` under `User-agent: *`. Google says that explicitly allows crawlers and matches having no robots.txt file, or using `Allow: /` [8]. Search Console's report says a missing file is fine and means Google can crawl the site's URLs [9]. Adding a file is optional. Adding a careless one is not.

A star disallow is also a poor login wall. The file is public, disallowed URLs might still be indexed, and the protocol is not access control [3][8]. Accounts, drafts, and checkout belong behind authentication.

Check the live file, then wait for the cache

Save UTF-8 plain text from a text editor. Google's create guide says not to use a word processor, because curly quotes and similar characters can break parsing [7]. Put the file at the root. A copy in a subfolder does not govern the host [2][3].

Fetch the URL a crawler will fetch. Google says to open a private window on the robots.txt address and confirm the contents [7]. Repeat that for the apex name, for `www` if it answers on its own, and for each subdomain, and for both `http` and `https` if both respond. Search Console checks both schemes for the hosts it lists, and these files are not inherited across subdomains or parent domains [9]. A direct download is the same test [10]:

``` curl https://example.com/robots.txt -o robots.txt ```

Compare those bytes with the file you meant to publish. A host panel, a CDN, or an error page can serve something else. On that response, confirm the `Googlebot` group still allows the public paths you want in search, because a star `Disallow: /` will not merge into a named group [2]. Confirm each training token is spelled the way the operator spells it (`Google-Extended`, `GPTBot`, `ClaudeBot`, `CCBot`). A near miss will not match. If the star group disallows the site, search tokens you meant to keep need their own allow groups. Path values are case-sensitive, so `Disallow: /Private/` does not disallow `/private/` [2].

Then compare the live copy with what Google last stored. The robots.txt report lists files for the top 20 hosts, the last crawl time, and warnings or errors [9]. Errors prevent a rule from being used. Warnings do not [9]. You usually do not need to request a recrawl, because Google recrawls these files often [9]. Ask for one after a fetch failure or after you unblock important URLs, and do not treat that request as a promise that those URLs will be recrawled immediately [9].

Google generally caches robots.txt for up to 24 hours, longer when a refresh fails [2]. In ordinary crawling it updates that cache about every 24 hours [10]. OpenAI's window is about 24 hours as well [4]. The live file can be right today while an operator is still using yesterday's copy.

A broken robots.txt URL is not a block. Google treats `4xx` responses, except `429`, as if no file existed, which means no crawl restrictions [2]. Codes `401` and `403` on that URL do not slow crawling [2]. A server error, DNS failure, or timeout makes Google pause and retry, then fall back to the last good copy, or assume there are no restrictions if it has no copy [2]. If the failure continues and the rest of the site is up, Google says it will behave as if there is no robots.txt file [2].

Do not trust the user-agent string alone. Google says that string can be spoofed and points people to reverse DNS plus published IP ranges [1]. Common Crawl warns about fake CCBot clients [6]. Anthropic's address list identifies its crawlers. It is not a substitute for the robots.txt opt-out [5]. OpenAI publishes ranges beside the token definitions [4]. None of that turns the file into access control [3].

Practical takeaways

• Keep `Googlebot` (Google Search crawling) separate from `Google-Extended` (future Gemini training and grounding, with no effect on Search inclusion or ranking) [1].
• Keep search tokens (`OAI-SearchBot`, `Claude-SearchBot`) separate from training tokens (`GPTBot`, `ClaudeBot`). Allowing both OpenAI bots can let one crawl cover both uses [4][5].
• Do not assume a training disallow covers `ChatGPT-User` or `Claude-User`. Read each operator's own limit [4][5].
• Decide `CCBot` on its own. It fills an open archive, not Google Search [6].
• Do not use `User-agent: *` and `Disallow: /` as a shortcut for blocking AI crawlers. That group is the default, including for Googlebot if you never named it. AdsBot must be named on its own [2][7].
• Fetch `/robots.txt` on every host, compare it with Search Console's last copy, and allow about a day for Google and OpenAI to refresh [2][4][9].
• Put private URLs behind a login. The file is public, and a disallow can still leave a bare URL visible [3][8].

How we can help

Have more questions or want to get in touch? We can walk through the live robots.txt file with you, name the training tokens and the search tokens apart, and check that the copy on the server matches the decision. Start at our contact page.

Citations

1. Google for Developers, "List of Google's common crawlers" (2026-07-14)
2. Google for Developers, "How Google interprets the robots.txt specification" (2026-08-31)
3. RFC Editor, "RFC 9309: Robots Exclusion Protocol" (2022-09)
4. OpenAI, "Overview of OpenAI Crawlers" (2026)
5. Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?" (2026)
6. Common Crawl, "CCBot" (2026-10-12)
7. Google for Developers, "Create and submit a robots.txt file" (2025-11-21)
8. Google for Developers, "Useful robots.txt rules" (2026-06-12)
9. Google Search Console Help, "robots.txt report" (2026)
10. Google for Developers, "Update your robots.txt file" (2025-11-21)