Should you block GPTBot and other AI crawlers?
Block the training crawlers if you don't want your pages used to train AI models, and keep the search crawlers allowed if you want to be shown and linked in AI answers. They are separate switches: OpenAI says blocking GPTBot only signals that your content shouldn't be used for training, while OAI-SearchBot decides whether you can appear in ChatGPT search, and Google says Google-Extended does not affect your inclusion or ranking in Google Search.
By Jarek Piotrowski · Last updated
Search crawlers and training crawlers, side by side
| Company | Allow to be shown in answers | Block to opt out of training |
|---|---|---|
| Googlebot (Search, AI Overviews and AI Mode) | Google-Extended: "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" (Google) | |
| OpenAI | OAI-SearchBot (ChatGPT search) | GPTBot: "Each setting is independent of the others" (OpenAI) |
| Anthropic | Claude-SearchBot; blocking it "may reduce your site's visibility and accuracy in user search results" | ClaudeBot (Anthropic) |
| Perplexity | PerplexityBot, "designed to surface and link websites in search results on Perplexity" | None needed: Perplexity says PerplexityBot "is not used to crawl content for AI foundation models" (Perplexity) |
The block you didn't choose
Not every block is in robots.txt. Security plugins, hosts and CDNs have their own switches for AI crawlers. Cloudflare, for example, has a setting that "blocks verified bots that are classified as crawling for the purpose of AI training", and says that from 15 September 2026 it sets updated defaults for new domains, with training and agent bots blocked on pages that display ads and search crawlers left allowed (Cloudflare docs). Check what your own setup does rather than assuming.
What robots.txt can't control
When a person asks an AI tool to read your page, the visit comes from a user agent such as ChatGPT-User or Perplexity-User. OpenAI says "robots.txt rules may not apply" to ChatGPT-User because a user started it, and Perplexity says Perplexity-User generally ignores robots.txt rules. Anthropic says blocking Claude-User stops Claude retrieving your content for a user's question (Anthropic).
The steps
- Decide on training separately from search. Choose whether you mind your pages being used to train AI models. That choice is separate from whether you want to be shown in AI answers.
- Read your robots.txt. Look for User-agent groups naming GPTBot, OAI-SearchBot, Google-Extended, ClaudeBot, Claude-SearchBot or PerplexityBot, and for a Disallow under User-agent: *.
- Check the firewall too. Security plugins, CDNs and hosts can block AI crawlers by name without touching robots.txt. Check their bot settings.
- Allow the search crawlers. Keep OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot allowed if you want to be linked in their answers. Block the training crawlers if you choose to.
- Re-check after a day. OpenAI and Perplexity both say changes can take up to about 24 hours to reach their systems. Run the check again after that.
What our checks can and can't tell you
The free AI crawler checker reads your robots.txt for 13 search and AI crawlers and tests your homepage with four AI crawler names, so you can see which switches are on. We don't publish how many sites block AI crawlers. In our Australian AI Overview Index (scan of 23 Oct 2026: 40 Australian questions across five industries, 3 runs each), Google showed an AI Overview in 119 of 120 runs, each listing about 7 to 10 sources. Google's rule is that a link in an AI Overview must be indexed and able to show a snippet, so a page Googlebot can't crawl can't be one of them.
Questions people ask
Does blocking GPTBot remove me from ChatGPT search?
Not on OpenAI's description: GPTBot is the training crawler and OAI-SearchBot is the search one, and each setting is independent (OpenAI).
Does blocking Google-Extended hurt my Google ranking?
Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" (Google).
How long do changes take?
OpenAI says about 24 hours for ChatGPT search, and Perplexity says up to 24 hours.
Should I block everything to be safe?
Only if you don't want to be shown in AI answers at all. Blocking the search crawlers removes the chance of being linked; blocking only the training crawlers keeps it.
If you would rather have it done, SEO and AI-answer visibility covers this work every month (from $2,500 a month), measured in Google's AI Overviews and ChatGPT.
Jarek Piotrowski runs LayerOps from Canberra: fast, lead-ready websites, and SEO and AI-answer visibility for professional and service businesses Australia-wide.