Across the sites we run · 110,319 Google impressions · 2,643 clicks since May 2026 · Cited in AI answers on 36% of tracked questions 110k impressions · 2.6k clicks · AI-cited on 36% of questions
LayerOps Book a call Book a call
Free tool · no account

Can AI answers read your site? Check your AI crawler access.

An AI answer can only cite a page its crawler is allowed to read. Google says a page needs to be indexed and eligible for a snippet to appear as a link in AI Overviews (Google Search Central), and OpenAI says sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers (OpenAI). This checker reads your robots.txt for 13 crawlers and tests your homepage for firewall blocks, then tells you what to change.

Checks
robots.txt for 13 crawlers, plus a firewall test with 4 crawler user-agents
Takes
About 5 to 10 seconds
Cost
Free, no account, nothing stored about you
Limit
10 checks an hour per connection

Check a domain

Just the domain, like yourbusiness.com.au. We check https:// only.

The 13 crawlers it checks

Names as each company documents them (checked 23 October 2026). Search crawlers decide whether you can be shown and linked in that company's answers; training crawlers only decide whether your pages are used to train models. Blocking training crawlers does not, on these companies' own descriptions, remove you from their search answers.

Crawler names, owners and what they do, from the owners' documentation (23 October 2026)
CrawlerCompanyKindWhat it does
GooglebotGoogleSearchGoogle Search, including AI Overviews and AI Mode
Google-ExtendedGoogleTrainingGemini training and grounding. Not used for Search and not a ranking signal
BingbotMicrosoftSearchBing search
OAI-SearchBotOpenAISearchShows sites in ChatGPT search results
ChatGPT-UserOpenAIUser requestVisits a page when a ChatGPT user asks; robots.txt may not apply
GPTBotOpenAITrainingTraining OpenAI models
Claude-SearchBotAnthropicSearchImproves Claude search results
Claude-UserAnthropicUser requestVisits a page when a Claude user asks
ClaudeBotAnthropicTrainingTraining Anthropic models
PerplexityBotPerplexitySearchShows sites in Perplexity search results
Perplexity-UserPerplexityUser requestVisits a page when a Perplexity user asks; generally ignores robots.txt
Applebot-ExtendedAppleTrainingApple model training. It does not crawl; it controls use of what Applebot fetched
CCBotCommon CrawlArchiveThe Common Crawl open web archive

How to fix a blocked crawler

  • Blocked in robots.txt. Find the User-agent group that names the crawler (or the User-agent: * group if none does) and remove the Disallow: / line, or add Allow: /. The longest matching rule wins, and Allow wins a tie (RFC 9309). Keep blocks on training crawlers if you want to: that is a separate choice from being cited (which AI crawlers to block, and which to allow).
  • Blocked at the server or firewall (HTTP 403, 429 or 503, or a "checking your browser" page). Look at your host's or CDN's bot settings: Cloudflare, Wordfence, Sucuri and managed WordPress hosts all have switches that block AI crawlers by name. Allow the search crawlers you want to be cited by.
  • robots.txt errors. If robots.txt answers a server error, Google pauses crawling for up to 12 hours and then works from its last good copy for up to 30 days (Google's robots.txt rules). A missing robots.txt (404) is fine: it allows everything.
  • Googlebot. We don't send a fake Googlebot request, because well-run sites block fake Googlebots. Use URL Inspection in Google Search Console to see what the real Googlebot gets.

How the check works, and its limits

  • robots.txt. We fetch https://yourdomain/robots.txt once and apply the standard rules (RFC 9309) to your homepage for each crawler name. Other pages may have their own rules; the result lists how many Disallow lines apply to each crawler.
  • Firewall test. One request to your homepage with our own user-agent, then one each with the OAI-SearchBot, GPTBot, ClaudeBot and PerplexityBot user-agents. A block shows that your site refuses that name from an unknown address. The real crawler, coming from its company's published addresses, may be treated differently.
  • What we keep. The result for a domain is cached for an hour so repeat checks are instant. We don't store your IP address or log what you check; the rate limit uses a one-way hash that expires within two hours.
  • What we don't do. No AI is used to produce the result, and it isn't a full audit: it doesn't check whether your pages are indexed or cited. For that, there's the free AI answer check.
  • Next step. Want this checked every week instead of once? See the dashboard (A$99, then A$29/month).

More free data and tools: which kinds of sites Google's AI Overviews cite (the monthly Index), and WordPress vulnerabilities this month.

Crawlers allowed in? See what the AI answers say.

The free AI answer check runs three of your customers' questions through Google's AI Overviews and ChatGPT. The AI Visibility Audit does it for 25 questions and three competitors.

Book a 15-minute call Free AI answer check

Or call 0404 003 240 jarek@layerops.tech. I reply within one business day.

Request a call back

Or ask for a call back

A phone number or an email is enough. I reply within one business day.

Jarek Piotrowski · Canberra-based, working with businesses Australia-wide. Your details go to me, and I don't share them. Privacy.