What this checker tests
AI search tools can only quote pages their crawlers are able to fetch. Two separate things decide that, and most robots.txt checkers only look at the first one.
- Your robots.txt rules. We fetch
/robots.txt, find the group that applies to each crawler's user-agent token, and apply the most specific matching Allow or Disallow rule to the path you entered. That is the matching method the robots.txt standard (RFC 9309) describes and the major crawlers follow. - What your server actually answers. We request the same page with a normal browser user agent and with the published user agents of four AI crawlers. If the browser gets a
200but a crawler gets a403,429or503, something in front of your site, such as a firewall, CDN bot protection, a security plugin or your host, is refusing AI crawlers no matter what robots.txt says.
The second test is a strong signal, not a guarantee. Some protection services verify real crawlers by IP address, so a crawler can be treated differently from our test request. If you see a mismatch, check your firewall or CDN logs for requests from the real crawler.
Training crawlers, search crawlers and user fetchers
"Blocking AI" is not one switch. Each AI company runs several crawlers for different jobs, and blocking the wrong one can remove you from AI answers without protecting anything you care about.
Training crawlers
GPTBot (OpenAI), ClaudeBot (Anthropic), Meta-ExternalAgent, CCBot (Common Crawl) and Bytespider collect content that may be used to train models. Google-Extended and Applebot-Extended are not separate crawlers. They are control tokens you name in robots.txt to say whether content Google and Apple already crawl may be used for their AI models.
Search and answer crawlers
OAI-SearchBot finds pages for ChatGPT search, Claude-SearchBot does the same for Claude, and PerplexityBot builds Perplexity's index. These are the crawlers that decide whether your pages can be shown and linked in AI answers. Googlebot covers Google Search, including AI Overviews, and Bingbot covers Bing and Microsoft Copilot.
User-triggered fetchers
ChatGPT-User, Claude-User and Perplexity-User fetch a page when someone asks the assistant about it or pastes a link. Operators document their robots.txt handling for these agents separately, so read each operator's crawler documentation before relying on a rule for them.
How to read your results
Allowed with no rule means no Allow or Disallow line matched the path, so the crawler may fetch it. Blocked shows the exact rule responsible, for example Disallow: / in the * group. The group column matters: if you add a User-agent: GPTBot group, GPTBot follows that group only and ignores everything under User-agent: *. That catches a lot of people out when they add one bot-specific rule and accidentally unblock private folders for that bot.
Two robots.txt states are easy to misread:
- robots.txt returns 404 (or another 4xx). Crawlers treat the whole site as allowed.
- robots.txt times out or returns a 5xx error. Crawlers treat the whole site as disallowed. A broken robots.txt on an overloaded server can quietly hide an entire site from every compliant crawler.
Common setups (copy and adapt)
Allow AI search, opt out of model training
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: * Allow: /
OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot fall through to the * group and stay allowed, so your pages can still be found and cited.
Allow everything except private areas
User-agent: * Disallow: /wp-admin/ Disallow: /cart/ Allow: /wp-admin/admin-ajax.php Sitemap: https://example.com/sitemap.xml
Any bot-specific group you add later has to repeat the private-area rules, because a crawler with its own group no longer reads the * group.
When robots.txt says "allowed" but crawlers still can't get in
This is the failure the live test exists for. Typical causes we see:
- CDN or WAF bot protection set to challenge or block "AI bots" or unverified automated traffic.
- Security plugins with user-agent blocklists that include AI crawlers.
- Shared hosting limits. Some hosts rate-limit or block AI crawlers across every site on the account and answer with
429 Too Many Requests. Nothing in your robots.txt or CMS will show it. - Geo-blocking or login walls that apply to all non-browser traffic.
Fix these at the layer that returns the error. Changing robots.txt won't help.
What access alone doesn't tell you
Being crawlable is the entry ticket, not the result. Whether ChatGPT, Perplexity or AI Overviews actually cite a page depends on whether it answers the question clearly, whether the answer is easy to lift out of the page, whether the content renders without JavaScript, and how much the rest of the web trusts the site. If you want the next steps, read how to get your website cited by ChatGPT and what llms.txt is and whether you need one.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT?
Not by itself. GPTBot is OpenAI's training crawler. ChatGPT search relies on OAI-SearchBot, and ChatGPT-User fetches pages when a user asks about them. You can block training and still allow search by disallowing GPTBot while allowing OAI-SearchBot.
Does Google-Extended affect Google Search or AI Overviews?
Google documents that Google-Extended does not affect inclusion or ranking in Google Search. It controls whether your content is used for Gemini model training and grounding. AI Overviews are part of Search and follow Googlebot's rules.
How long until a robots.txt change takes effect?
Crawlers cache robots.txt, commonly for up to about a day, so expect changes to apply after their next fetch rather than instantly.
Is being crawlable enough to get cited by AI search?
No. Access is the first requirement. Whether an AI answer cites you also depends on whether your pages clearly answer the question, how well they are structured, and how much the wider web trusts your site.