AI crawler checker
Enter a domain and see whether GPTBot, ClaudeBot, PerplexityBot and 13 more AI crawlers may access it, according to its robots.txt. Free, no signup, results in a few seconds.
robots.txt says who may come. XPmetric shows who actually did.
See which AI bots really request your pages, which pages they read and which return a 404, next to your human traffic and revenue. How AI crawler tracking works.
Three kinds of AI crawler
“AI crawler” covers three different jobs, and each provider gives each job its own robots.txt name. That is why you can block one and keep another.
AI search and answer engines
OAI-SearchBot, Claude-SearchBot, PerplexityBot and Bingbot build the indexes that ChatGPT search, Claude, Perplexity and Copilot answer from. Blocking them is the most direct way to drop out of AI answers and the links they cite.AI assistants fetching for a user
ChatGPT-User, Claude-User and Perplexity-User fetch one page because a person asked about it or pasted the link. Some providers say these user-triggered fetches may not follow robots.txt, because a person, not a crawler, asked for the page.AI training crawlers
GPTBot, ClaudeBot, CCBot, Bytespider, meta-externalagent and others collect pages to train models. Google-Extended and Applebot-Extended never crawl by themselves: they are tokens that tell Google and Apple whether pages they already crawl may be used for AI.
What blocking costs you
Blocking training crawlers is a reasonable choice if you do not want your writing in future models, and it usually costs you little in visibility today. Blocking search and answer crawlers is different: if an answer engine cannot read your pages, it cannot quote you or link to you when someone asks a question you answer.
The common mistake is a blanket rule. A User-agent: * group with Disallow: /, often left over from a staging setup, blocks every crawler that has no group of its own, search engines included. A crawler with its own group ignores the * group completely, so one specific Allow can undo a block you meant to keep.
Rules are matched by path. The longest matching rule wins, and if an Allow and a Disallow are equally long, Allow wins. This checker applies the same rules to your home page and shows the line that decided.
Common questions
Does blocking GPTBot keep my site out of ChatGPT?
Not entirely. GPTBot collects pages for model training. ChatGPT search uses OAI-SearchBot, and when a user pastes your link ChatGPT-User fetches it. Each has its own robots.txt token, so you can block training and still appear in search answers.
Does robots.txt actually stop AI crawlers?
Only the ones that choose to follow it. robots.txt is a request, not a lock. The major providers say their crawlers respect it, but a crawler that ignores it, or sends a browser user agent, is not stopped. Blocking that does work happens in your server, CDN or firewall.
Why does a crawler show as allowed when my firewall blocks it?
This tool reads robots.txt only. A Cloudflare bot rule, a WAF or a server rule can turn a crawler away even though robots.txt allows it, and this check does not test that. When robots.txt itself answers with a 4xx status other than 404 or 410, the checker reports it as not readable instead of guessing a score. A 403, 418 or 429 there is usually such a rule at work.
What does "Some paths blocked" mean?
The crawler may fetch your home page, but robots.txt disallows some paths for it, such as /admin or /blog. The paths are listed under the crawler. Excluding private areas is normal and does not lower the score, but check that none of your content pages are on the list.
Does Google-Extended affect my Google search ranking?
No. Google says Google-Extended only controls whether your content may be used for Gemini models. Googlebot still crawls and ranks your pages for Search either way.
Do I need an llms.txt file?
It is optional. llms.txt is a proposed plain-text index of your most useful pages for AI tools. No major provider has said it uses the file for crawling or ranking, so treat it as a cheap extra, not a requirement.
How do I see which AI crawlers actually visit?
robots.txt only says who may come. Your server logs, or XPmetric's AI crawler tracking, show which bots really requested which pages and which of those returned a 404.