Free AI Crawler Robots.txt Tester: GPTBot, ClaudeBot & More
An AI crawler robots tester reads your robots.txt file and reports whether AI crawlers like GPTBot, ClaudeBot, PerplexityBot, and Google-Extended are allowed or blocked from your site. You might be blocking the AI engines you want citations from — enter your domain below and test it now.
Fetches yourdomain.com/robots.txt once and evaluates it locally. Nothing is stored.
| Crawler | Used by / for | Verdict |
|---|
What does this tool check?
This tool fetches your live robots.txt file and evaluates it against 14 AI crawlers the way the crawlers themselves do: it finds the most specific User-agent group that matches each bot, applies the longest-matching Allow or Disallow rule for your chosen path, and reports the verdict. It covers the retrieval and training bots for every major AI engine — OpenAI's GPTBot, OAI-SearchBot, and ChatGPT-User; Anthropic's ClaudeBot, Claude-SearchBot, and Claude-User; PerplexityBot and Perplexity-User; Google-Extended, which controls Gemini training; plus Applebot-Extended, Amazonbot, cohere-ai, and Bytespider. Because the check runs against your real robots.txt rather than a copy you paste in, what you see here is exactly what each crawler sees when it visits your site today.
How to use it (3 steps)
- Enter your domain — the tool fetches your live robots.txt automatically.
- Review the verdict table: each AI crawler is marked allowed or blocked for the path you chose.
- Fix any accidental blocks by editing the matching User-agent group in robots.txt, then re-test.
Why one wrong robots.txt line erases your AI visibility
Every AI citation starts with a crawl. If GPTBot can't fetch your pages, ChatGPT can't cite them — no amount of great content or schema markup routes around a Disallow rule. The blocks are almost never deliberate: a developer copies a "block bad bots" list from a 2023 forum thread, a security plugin ships aggressive defaults, or a CDN's bot-protection setting quietly serves AI crawlers a 403. The site owner never notices, because human visitors and Googlebot are unaffected — the only symptom is silence in AI answers, which is easy to blame on everything else. I check crawler access first on every AI-visibility engagement for exactly this reason, and I find accidental blocks on roughly a third of the sites I audit. The engines fetch robots.txt frequently, so a fix today usually restores access within a day. Once your crawlers are green here, hand them a reading list with the llms.txt Generator, and check your brand's full signal stack with the AI Visibility Checker. My ChatGPT SEO guide covers each OpenAI bot in depth, and a free AI SEO audit will catch access problems robots.txt alone can't show, like firewall-level blocks.
FAQ
It tests 14 crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Amazonbot, cohere-ai, and Bytespider, against the rules in your robots.txt.
If you want citations and referrals from ChatGPT, Claude, Perplexity, and Gemini, allow their crawlers. Blocking makes sense only for content businesses that license their archives. Most local and service businesses lose visibility and gain nothing by blocking.
GPTBot collects training data, OAI-SearchBot indexes for ChatGPT search, and ChatGPT-User fetches pages live when a user asks. Blocking GPTBot alone doesn't stop search citations; blocking all three makes you invisible to ChatGPT entirely.
Find the User-agent group naming that bot and change Disallow: / to Allow: /, or delete the group so the bot falls through to your default rules. Changes take effect the next time the crawler fetches robots.txt, usually within a day.
No. Without a robots.txt, every crawler assumes full access, so AI bots can read your site. Adding one is still smart: it lets you welcome the crawlers you want, exclude junk bots, and point everything at your sitemap.
Last updated: August 9, 2026
Crawlers allowed? Now tell them what to read. Build your llms.txt in two minutes.
Create llms.txtCrawler access is one of four signal groups. Scan them all for your brand in one go.
Check my brand
8+ years in SEO, top 3% of Upwork freelancers, 100% Job Success Score. Crawler access is the first thing I check on every AI visibility audit — about a third of sites are blocking a bot they need.
Crawlers can reach you. Will they cite you?
Access is step one. Get a free AI SEO audit and I'll show you what the AI engines actually find when they read your site — and what to fix.