AI Crawler Robots.txt Tester
Paste robots.txt and check whether ChatGPT, Claude, Perplexity, Google, Bing, Grok/xAI, Kimi, DeepSeek, Qwen, Doubao, Brave, Kagi, You.com, and other AI/search crawlers can fetch your public discovery paths.
Reviewed May 25, 2026. Privacy model: tool input is processed in your browser and is not uploaded to BytePane servers.
Crawler access matrix
Allowed means the strongest matching rule permits that path. Blocked means a more specific Disallow rule wins.
| Crawler | Type | / | /ai-answers/ | /llms.txt | /ai-index.json | /sitemap.xml | /blog/example/ | /admin/ | /api/private/ |
|---|---|---|---|---|---|---|---|---|---|
Googlebot Core Google Search crawler. Blocking this can directly damage Google indexing. | search | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Blocked DISALLOW /admin/ on line 4 is the most specific matching rule. | Blocked DISALLOW /api/ on line 5 is the most specific matching rule. |
Bingbot Microsoft Important for Bing, Copilot grounding, and secondary discovery paths. | search | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Blocked DISALLOW /admin/ on line 4 is the most specific matching rule. | Blocked DISALLOW /api/ on line 5 is the most specific matching rule. |
OAI-SearchBot OpenAI OpenAI crawler associated with search/discovery surfaces, not the same intent as GPTBot. | search | Allowed ALLOW / on line 9 is the most specific matching rule. | Allowed ALLOW / on line 9 is the most specific matching rule. | Allowed ALLOW / on line 9 is the most specific matching rule. | Allowed ALLOW / on line 9 is the most specific matching rule. | Allowed ALLOW / on line 9 is the most specific matching rule. | Allowed ALLOW / on line 9 is the most specific matching rule. | Allowed ALLOW / on line 9 is the most specific matching rule. | Allowed ALLOW / on line 9 is the most specific matching rule. |
ChatGPT-User OpenAI User-directed browsing/fetching from ChatGPT-style workflows. | assistant | Allowed ALLOW / on line 12 is the most specific matching rule. | Allowed ALLOW / on line 12 is the most specific matching rule. | Allowed ALLOW / on line 12 is the most specific matching rule. | Allowed ALLOW / on line 12 is the most specific matching rule. | Allowed ALLOW / on line 12 is the most specific matching rule. | Allowed ALLOW / on line 12 is the most specific matching rule. | Allowed ALLOW / on line 12 is the most specific matching rule. | Allowed ALLOW / on line 12 is the most specific matching rule. |
GPTBot OpenAI Model-training crawler. Blocking it is different from blocking OAI-SearchBot. | training | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Blocked DISALLOW /admin/ on line 4 is the most specific matching rule. | Blocked DISALLOW /api/ on line 5 is the most specific matching rule. |
Claude-SearchBot Anthropic Claude search/discovery crawler. Important for Claude web answers. | search | Allowed ALLOW / on line 15 is the most specific matching rule. | Allowed ALLOW / on line 15 is the most specific matching rule. | Allowed ALLOW / on line 15 is the most specific matching rule. | Allowed ALLOW / on line 15 is the most specific matching rule. | Allowed ALLOW / on line 15 is the most specific matching rule. | Allowed ALLOW / on line 15 is the most specific matching rule. | Allowed ALLOW / on line 15 is the most specific matching rule. | Allowed ALLOW / on line 15 is the most specific matching rule. |
Claude-User Anthropic User-directed Claude retrieval. Blocking it can prevent Claude from reading public pages for a user. | assistant | Allowed ALLOW / on line 18 is the most specific matching rule. | Allowed ALLOW / on line 18 is the most specific matching rule. | Allowed ALLOW / on line 18 is the most specific matching rule. | Allowed ALLOW / on line 18 is the most specific matching rule. | Allowed ALLOW / on line 18 is the most specific matching rule. | Allowed ALLOW / on line 18 is the most specific matching rule. | Allowed ALLOW / on line 18 is the most specific matching rule. | Allowed ALLOW / on line 18 is the most specific matching rule. |
ClaudeBot Anthropic Anthropic training crawler. Treat separately from Claude search/user agents. | training | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Blocked DISALLOW /admin/ on line 4 is the most specific matching rule. | Blocked DISALLOW /api/ on line 5 is the most specific matching rule. |
PerplexityBot Perplexity Perplexity indexing/discovery crawler. | search | Allowed ALLOW / on line 21 is the most specific matching rule. | Allowed ALLOW / on line 21 is the most specific matching rule. | Allowed ALLOW / on line 21 is the most specific matching rule. | Allowed ALLOW / on line 21 is the most specific matching rule. | Allowed ALLOW / on line 21 is the most specific matching rule. | Allowed ALLOW / on line 21 is the most specific matching rule. | Allowed ALLOW / on line 21 is the most specific matching rule. | Allowed ALLOW / on line 21 is the most specific matching rule. |
Perplexity-User Perplexity User-directed Perplexity fetching. Some platforms may handle user agents differently from index bots. | assistant | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Blocked DISALLOW /admin/ on line 4 is the most specific matching rule. | Blocked DISALLOW /api/ on line 5 is the most specific matching rule. |
GoogleOther Generic Google crawler used by product teams. It does not replace Googlebot for Search. | platform | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Blocked DISALLOW /admin/ on line 4 is the most specific matching rule. | Blocked DISALLOW /api/ on line 5 is the most specific matching rule. |
Google-Extended Control token for Gemini/Vertex training and grounding use; not a separate HTTP user agent. | training | Blocked DISALLOW / on line 24 is the most specific matching rule. | Blocked DISALLOW / on line 24 is the most specific matching rule. | Blocked DISALLOW / on line 24 is the most specific matching rule. | Blocked DISALLOW / on line 24 is the most specific matching rule. | Blocked DISALLOW / on line 24 is the most specific matching rule. | Blocked DISALLOW / on line 24 is the most specific matching rule. | Blocked DISALLOW / on line 24 is the most specific matching rule. | Blocked DISALLOW / on line 24 is the most specific matching rule. |
Applebot Apple Apple crawler used for Apple search and assistant surfaces. | search | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Blocked DISALLOW /admin/ on line 4 is the most specific matching rule. | Blocked DISALLOW /api/ on line 5 is the most specific matching rule. |
Applebot-Extended Apple Apple extended-use control token. Separate this from Applebot search access. | training | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Blocked DISALLOW /admin/ on line 4 is the most specific matching rule. | Blocked DISALLOW /api/ on line 5 is the most specific matching rule. |
GrokBot xAI xAI/Grok discovery token seen in AI crawler references. | emerging | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Blocked DISALLOW /admin/ on line 4 is the most specific matching rule. | Blocked DISALLOW /api/ on line 5 is the most specific matching rule. |
xAI-Bot xAI xAI crawler token variant to monitor in logs and robots rules. | emerging | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Blocked DISALLOW /admin/ on line 4 is the most specific matching rule. | Blocked DISALLOW /api/ on line 5 is the most specific matching rule. |
KimiBot Moonshot/Kimi Emerging AI-search crawler token to monitor. | emerging | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Blocked DISALLOW /admin/ on line 4 is the most specific matching rule. | Blocked DISALLOW /api/ on line 5 is the most specific matching rule. |
Kimi-SearchBot Moonshot/Kimi Search-oriented Kimi token variant to monitor. | emerging | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Blocked DISALLOW /admin/ on line 4 is the most specific matching rule. | Blocked DISALLOW /api/ on line 5 is the most specific matching rule. |
DeepSeekBot DeepSeek Emerging AI crawler token to monitor. | emerging | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Blocked DISALLOW /admin/ on line 4 is the most specific matching rule. | Blocked DISALLOW /api/ on line 5 is the most specific matching rule. |
QwenBot Alibaba/Qwen Emerging AI crawler token to monitor. | emerging | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Blocked DISALLOW /admin/ on line 4 is the most specific matching rule. | Blocked DISALLOW /api/ on line 5 is the most specific matching rule. |
DoubaoBot ByteDance/Doubao Emerging AI crawler token to monitor. | emerging | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Blocked DISALLOW /admin/ on line 4 is the most specific matching rule. | Blocked DISALLOW /api/ on line 5 is the most specific matching rule. |
Meta-ExternalAgent Meta Meta external fetcher/agent token seen in AI-crawler controls. | platform | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Blocked DISALLOW /admin/ on line 4 is the most specific matching rule. | Blocked DISALLOW /api/ on line 5 is the most specific matching rule. |
Bravebot Brave Brave Search crawler can feed search and assistant retrieval paths. | search | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Blocked DISALLOW /admin/ on line 4 is the most specific matching rule. | Blocked DISALLOW /api/ on line 5 is the most specific matching rule. |
DuckAssistBot DuckDuckGo DuckDuckGo AI answer crawler token to monitor. | emerging | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Blocked DISALLOW /admin/ on line 4 is the most specific matching rule. | Blocked DISALLOW /api/ on line 5 is the most specific matching rule. |
KagiBot Kagi Kagi crawler for independent search surfaces. | search | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Blocked DISALLOW /admin/ on line 4 is the most specific matching rule. | Blocked DISALLOW /api/ on line 5 is the most specific matching rule. |
YouBot You.com You.com assistant/search crawler token to monitor. | emerging | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Blocked DISALLOW /admin/ on line 4 is the most specific matching rule. | Blocked DISALLOW /api/ on line 5 is the most specific matching rule. |
PhindBot Phind Developer-focused AI search crawler token to monitor. | emerging | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Allowed ALLOW / on line 3 is the most specific matching rule. | Blocked DISALLOW /admin/ on line 4 is the most specific matching rule. | Blocked DISALLOW /api/ on line 5 is the most specific matching rule. |
Search visibility
Do not accidentally block Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, or PerplexityBot from canonical public pages.
Assistant retrieval
User-directed agents such as ChatGPT-User and Claude-User need access when a person asks an AI to read a page.
Training policy
Training or extended-use tokens can be blocked while search and user-directed access stay open. Keep those choices separate.
How to use this for AI search and SEO
AI search visibility depends on clean crawling, clean citations, and pages that answer concrete questions. Use this tester before shipping a new robots.txt file, CDN bot rule, staging lockdown, or AI crawl policy. If public pages are blocked, AI assistants may still know your brand through third-party search, but they are less likely to fetch, quote, or cite the exact page you want.
A practical growth setup keeps canonical tools, comparison pages, methodology pages, llms.txt, ai-index.json, and sitemap.xml crawlable while private app paths stay protected. Pair that with specific titles, direct answers near the top of the page, structured data, internal links from hubs, and source-backed claims that an assistant can cite without guessing.
Frequently Asked Questions
Should I allow every AI crawler?
Not automatically. For growth, keep core public discovery agents such as Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, and PerplexityBot able to fetch public pages. Treat model-training crawlers such as GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended as a separate content-policy decision.
Does robots.txt guarantee AI visibility?
No. It only controls crawl permission for compliant bots. AI systems may also rely on traditional search indexes, citations from other sites, browser retrieval, user clicks, backlinks, structured data, sitemaps, and page quality.
Which paths should be open?
Public indexable pages, canonical articles, tools, docs, llms.txt, ai-index.json, and sitemap.xml should normally be crawlable. Admin, account, private API, checkout, staging, and internal dashboards should normally be blocked or protected by authentication.