nalyzed.

robots.txt Tester

Paste your robots.txt, pick a path and see instantly which crawlers may fetch it. Matching follows RFC 9309: the most specific user-agent group applies, the longest matching rule wins, and Allow wins a tie.

Search engines
CrawlerAllowed
Googlebot✓ Allowed
Bingbot✓ Allowed
DuckDuckBot✓ Allowed
YandexBot✓ Allowed
Applebot✓ Allowed
AI crawlers
CrawlerAllowed
GPTBot✕ Blocked
OAI-SearchBot✓ Allowed
ChatGPT-User✓ Allowed
ClaudeBot✓ Allowed
Claude-SearchBot✓ Allowed
Claude-User✓ Allowed
PerplexityBot✓ Allowed
Google-Extended✓ Allowed
Applebot-Extended✓ Allowed
CCBot✓ Allowed
Bytespider✓ Allowed
meta-externalagent✓ Allowed
Amazonbot✓ Allowed
DuckAssistBot✓ Allowed
MistralAI-User✓ Allowed
YouBot✓ Allowed
cohere-ai✓ Allowed
Diffbot✓ Allowed
AI2Bot✓ Allowed
ImagesiftBot✓ Allowed
PetalBot✓ Allowed
Timpibot✓ Allowed

What it checks

  • Specific user-agent groups over the * group
  • Longest-match precedence with * wildcards and $ anchors
  • Every major search and AI crawler at once
  • Runs in your browser; nothing is uploaded

Frequently asked questions

Why is my page still indexed if robots.txt blocks it?
robots.txt controls crawling, not indexing. A blocked URL can still be indexed from links. Use a noindex meta tag or header, and leave the page crawlable so the tag can be seen.
Are rules case sensitive?
Paths are case sensitive; user-agent names are not.
Can I test my live robots.txt?
Run a full report: it fetches your live robots.txt and shows access for each crawler.

More free tools

Türkçe