Webmaster Zone
Feedback
Security
SEO
Social Media
DNS & Network
Developer Tools
Blog
Home
Security
Robots.txt Tester
Robots.txt Tester
Test if URLs are allowed or blocked by a website's robots.txt rules.
Website Domain
User Agent
Googlebot
Bingbot
GPTBot (OpenAI)
Claude-Web (Anthropic)
Bytespider (TikTok)
CCBot (Common Crawl)
Applebot
DuckDuckBot
Yahoo Slurp
All Bots (*)
URLs to Test
Add URL
Test Robots.txt
Related Tools
View all Security tools →
robots.txt Generator
Block AI crawlers with robots.txt rules
.htaccess Generator
Block crawlers at server level
Website Security Audit
Combined SSL, headers, and meta check
Security Headers Checker
Check security headers and get a grade