Webmaster Zone
Feedback
Security
SEO
Social Media
DNS & Network
Developer Tools
Blog
Home
Security
Robots.txt Tester
Robots.txt Tester
Test if URLs are allowed or blocked by a website's robots.txt rules.
Website Domain
User Agent
Googlebot
Bingbot
GPTBot (OpenAI)
Claude-Web (Anthropic)
Bytespider (TikTok)
CCBot (Common Crawl)
Applebot
DuckDuckBot
Yahoo Slurp
All Bots (*)
URLs to Test
Add URL
Test Robots.txt
Related Tools
View all Security tools →
robots.txt Generator
Block AI crawlers with robots.txt rules
.htaccess Generator
Block crawlers at server level
Security Headers Checker
Check security headers and get a grade
SSL Certificate Checker
Verify SSL validity and expiry