AI robots.txt Generator
Generate robots.txt rules to block AI crawlers from indexing and scraping your content.
Select AI Crawlers to Block
Choose which AI crawlers you want to block. We recommend blocking all for maximum protection.
20 of 20 selected
Allen Institute for AI1/1
Amazon1/1
Anthropic2/2
Apple1/1
ByteDance1/1
Cohere1/1
Common Crawl1/1
Diffbot1/1
Google1/1
Meta2/2
Microsoft1/1
OpenAI3/3
Perplexity1/1
Various1/1
Webz.io1/1
You.com1/1
Generated robots.txtrobots.txt
# Block AI Crawlers # Generated by webmaster-zone.com User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: anthropic-ai Disallow: / User-agent: ClaudeBot Disallow: / User-agent: FacebookBot Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: cohere-ai Disallow: / User-agent: YouBot Disallow: / User-agent: Diffbot Disallow: / User-agent: Omgilibot Disallow: / User-agent: Amazonbot Disallow: / User-agent: bingbot Disallow: / User-agent: AI2Bot Disallow: / User-agent: img2dataset Disallow: / # Allow search engine crawlers (not AI training) User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: * Allow: /
Some bots may ignore robots.txt
Bytespider, img2dataset are known to sometimes ignore robots.txt directives. For reliable blocking, also use our .htaccess generator.
How to Use
- 1Copy the generated robots.txt content above or download the file.
- 2Upload the file to the root directory of your website (e.g.,
yoursite.com/robots.txt). - 3If you already have a robots.txt, merge our rules with your existing file. Place our rules at the top.
- 4Verify by visiting
yoursite.com/robots.txtin your browser.
Good to know
robots.txt is a standard way to communicate with web crawlers. Well-behaved bots will respect these rules. For bots that ignore robots.txt, use server-level blocking with .htaccess rules.