🤖 User-agents

🚫 Disallow Paths

Quick add:

✅ Allow Paths (optional)

⚙️ Settings

Block AI crawlers (GPTBot, Claude, etc.) Block aggressive scrapers Block known bad bots

Crawl delay (sec):

Sitemap URL:

robots.txt Preview

Robots.txt Generator