AI Crawler robots.txt Generator
Decide exactly which AI crawlers can read your site — the answer engines that cite you and the
bots that only train on you — then copy a ready-to-paste robots.txt block.
Runs entirely in your browser. Nothing is uploaded, nothing is stored.
AI answer engines
These bots power AI search results — allowing them is how you get cited.
AI training crawlers
These harvest content to train models. Blocking them does not affect citations.
Checked = allowed to crawl. Unchecked = blocked. Add these lines to the robots.txt
at your domain root. robots.txt is an honor-system directive — most named AI crawlers respect it, but it is not an enforced block.
Keep going
FAQ Schema Generator →
Turn Q&A into FAQPage JSON-LD so assistants can lift clean answers from your page.
Answer Snippet Checker →
Score any passage on how citation-ready it is for AI answers.
Want to know if assistants actually cite you today?
Run a free visibility scan