AI crawler robots.txt generator
AI engines send their own crawlers to read your site: GPTBot, PerplexityBot, ClaudeBot, Google-Extended and more. This tool builds the robots.txt rules to let the ones you want in and keep the rest out. Choose per bot, then copy or download. No app, no signup, nothing stored.
Let these in to get cited in AI answers.
Your call — allow to feed the models, block to opt out.
# AI crawler rules — generated with matthapgood.co/ai-crawler-robots-txt-generator # Add these blocks to your robots.txt (at yourdomain.com/robots.txt). User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Claude-User Allow: / User-agent: Applebot Allow: / User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Amazonbot Disallow: / User-agent: Meta-ExternalAgent Disallow: /
Paste these blocks into your robots.txt at the root of your domain (yourdomain.com/robots.txt). On Shopify you customise it through robots.txt.liquid in your theme — add your rules through the template, don't paste raw blocks over it. A rule only works if the bot respects robots.txt: most named ones do, but it's a request, not a wall.
Should you let AI crawlers in?
For most businesses, yes, at least the answer bots. If you block the crawlers behind ChatGPT search and Perplexity, you can't be cited when someone asks those tools a buying question, and a rival gets named instead. The training crawlers are a separate decision: allowing them feeds your content into future models, blocking them opts you out. There's no single right answer, which is why this tool lets you choose per bot.
Answer bots vs training crawlers
Answer bots (like OAI-SearchBot and PerplexityBot) fetch pages to build the cited answers users see. Training crawlers (like GPTBot and CCBot) gather content to train models. The recommended preset lets the answer bots in, so you stay visible in AI results, and blocks the pure training crawlers. Adjust to taste.
Where the rules go
These blocks belong in your robots.txt, at the root of your domain. On Shopify you customise it through robots.txt.liquid in your theme — add your rules through the template rather than pasting the raw blocks over it. Remember robots.txt is a request, not a firewall: the well-behaved named crawlers respect it, but it won't stop one that ignores the rules.
Letting AI in is step one. To actually get cited, read the guide to generative engine optimization and generate an llms.txt for your store.