AI crawlers and robots.txt: what to allow
Updated: · Orbtown
AI companies now separate crawlers for search from crawlers for training. That means you can stay visible in AI answers and still say no to training.
Who does what
| Crawler | Company | Purpose |
|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT search |
| GPTBot | OpenAI | Model training |
| ChatGPT-User | OpenAI | Pages a user asks for |
| Claude-SearchBot | Anthropic | Claude search index |
| ClaudeBot | Anthropic | Model training |
| Claude-User | Anthropic | Pages a user asks for |
| PerplexityBot | Perplexity | Perplexity search results |
| Googlebot | Search, including AI Overviews |
Visible in AI, no training
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
Sitemap: https://example.com/sitemap.xmlWatch out for
- Your firewall. A CDN or security plugin may block these bots even when robots.txt allows them. Check your logs.
- User-triggered fetchers. OpenAI says robots.txt rules may not apply to ChatGPT-User.
- Googlebot. Blocking it removes you from Google Search. Google-Extended is a separate control for training and grounding in some other Google systems.
- Names change. Check the vendors' current documentation before relying on any list, including this one.
Common questions
Should I block GPTBot?
That is a business decision about training. Blocking it does not stop OAI-SearchBot from surfacing your site in ChatGPT search.
How fast do robots.txt changes work?
OpenAI says about 24 hours for its search results. Other companies may differ.