robots.txt for AI crawlers: the practical guide
Every AI assistant that talks about your business first sent a bot to your website. robots.txt is where you decide what those bots may read — and most websites have never made that decision consciously.
Why this file suddenly matters
robots.txt has existed since 1994, but until recently only search engines read it. Today it is also the gatekeeper for AI: the bots of OpenAI, Anthropic, Google, Perplexity and Microsoft all check it before reading your pages. What they find there decides whether AI assistants can learn about your business, cite it, and recommend it.
One rule of thumb before the details: the bots have different jobs. Blocking one does not make you "invisible to AI" — and allowing one does not make you visible. There are three roles to understand.
The AI bots and what each one does
| Bot | Operator | Role | If you block it |
|---|---|---|---|
| GPTBot | OpenAI | Training — builds the model's knowledge | Future models know less about you |
| OAI-SearchBot | OpenAI | Search — powers ChatGPT Search citations | ChatGPT stops citing your pages live |
| ChatGPT-User | OpenAI | User-fetch — opens your page when a user asks | Fetches on direct user request |
| ClaudeBot | Anthropic | Training | Claude models know less about you |
| Claude-SearchBot | Anthropic | Search — citations in Claude's web search | Claude stops citing your pages |
| Claude-User | Anthropic | User-fetch | Fetches on direct user request |
| PerplexityBot | Perplexity | Search index — it does not train models | You drop out of Perplexity answers |
| Google-Extended | Gemini training and Search grounding | Excluded from grounded Gemini answers | |
| Bingbot | Microsoft | Bing index — which also feeds ChatGPT's search | Weaker presence in Bing and ChatGPT Search |
Sources: the official bot documentation of OpenAI (developers.openai.com), Anthropic (support.claude.com), Perplexity (docs.perplexity.ai), Google (developers.google.com) and Bing — as of July 2026. Google's AI Overviews use the regular Googlebot, so they are governed by your normal search rules. User-fetch bots act on a person's direct request and largely operate outside robots.txt.
The setup most businesses want
If you want AI assistants to know and recommend your business, the answer is simple — make sure nothing blocks the AI bots. A minimal robots.txt that allows everyone and points to your sitemap:
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
If you deliberately do not want your content used for model training but still want to be cited in AI search, block the training bots and leave the search bots alone:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# …everyone else (incl. search bots) stays allowed
User-agent: *
Allow: /
That trade-off is real, though: training is how models learn who you are. For most small businesses trying to win customers, blocking training bots costs more than it protects.
The four mistakes we actually see
We benchmarked 447 Norwegian trade businesses with our audit methodology. The robots.txt failures fall into four patterns:
1. No robots.txt at all (8% of businesses). Not fatal — no file means "everything allowed" — but you also lose the sitemap pointer and any conscious control.
2. A global Disallow: /. Usually a leftover from a staging environment. It tells every crawler, AI included, to read nothing. We see it rarely, but when we do, the business is invisible in every AI system at once.
3. Copy-pasted "block AI bots" lists. Templates that circulated in 2023–24 block GPTBot "to protect content" — written before AI search existed. The result in 2026: your competitors get cited, you do not. 4% of the benchmark still blocks GPTBot, mostly via such templates.
4. The firewall blocks what robots.txt allows (1 in 11 businesses). This is the sneaky one. Your robots.txt says "welcome" — but the WAF or hosting provider returns 403 to requests with AI bot user-agents. 9% of the 447 businesses had at least one AI bot blocked this way, and none of them knew. In one audit we found a firewall blocking OpenAI's training crawler while letting the search bots through — the site owner had made no such decision.
Test it in two minutes
Don't trust the config — test the behaviour. From any terminal, fetch your homepage as an AI bot and compare with a normal request:
curl -sI -A "OAI-SearchBot" https://www.example.com/ | head -1
curl -sI -A "ClaudeBot" https://www.example.com/ | head -1
curl -sI -A "PerplexityBot" https://www.example.com/ | head -1
All four should return the same code as a normal browser request — 200. A 403 on any of them means your firewall is making AI policy for you. (Our audit runs exactly this test, among 33 checks.)
Check your whole setup, not just robots.txt
robots.txt is 1 of the 33 things we test. The free mini-audit checks your domain in under a minute — including whether AI bots can actually reach your pages.
Get a free mini-audit → How we measure