AI crawlers
What GPTBot, ClaudeBot and PerplexityBot do, how to block or allow them in robots.txt, and what we found when we checked the robots.txt of every site ChatGPT, Perplexity and Google AI Overviews cited in our test.
Last updated November 2, 2026 · ~7 min read
The short answer: GPTBot is OpenAI’s crawler for training data. Blocking it tells OpenAI not to train on your content, but it does not keep you out of ChatGPT’s answers: that is OAI-SearchBot, a separate crawler. Of 677 pages ChatGPT cited in our test, GPTBot was not allowed to fetch 45, and only 6 were closed to OAI-SearchBot. Across 762 cited sites, 11.9% block GPTBot and 4.2% block OAI-SearchBot; review sites and media block far more often than brands.
GPTBot is the web crawler OpenAI uses to collect public pages that may be used to train its AI models. It identifies itself with “GPTBot” in its user agent, and sites control it in robots.txt. OpenAI’s documentation says that disallowing GPTBot indicates a site’s content should not be used in training.
GPTBot is not the crawler that decides whether ChatGPT can show your pages in its search answers. OpenAI runs separate bots, and each robots.txt setting is independent of the others:
| Bot | Company | What it does | If you block it |
|---|---|---|---|
| GPTBot | OpenAI | Collects pages that may be used to train models | Your content should not be used in training |
| OAI-SearchBot | OpenAI | Finds pages to show in ChatGPT search answers | OpenAI says to allow it if you want to appear in ChatGPT search |
| ChatGPT-User | OpenAI | Visits a page when a ChatGPT user’s request needs it; not an automatic crawler | OpenAI says robots.txt rules may not apply, because a user started the visit |
| ClaudeBot | Anthropic | Collects pages that may be used to train models | Your future pages are left out of training |
| Claude-SearchBot | Anthropic | Indexes pages to improve Claude’s search results | May reduce your visibility in Claude’s search answers |
| Claude-User | Anthropic | Visits a page when a Claude user’s question needs it | Claude can’t fetch your page to answer a user |
| PerplexityBot | Perplexity | Finds and links pages in Perplexity’s answers; Perplexity says it is not used for model training | Perplexity recommends allowing it so your pages can appear in its answers |
| Perplexity-User | Perplexity | Visits a page a user’s question needs | Perplexity says it generally ignores robots.txt |
| Google-Extended | Not a crawler: a robots.txt token for Gemini training and grounding | No effect on Google Search, which includes AI Overviews |
Anthropic says all three of its bots honor robots.txt. For Google Search, including AI Overviews and AI Mode, the control is Googlebot in robots.txt, plus the nosnippet and noindex controls on the page.
We wanted to know how sites treat these bots in practice, and whether blocking one changes what AI answers cite. On October 22, 2026, we asked ChatGPT, Perplexity and Google 60 shopping research questions, three times each (the test behind our Perplexity vs ChatGPT comparison). We kept every page each answer cited: 744 from ChatGPT, 612 from Perplexity and 663 from Google AI Overviews, on 816 sites in all.
On November 2, we read each site’s robots.txt once and applied the standard rules (RFC 9309) to every cited page, for each bot. 42 sites refused even our request for robots.txt (403), and 11 more returned an error or did not respond, so they are left out. We also left Reddit out: its robots.txt closes the whole site to every crawler, so it says nothing about AI bots in particular. That leaves 762 sites.
Share of the 762 cited sites whose robots.txt closes the whole site to each bot:
| Bot | Used for | Sites that block it |
|---|---|---|
| CCBot (Common Crawl) | Open web archive, widely used for training | 134 (17.6%) |
| ClaudeBot | Training | 107 (14.0%) |
| Google-Extended | Gemini training and grounding | 106 (13.9%) |
| GPTBot | Training | 91 (11.9%) |
| PerplexityBot | Search | 85 (11.2%) |
| Claude-SearchBot | Search | 59 (7.7%) |
| OAI-SearchBot | Search | 32 (4.2%) |
| Googlebot | Google Search | 1 (0.1%) |
GPTBot is blocked nearly three times as often as OAI-SearchBot, and ClaudeBot nearly twice as often as Claude-SearchBot. PerplexityBot is the exception: it is a search crawler, yet sites block it about as often as GPTBot. 59 sites block GPTBot but leave OAI-SearchBot open, which is the split OpenAI’s settings allow: no training, still in search. 28 sites close the door to all three AI search crawlers. 229 sites name GPTBot in their robots.txt, and 275 (36%) name at least one AI crawler.
Who blocks depends on who they are:
| Site type | GPTBot | OAI-SearchBot | ClaudeBot | PerplexityBot |
|---|---|---|---|---|
| Review sites and media (285) | 19.6% | 7.7% | 26.3% | 25.6% |
| Retailers (73) | 6.8% | 2.7% | 5.5% | 4.1% |
| Brands’ own sites (239) | 2.5% | 0.4% | 2.1% | 0.8% |
Publishers, whose content is the product, close the door most often. Brands almost never do: 6 of 239 brand sites block GPTBot, and one blocks OAI-SearchBot.
Of the 677 pages ChatGPT cited on sites whose robots.txt we could read, GPTBot was not allowed to fetch 45 (6.6%). OAI-SearchBot was allowed on 39 of those 45. Only 6 cited pages (0.9%) were closed to OAI-SearchBot: five were links to Amazon search results and one was a PDF. Amazon’s robots.txt closes the whole site to OpenAI’s bots and to most other AI crawlers, yet ChatGPT still pointed buyers to it.
So in ChatGPT the line runs where OpenAI draws it: the training crawler’s rules didn’t stop citations, and the search crawler’s rules almost always held.
robots.txt is a request to crawlers about fetching. It is not a filter on what an answer can cite, and each company decides which of its bots the request covers.
robots.txt sits at the root of each host: yoursite.com/robots.txt, and separately for each subdomain. To keep your content out of OpenAI’s training while staying in ChatGPT search:
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: /
To block every OpenAI crawler, list both with Disallow: /. ChatGPT-User visits that a user starts may still happen, as OpenAI notes. The same pattern works for the others: ClaudeBot and Claude-SearchBot for Anthropic, PerplexityBot for Perplexity, Google-Extended for Gemini training. To allow everything, you don’t need to name any bot, as long as nothing closes the site to all of them:
User-agent: * Allow: /
Three things to know:
User-agent: * group. If you name GPTBot to block a folder, repeat any general rules it should also follow.Separate the two decisions:
Disallow: / under User-agent: *. Ask whoever runs your hosting or CDN whether a bot-protection setting blocks AI crawlers. Then ask the assistants your buyers’ questions and see whether they cite you.For what helps once crawlers can read you, see how to get ChatGPT to recommend your business and LLM visibility.
Answers: 60 shopping research questions in 12 categories, asked three times on October 22, 2026, US location; ChatGPT through chatgpt.com search, Perplexity through its Sonar API with web search, Google through the search results page, all collected through DataForSEO. robots.txt: fetched once per host on November 2, 2026, with a user agent that identifies us; ChatGPT’s utm_source tag removed from links before matching. A site’s rules may have changed between the two dates, and some sites may serve different rules to different bots. We read robots.txt only, not firewall settings, and we can’t see how each engine found a page it cited. Site types were assigned by hand.
Hasmord is the AEO-native sales platform. Buyers ask questions across AI assistants, search, and sales conversations. Hasmord uncovers where your answers about your products and services fall short, turns evidence into better answers, finds potential customers for you, and connects every lead to your sales process.
Crawler access comes before everything else in the first step, which uncovers where your answers about your products and services fall short. AI answers asks ChatGPT, Claude, Gemini and Perplexity your buyers’ questions and keeps every answer and the pages it cites, so you can see whether assistants read your site at all, and who they quote instead.
The second step turns evidence into better answers: Hasmord compares those answers with your product facts and prepares fixes, product by product, for your team to approve. Find customers then finds potential customers already looking for what you sell, with the reason each one fits; only the people you verify join your CRM, and Hasmord never contacts anyone for you. In the CRM, each lead keeps the question that brought it, so what buyers ask in sales becomes the next question to check. See pricing.
GPTBot is OpenAI’s web crawler for model training. It collects public pages that may be used to train OpenAI’s AI models and follows the rules a site sets for it in robots.txt. Disallowing GPTBot tells OpenAI your content should not be used for training.
No. ChatGPT’s search answers use OAI-SearchBot, a separate crawler, and OpenAI says each setting is independent. In our check, ChatGPT cited 45 pages that GPTBot was not allowed to fetch; 39 of them were open to OAI-SearchBot.
GPTBot collects pages that may be used to train models. OAI-SearchBot finds pages to show in ChatGPT search answers. You can block one and allow the other in robots.txt; 59 of the 762 sites we checked block GPTBot but leave OAI-SearchBot open.
It depends on whether you want your content used for training. Publishers often block it: 19.6% of the review sites and media we checked did. Brands rarely do: 2.5%. If you want ChatGPT to recommend you, keep OAI-SearchBot allowed either way.
ClaudeBot is Anthropic’s crawler for content that may be used to train Claude. Anthropic also runs Claude-SearchBot for search and Claude-User for pages a user’s question needs, and says all three honor robots.txt. 14.0% of the 762 sites we checked block ClaudeBot, against 7.7% for Claude-SearchBot.
PerplexityBot finds and links websites in Perplexity’s answers; Perplexity says it is not used to train AI models. Perplexity-User, which fetches pages when a user asks, generally ignores robots.txt. Perplexity cited 181 pages in our test whose sites close them to PerplexityBot.
OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust.
Related: LLM visibility · ChatGPT citations · Perplexity vs ChatGPT · the AI assistant source study · AI visibility checker · appear in Claude.