SEO
Free robots.txt generator with AI crawler controls
By Charles Summers · Updated · Free, no signup
Short answer
This generates a complete robots.txt and, unlike most generators, separates AI crawlers into the two classes that behave differently. Citation crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot decide whether your pages can appear in AI answers. Training crawlers such as GPTBot, ClaudeBot and Google-Extended only affect model training. Blocking GPTBot does not remove you from ChatGPT search results; blocking OAI-SearchBot does, and OpenAI says so plainly. The two decisions are unrelated and most guides conflate them.
Use the robots.txt generator
What does this tool actually do?
This generates a complete robots.txt and, unlike most generators, separates AI crawlers into the two classes that behave differently. Citation crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot decide whether your pages can appear in AI answers.
It runs entirely in your browser. Nothing you type is sent to a server, no account is required, and there is no usage limit, because there is no cost per run to control.
Two kinds of AI crawler, and the block that does the opposite of what people think
The single most consequential misunderstanding in technical SEO right now is that AI crawlers are one thing. They are two things, run by the same companies, under different user agent tokens, doing unrelated jobs. Training crawlers collect text that may be used to train future models. Citation crawlers build the retrieval index that an assistant searches at question time, which is what decides whether your page can be quoted and linked in an answer. Blocking one has no effect whatsoever on the other, and the names are similar enough that people block the wrong one constantly.
OpenAI is the clearest illustration because it operates three separate agents. GPTBot crawls for model training. OAI-SearchBot crawls to build the search index behind ChatGPT, and OpenAI states plainly that sites which opt out of OAI-SearchBot will not be shown in ChatGPT search results. ChatGPT-User is the fetch that happens when a user asks about a specific page. So the widely copied advice to add a GPTBot disallow, presented as a way to keep your content out of AI, removes you from training data and leaves your visibility in ChatGPT answers untouched. And the reverse holds: a site that blocks OAI-SearchBot while allowing GPTBot has arranged the worst of both, contributing to training while removing itself from the answers.
Anthropic splits the same way, with ClaudeBot for training, Claude-SearchBot for the index that supports citations, and Claude-User for user-initiated fetches. Perplexity runs PerplexityBot for indexing alongside a user-triggered fetcher. Google and Apple do something different again, and it catches people out: Google-Extended and Applebot-Extended are not crawlers at all. Nothing arrives at your server bearing those names. They are opt-out tokens applied to content Googlebot and Applebot already fetched, controlling whether it feeds Gemini and Apple Intelligence training. Disallowing Google-Extended has no effect on Google Search indexing, and no effect on AI Overviews either, because those are generated from the Search index.
Deciding, one class at a time
Split the decision because the two classes have different economics. The training question is about intellectual property and licensing, and the honest answer is that nobody can show you a traffic number either way, because training crawlers send no referrals. If your position is that your archive should not be free training material, block them. That is a legitimate business decision and it costs you nothing measurable in traffic. If you do not hold that position strongly, the block buys you nothing either.
The citation question is a distribution decision, and it is the same kind of decision as whether to let Googlebot in. These crawlers are the mechanism by which your page becomes the source an assistant cites, with a link, in front of somebody who is mid-task. Blocking them to protect content is a category error: you are not preventing your material being used, you are preventing it being credited. The plausible exceptions are narrow and specific. A subscription publisher whose whole product is the article body has a real argument. A membership community protecting members has one. A B2B software company that wants to be recommended when somebody asks an assistant for options does not, and blocking that class is roughly as sensible as returning a 403 to Googlebot.
Bingbot is the one that requires care in both directions, because it is not an AI crawler at all in origin. It is the classic search crawler behind Bing, and its index also grounds Copilot. Block it and you lose Bing organic traffic, DuckDuckGo results that draw on that index, and the AI surface at once. Applebot has the same dual role for Siri, Spotlight and Apple Intelligence. Neither belongs on a list of things you casually disallow because a blog post grouped them under AI.
The robots.txt mechanic that silently breaks AI bot rules
A crawler does not read your whole file and combine the rules. It selects exactly one group, the one whose user-agent line matches it most specifically, and it obeys that group alone while ignoring every other group in the file. This is where AI crawler advice goes wrong in practice. You have a working file with a wildcard group full of disallows for your admin area, your cart and your internal search. You append a group naming GPTBot with a single Allow line, and GPTBot immediately stops obeying every disallow above, because it now matches a more specific group and the wildcard group no longer applies to it. Nothing warns you. The crawler simply starts walking your checkout pages.
This generator repeats the site-wide disallow lines inside every named group for exactly that reason, which is why the output looks more repetitive than the files you see elsewhere. Within a group, Google resolves conflicts by the longest matching path rather than by order, with Allow winning an exact-length tie, which is what makes the standard WordPress pattern work: Disallow /wp-admin/ combined with a longer, more specific Allow for admin-ajax.php.
- Disallow is not deindex. It blocks crawling. A blocked URL that other pages link to can still be listed, without a snippet. To remove a page you need a noindex tag, which requires the crawler to be able to fetch the page and see it, so the two directives cannot both be applied to the same URL.
- The noindex directive in robots.txt does nothing. Google stopped supporting it in 2019. Files still carry those lines and their owners still believe they work.
- Paths are case sensitive, agent names are not. Disallow: /Private/ does not block /private/. GPTBot and gptbot match the same crawler.
- One file per host, scheme and port. The rules at example.com do not apply to shop.example.com or to the http version. Subdomains need their own file, which is how staging sites end up indexed.
- Google caps the file at 500 KiB and ignores everything after that point, which only matters on sites that generate a disallow line per URL, and those sites exist.
- Crawl-delay is ignored by Google. Bing and others honour it. Use crawl rate settings or server-side rate limiting if crawl budget is genuinely the problem.
Platform quirks worth knowing before you paste anything
Where the file lives varies more than the syntax does. WordPress serves a virtual robots.txt generated on the fly, which is silently overridden the moment a real file appears in the web root, so a plugin edit that appears to do nothing usually means someone uploaded a physical file years ago. The Discourage search engines checkbox in Reading settings is a separate mechanism again and adds a noindex behaviour rather than a robots rule, and it survives launch far too often.
Shopify does not let you edit robots.txt directly; you edit robots.txt.liquid in the theme, which patches the generated default rather than replacing it. Squarespace and Wix both maintain their own file with an editor of limited scope, and both have added their own AI crawler toggles which write rules you cannot see in the same place. On a framework build, robots.txt is usually generated at build time, which means changing it is a deploy rather than a text edit, and it means a route that generates it dynamically can fail without anybody noticing until traffic moves.
One rule survives every platform: never block the paths your CSS and JavaScript are served from. Google renders pages to evaluate them, and a disallowed asset directory produces a render of an unstyled skeleton. Blocking a framework build directory, or the uploads folder that also serves your images, is the version of this that shows up most often, and the symptom is a mobile usability or layout problem that no amount of front-end work explains.
Numbers worth knowing
| Metric | Typical | What it means |
|---|---|---|
| Blocking GPTBot | no effect on ChatGPT answers | GPTBot is the training crawler. Disallowing it removes your pages from training data and changes nothing about whether ChatGPT search can cite you. |
| Blocking OAI-SearchBot | removed from ChatGPT search | OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search results. This is the block that costs you AI visibility. |
| Google-Extended | not a crawler | A control token applied to content Googlebot already fetched, governing Gemini training use. No request ever arrives with that user agent, and disallowing it does not affect Search. |
| Groups a crawler obeys | exactly one | The most specific matching user-agent group, with every other group ignored. Adding a named AI bot group without repeating your global disallows exempts that bot from all of them. |
| Robots.txt size Google reads | 500 KiB | Everything past the cap is ignored. Relevant to large sites that generate a disallow line per parameter or per URL rather than using patterns. |
Mistakes that quietly cost you results
- Adding a GPTBot disallow and telling the board you have opted out of AI
- You have opted out of training only. ChatGPT search still reaches your pages through OAI-SearchBot, and if the goal was genuinely to leave AI answers you have to block the citation crawlers instead, and accept losing that referral channel.
- Appending a named AI crawler group to a file that already has wildcard disallows
- That crawler now obeys only the new group and ignores every rule above it, including your admin and checkout disallows. Repeat the site-wide disallow lines inside each named group, which is why the output here is deliberately repetitive.
- Using Disallow to keep a page out of search results
- A disallowed URL can still be listed if other pages link to it, and the crawler can never see the noindex tag because it is not allowed to fetch the page. Allow the crawl and serve noindex, or put the page behind authentication.
- Blocking Bingbot as part of an AI crawler cleanup
- Bingbot is the ordinary search crawler behind Bing, DuckDuckGo results and Copilot grounding all at once. Blocking it costs you conventional organic traffic that has nothing to do with AI, for no additional protection.
- Disallowing the directory your CSS and JavaScript are served from
- Google renders pages before judging them, so blocked assets produce a render of an unstyled page. Framework build directories and the uploads folder that also serves images are the two that catch people out most.
What does the output look like?
This is the exact output the tool produces from the example inputs. It is generated by the same code that runs when you click the button, so what you see here is what you get.
Frequently asked questions
What is the actual difference between GPTBot and OAI-SearchBot?
GPTBot gathers text that may be used to train future OpenAI models. OAI-SearchBot builds the search index that ChatGPT queries when it answers a question, which is what makes your page eligible to be cited and linked. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search results. They are separate user agents with separate consequences, so blocking one tells you nothing about the other.
Should I block AI training crawlers?
It depends entirely on whether you object to your archive being used as training material, because there is no traffic consequence either way. Training crawlers send no referrals, so blocking them costs nothing measurable and gains nothing measurable. Publishers with licensing ambitions and sites with proprietary research usually block. Companies who want to be found and recommended usually do not bother, and spend the effort on the citation side instead.
Why does the generated file repeat the same disallow lines several times?
Because a crawler obeys only the single most specific group that names it and ignores every other group in the file. If your admin and checkout disallows live in the wildcard group and you add a group for a named AI crawler, that crawler stops obeying the wildcard rules entirely. Repeating the lines inside each named group is what keeps them enforced, and it is the step most generators skip.
Does robots.txt keep a page out of Google?
No. It stops the page being crawled, not indexed. A disallowed URL can still be listed if other sites link to it, typically with no description, and the crawler cannot see a noindex tag because it is not permitted to fetch the page. To remove a page, let it be crawled and serve a noindex, use the removals tool for anything urgent, or require authentication.
Do I need one robots.txt for my subdomains?
Yes. Rules apply per host, scheme and port, so example.com and shop.example.com are separate scopes and neither inherits the other. This is the usual reason a staging subdomain ends up indexed: the production file has all the rules and the staging host serves an empty one, or none at all, which is treated as full permission to crawl.
Related free tools
- Readability Scorer SEO
- SEO Meta Tag Generator for ecommerce product pages SEO
- SEO Meta Tag Generator for local business SEO
- AI B2B Headline Copywriter for saas landing pages Copywriting
- AI B2B Headline Copywriter for fintech Copywriting
Some links on this site are affiliate links, which means Hacking Demand may earn a commission if you buy through them at no extra cost to you. This does not influence which tools are listed. The tools on this page are free and have no affiliate relationship of any kind.