How to use the robots.txt generator
- Choose a default rule. Almost every public site should allow everything and block only specific private paths.
- List paths to block, one per line, such as /admin/ or /cart/. Add allowed exceptions if needed.
- Paste your sitemap URL so crawlers can find it, then tick any AI crawlers you want to block.
- Copy or download the file and upload it to your site root. Test it in Google Search Console afterwards.
What robots.txt does and does not do
A robots.txt file tells well-behaved crawlers which parts of your site they may fetch. It controls crawling, not indexing. A page blocked in robots.txt can still appear in search results, without a description, if other sites link to it. To keep a page out of Google, use a noindex tag or password protection, and leave the page crawlable so Google can see the tag.
Robots.txt is also not a security tool. The file is public, and bad bots ignore it. Never use it to hide sensitive URLs: listing them just advertises where they are.
Blocking AI crawlers: training vs search
Some AI crawlers collect content to train models. Others power AI search or fetch a page when a user asks. They are different decisions. Blocking a training crawler such as GPTBot or ClaudeBot keeps your pages out of future training data. Blocking a search crawler such as OAI-SearchBot or PerplexityBot can stop your site being cited in those products.
Google-Extended is a special case: it controls whether Google uses your content for Gemini training and grounding, and it has no effect on ranking in Google Search. Compliance is voluntary, so robots.txt only works on crawlers that choose to honour it. Crawler names change, so check each vendor's documentation before relying on this list.
Common robots.txt mistakes
- Disallow: / left over from a staging site. This blocks your whole site and is the most damaging mistake.
- Blocking CSS and JavaScript. Google needs them to render the page properly.
- Using it to remove indexed pages. Blocking a URL stops Google reading its noindex tag.
- Wrong location. The file must be at the root of each host. A subdomain needs its own file.
- Relying on crawl-delay. Google ignores it.
How the rules are matched
Each group starts with a User-agent line, followed by Allow and Disallow rules. A crawler follows the most specific group that names it and ignores the rest, including the * group. For Google, the longest matching path wins, and if an allow and a disallow rule match equally, the less restrictive one applies. The format is defined in RFC 9309, and Google explains its handling in its robots.txt documentation.
Next steps after publishing
Open /robots.txt in your browser to confirm it loads, check it with the robots.txt report in Google Search Console, and generate an XML sitemap if you do not have one. The free SEO audit also flags pages that are blocked or marked noindex by mistake.
Example robots.txt files for common sites
A simple blog usually needs only a few lines: allow everything, block the admin area and the internal search results, and point to the sitemap. A shop often also blocks the cart, checkout and account pages, plus faceted filter URLs that create endless variations. A staging site should block everything, and should also be password protected, since robots.txt alone does not keep people out.
Use the generator to build each of these, then read the output line by line so you understand every rule you publish.
Testing and monitoring your robots.txt
After publishing, open the file in your browser to confirm it loads with a 200 status and plain text. Use the robots.txt report in Google Search Console to see when Google last fetched it and whether it found errors. Test a few important URLs to make sure they are allowed and a few private ones to make sure they are blocked.
Add a calendar reminder to review the file after every site redesign or migration, since a rule left over from a staging site is one of the most common causes of sudden drops in traffic.
A short checklist before you upload
- The file is called robots.txt and sits at the root of the host.
- The default rule allows crawling unless the site is private.
- No rule blocks CSS, JavaScript or images that pages need to render.
- Private areas are protected by a login or noindex, not only by robots.txt.
- The Sitemap line uses a full https address.
- Any AI crawler rules match a decision you made on purpose.
- You tested the live file in Google Search Console afterwards.
Keep a copy of every version you publish, so that if traffic changes you can see exactly what you changed and when.
Frequently asked questions
Do I need a robots.txt file?
No. If a site has no file, crawlers assume everything is allowed. You need one only to block certain paths, point to your sitemap or set rules for specific crawlers.
Will robots.txt remove a page from Google?
No. It stops crawling but does not guarantee removal from search results. Use a noindex tag, and keep the page crawlable, so Google can see it.
Should I block AI crawlers?
It depends on your goals. Blocking training crawlers keeps content out of model training. Blocking AI search crawlers can reduce your visibility in AI answers. Many publishers block the first group and allow the second.
Does Google-Extended affect my rankings?
No. Google says it does not affect inclusion or ranking in Google Search. It controls use of your content for Gemini and related AI products.
Where does robots.txt go?
At the root of the host, for example https://example.com/robots.txt. It does not work in a subfolder.