TL;DR: A robots.txt file tells crawlers which URLs they may fetch. It manages crawling, not indexing, so it cannot reliably hide a page from Google. Allow search crawlers, block only paths with no search value, add your sitemap, and decide about AI crawlers on purpose, because training bots and search bots are different.
Last reviewed: 15 October 2026
If you have ever wondered what to put in a robots.txt file, the safe answer is: less than you think. Most small sites need only a few lines. The risk is not an empty file. The risk is one wrong line that blocks the whole site.
This guide explains what robots.txt is, what it cannot do, the exact rules Google follows, and how to treat AI crawlers such as GPTBot and ClaudeBot. It is written by Janak Pokharel, an SEO practitioner and co-founder of Limi Creatives, for site owners who want clear rules instead of folklore.
What is a robots.txt file?
A robots.txt file is a plain text file at the root of a website that tells crawlers which URLs they can access. Its main purpose is to manage crawler traffic and avoid overloading a site. It is not a security tool and it is not an indexing control.
Google describes it this way in its introduction to robots.txt: the file “tells search engine crawlers which URLs the crawler can access on your site.” The file must live at the top level of the host, for example the address of your domain followed by /robots.txt. Rules apply only to the exact protocol, host and port where the file is served, so a subdomain needs its own file.
You can build a valid file in a minute with the free robots.txt generator, then read the rest of this guide to understand every line it creates.
Can robots.txt remove a page from Google?
No. Robots.txt stops crawling but does not keep a page out of Google. A blocked URL can still be indexed without a description if other sites link to it. To keep a page out of search, use a noindex tag or password protection, and leave the page crawlable so Google can see the tag.
Google’s documentation states that robots.txt “is not a mechanism for keeping a web page out of Google” and warns that a disallowed page can still be indexed if it is linked from other sites. It also notes that combining crawling and indexing rules can make them counteract each other. For example, if you block a page and also add noindex, Google cannot read the noindex tag because it is not allowed to fetch the page.
| Goal | Correct tool | Why |
|---|---|---|
| Reduce crawling of low-value URLs | robots.txt Disallow | Saves crawler time on filters and internal search pages |
| Keep a page out of search results | noindex tag or password | Robots.txt does not prevent indexing |
| Protect private data | Login or server rules | Robots.txt is public and only a request |
| Point crawlers to your pages | Sitemap line and XML sitemap | Helps discovery of important URLs |
Which robots.txt rules does Google follow?
Google supports four fields: user-agent, allow, disallow and sitemap. It ignores crawl-delay. When rules conflict, the most specific rule, measured by path length, wins, and the least restrictive rule wins if they are equal. Google processes the first 500 KiB of the file.
These details come from Google’s robots.txt specification page. Several of them surprise people:
- One group per crawler. A crawler follows only the most specific matching user-agent group. The star group is not merged with a named group.
- Case matters in paths. Field names are not case sensitive, but path values are.
- Wildcards are supported. A star matches any characters and a dollar sign marks the end of a URL.
- Caching. Google generally caches robots.txt for up to 24 hours, so changes may not apply immediately.
- Errors can stop crawling. If the file returns a server error, Google pauses crawling of the site for up to 12 hours while it retries.
- Sitemaps need full URLs. The sitemap field takes an absolute address and can point to a different host.
What should you block in robots.txt?
Block only URLs that have no value in search and waste crawling, such as internal search results, cart and checkout steps, admin areas and endless filter combinations. Do not block CSS, JavaScript or pages you want to rank. When unsure, allow crawling and control indexing with tags.
A simple starting file for a small business site looks like this in words: allow all crawlers, disallow the admin folder and internal search path, and list the sitemap address. A shop might also disallow cart and account paths. A staging site should block everything and also use a password, because robots.txt alone is not protection.
Common mistakes include leaving a staging rule of “Disallow: /” on the live site, blocking CSS and script folders that Google needs to render pages, and listing sensitive URLs, which advertises them to anyone who reads the file. After any edit, test the live file and check the robots.txt report in Search Console. The technical SEO guide shows how crawling, indexing and rendering fit together, and the SEO check glossary explains related findings such as blocked resources.
Should you block AI crawlers like GPTBot and ClaudeBot?
It depends on your goal. Blocking training crawlers such as GPTBot keeps your content out of model training, while blocking search crawlers such as OAI-SearchBot can remove you from AI search answers. Google-Extended controls Gemini training and grounding and does not affect Google Search ranking.
OpenAI separates its bots by purpose in its crawler documentation. OAI-SearchBot is used for search in ChatGPT, GPTBot is used for content that may train generative models, and ChatGPT-User is used when a person asks ChatGPT to visit a page, where robots.txt rules may not apply. OpenAI says changes can take about 24 hours to show in search results. Anthropic explains how to opt out of ClaudeBot with robots.txt in its help centre article. Google states that Google-Extended “does not impact a site’s inclusion in Google Search.”
| Crawler | Operator | Main purpose | Effect of blocking |
|---|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT search results | May stop your pages appearing in ChatGPT search answers |
| GPTBot | OpenAI | Training generative models | Signals your content should not be used for training |
| ChatGPT-User | OpenAI | Visits triggered by a user request | Robots.txt rules may not apply |
| ClaudeBot | Anthropic | Web crawling for model training | Opts the site out of that crawling |
| Google-Extended | Gemini training and grounding control | No effect on Google Search inclusion or ranking |
Many publishers block training crawlers and allow search crawlers, but that is a business decision. Compliance is voluntary, so robots.txt works only on crawlers that choose to follow it. For the wider picture of AI visibility, read the AI SEO guide.
Conclusion
Keep robots.txt short. Allow what should be found, block only low-value paths, add your sitemap line and treat AI crawlers as a deliberate choice. Never use the file to hide private pages. Test every change, and review the file after any redesign or migration. If you want an expert to check your crawling and indexing setup, Limi Creatives offers a free SEO audit, which you can request through the contact page.
Frequently asked questions
Do I need a robots.txt file?
No. Without one, crawlers assume everything is allowed. You need it only to block certain paths, point to your sitemap or set rules for specific crawlers.
Where does robots.txt go?
It must be at the top level of the host, such as the root of your domain. Copies in subfolders are not checked, and each subdomain needs its own file.
Does Google obey crawl-delay?
No. Google’s specification lists user-agent, allow, disallow and sitemap as supported fields and states that crawl-delay and other fields are ignored.
Will blocking GPTBot remove my site from ChatGPT?
According to OpenAI, GPTBot is used for training, while OAI-SearchBot controls search appearance. Blocking GPTBot alone is intended to affect training, not search results.
How do I test my robots.txt?
Open the file in a browser to confirm it loads, then use the robots.txt report in Google Search Console. Test a few allowed and blocked URLs to confirm the rules behave as expected.