What Is robots.txt?

robots.txt is a plain text file at the root of your website, at /robots.txt, that tells web crawlers which parts of the site they may or may not request. It follows the Robots Exclusion Protocol, standardized as RFC 9309 by the IETF in September 2022. Compliance is voluntary: well-behaved bots like Googlebot and GPTBot obey it, a badly-behaved one can ignore it, and it does not hide a page from search.

What does robots.txt control?

robots.txt controls crawling, not indexing. It tells a crawler which URLs it may fetch, which saves your server from wasted requests and keeps bots out of areas like admin paths or search-result pages.

Here is the part that trips people up. Blocking a URL in robots.txt does not remove it from Google. If other pages link to that URL, Google can still list it in results without the description, because it was told not to read the page.

To keep a page out of search results, you use a noindex rule in the page's HTML or an HTTP header, not robots.txt. The two tools do different jobs. This is closely tied to indexing, the step where Google stores a page so it can appear at all.

What does a robots.txt file look like?

A robots.txt file is a short list of rules grouped by which crawler they target. Each group names a user-agent, then lists what it may and may not fetch.

Directive	What it does
User-agent	Names the crawler the rules apply to
Disallow	Blocks a path from being crawled
Allow	Permits a path inside a blocked area
Sitemap	Points crawlers to your sitemap file

A simple file might tell every crawler to stay out of the admin folder while allowing everything else, then point to the sitemap. The Sitemap: line is a common and useful addition, since it helps crawlers find your full URL list in the same place.

robots.txt and AI crawlers

robots.txt is now the main control for AI crawlers too, and the same voluntary-compliance rule applies. GPTBot (OpenAI's training crawler) and ClaudeBot (Anthropic's crawler) both read robots.txt and obey it. Blocking them stops those bots from fetching your pages.

This is a real decision, not a technical default. Blocking AI crawlers keeps your content out of their training and, for some bots, out of AI answers that could cite you. The first thing we check on a site that wants AI mentions is whether its own robots.txt is quietly blocking the crawlers it needs.

For guidance meant specifically for AI systems rather than crawl control, there is a separate proposal, llms.txt. It sits alongside robots.txt and does a different job. Whether to allow or block these bots is covered in should you block AI crawlers.

Common robots.txt mistakes

The costliest mistake is a leftover Disallow: / from a staging build, which tells every crawler to skip the entire site. It goes live with the launch and quietly makes the whole site invisible, a classic cause of a website not showing on Google.

Two others are common. Blocking your CSS and JavaScript files stops Google from rendering the page the way visitors see it, which can hurt how it is judged. And treating robots.txt as a security measure is a mistake, because the file is public and listing a secret folder in it just advertises where to look.

Frequently asked questions

Can robots.txt remove my page from Google?

No. robots.txt controls whether a crawler fetches a page, not whether Google shows it. A blocked page can still appear in results if other sites link to it. To keep a page out of search entirely, use a noindex rule or password-protect it, and do not also block it in robots.txt, or Google cannot see the noindex.

Where does the robots.txt file go?

At the root of your domain, reachable at yoursite.com/robots.txt. It only applies to the exact host, protocol, and port it sits on, so a file on the main domain does not cover a subdomain. If the file is anywhere else, crawlers ignore it and assume everything is allowed.

Is a robots.txt file required?

No. If a site has no robots.txt, crawlers assume they may fetch everything, which is fine for many small sites. You add one when you have areas to keep out of the crawl, or when you want to point crawlers to your sitemap. An empty or missing file is not an error.

Does robots.txt block all AI crawlers?

Only the ones that choose to obey it. Reputable crawlers like GPTBot and ClaudeBot respect robots.txt rules. A crawler built to ignore the standard can read your pages anyway, since compliance is voluntary under RFC 9309. For content that must stay private, use authentication, not robots.txt.

What is the difference between robots.txt and a noindex tag?

robots.txt says "do not fetch this page"; a noindex tag says "you may fetch it, but do not show it in results". Use robots.txt to manage crawl load and keep bots out of low-value areas. Use noindex when a page can be read but should never surface in search, like a thank-you page.

Getting crawling right

Storming Solutions runs SEO and AI-visibility work for Malaysian businesses. A misconfigured robots.txt is one of the first things we rule out when a site is missing from search or from AI answers. We check what your file is blocking before we touch anything else, because the quickest wins are often a stray line removed.

If your pages or citations have gone quiet, send us the site on WhatsApp and we will read your robots.txt with you, or start with the SEO in Malaysia pillar. Our SEO, AEO, and GEO work exists to keep the right pages open to the right crawlers.