The smallest file that can cost you everything

What it is

A plain text file at the root of your domain. Crawlers fetch it before anything else and follow what it says.

It lists paths a crawler should not request, optionally per crawler, and usually points at your sitemap.

It is a request rather than a lock. Well behaved crawlers obey it and badly behaved ones ignore it entirely.

It applies per protocol and per subdomain, which surprises people. The file on your main domain says nothing about a subdomain, and each needs its own if you want rules there.

It applies per protocol and per subdomain, which surprises people. The file on your main domain says nothing about a subdomain, and each needs its own.

The line that hides a site

A disallow rule covering the root blocks everything. It belongs on a staging site and nowhere else.

It survives launches more often than anybody expects, because the launch checklist covered content rather than configuration.

If organic traffic ever disappears overnight, this file is the first thing to read. It takes ten seconds.

It is worth checking after any hosting migration too. Some platforms generate this file automatically and will happily replace yours with a default you did not choose.

Check it after any hosting migration too. Some platforms generate this file automatically and will replace yours with a default you did not choose.

Crawling is not indexing

This is the distinction that causes the most confusion. Blocking a page stops it being fetched. It does not reliably keep it out of results.

A blocked page can still appear, listed from links pointing at it, with no description because nothing could be read.

To keep a page out of results, allow it to be crawled and give it a noindex instruction. Blocking it prevents the instruction from ever being seen.

To keep a page out of results, let it be crawled and give it a noindex instruction, which is covered in noindex and the robots meta tag.

It is public

Anybody can read yours by adding the filename to your domain. So can a competitor and so can anybody curious.

Never use it to hide sensitive paths. Listing a private folder there is an advertisement for it.

Protect private things with authentication. The robots file is traffic management, not security.

That visibility cuts both ways usefully. You can read anybody else's, which is occasionally instructive about how a larger site organizes itself.

What belongs in it

On most small sites, very little. A pointer to the sitemap and perhaps a rule for a search results path that would otherwise generate endless addresses.

Resist blocking scripts and stylesheets. Crawlers need them to see the page the way a visitor does.

If you cannot explain why a line is there, it probably should not be.

Naming the sitemap is the one line nearly every site should have. It is how a crawler arriving without any other context learns what you actually publish.

• A pointer to your sitemap
• A rule for an internal search path that would otherwise generate endless addresses
• Almost nothing else, on most small sites
• Never a block on scripts or stylesheets
• Never a private path you are trying to hide
• Nothing you cannot explain the reason for

Checking yours

Fetch it in a browser. Read it. It should be short enough to understand in a minute.

Search Console reports how it was interpreted and will tell you if pages are blocked.

Check it after every redesign and every hosting change. Those are the two moments it changes without anybody deciding.

Fetch it in a browser and read it. It should be short enough to understand in a minute, and if it is not, that is itself the finding.

How ours is handled

Sites published through Website Builder Studio serve a robots file that names the sitemap and does not block content, and the published site is checked before anything goes live.

There is no staging rule to leave behind, because the preview and the live site are built by the same process.

Every page also passes a check that runs before anything publishes, which includes confirming the page is actually reachable.