Keeping a page out of search, properly

The two places it lives

As a meta tag inside the page markup, or as a header sent with the response.

The header version is useful for files that are not markup, such as documents, where there is nowhere to put a tag.

Both are equally effective. The header is easier to miss when you are debugging, because it is invisible in the page source.

A crawler reads whichever it finds, and if both are present and disagree, the more restrictive instruction generally wins. That is worth knowing before you spend an afternoon debugging.

If both are present and disagree, the more restrictive instruction generally wins — worth knowing before spending an afternoon debugging.

Why the page must stay crawlable

The instruction is inside the page. If a crawler is blocked from fetching the page, it never reads the instruction.

So blocking a page in robots and adding a noindex tag achieves nothing. The two cancel out.

Let the page be crawled. Tell it to be excluded. That combination works reliably.

What to exclude

Thank you pages after a form. Internal search results. Duplicate printable versions. Staging and preview addresses.

Thin pages that exist for administrative reasons rather than for readers.

Not your ordinary content. If you find yourself excluding real pages, the problem is usually that they are too similar to each other.

Paginated listings are a common question. Usually leave them alone. Excluding page two of a listing can cut off the route to everything listed beyond it.

• Thank you pages shown after a form
• Internal search results
• Printable duplicates of a page
• Staging and preview addresses
• Campaign landing pages kept out of navigation deliberately
• Thin pages that exist for administrative reasons

The relaunch disaster

Most sites are built with everything excluded so the unfinished version stays private. Removing that at launch is one checkbox.

Forgetting it empties your search presence within weeks, and the symptom is a gradual disappearance rather than an obvious break.

Check it on launch day, and check it again a week later with the inspection tool. It is the single highest value two minutes in a launch.

The same applies to a staging site that is publicly reachable. If it can be crawled, it should carry the exclusion, or you end up competing with yourself for your own content.

Check it on launch day and again a week later with the inspection tool. It is the highest-value two minutes in any launch.

Other directives

Nofollow on a page asks the engine not to follow the links on it. It is rarely what a small site wants.

Directives limiting snippets or image previews exist and are mostly for publishers with particular concerns.

Leave them alone unless you have a specific reason. The defaults are what you want.

Leave them alone unless you have a specific reason. The defaults are what you want, and nofollow on a page can cut off parts of your own site.

Removing a page from results quickly

Add noindex, then use the removals tool in Search Console for a temporary hide while the change is picked up.

The removal is temporary and the noindex is what makes it stick. Using the tool alone means the page returns.

If the page is genuinely gone, returning a proper missing page response is the simplest permanent answer.

Checking it

Use the inspection tool and read what the crawler saw, because that catches a header you cannot see in the page source.

Check your important pages, not just one. Directives are usually applied by template, so the fault is rarely isolated.

Every page written by Website Builder Studio is checked before anything publishes, and a page that should be found carrying an exclusion is exactly the kind of fault that check exists to catch.

Check documents as well as pages. A file served with an exclusion header looks perfectly normal when you open it, which is exactly why nobody finds it.

Check documents as well as pages. A file served with an exclusion header looks perfectly normal when you open it, which is exactly why nobody finds it.