How a search engine finds your pages in the first place

How the crawler finds you

It starts from pages it already knows about and follows the links on them. That is the whole mechanism.

Your sitemap gives it a list to check as well. A link from another site is often how a brand new domain is found at all.

So a page with no link pointing at it, and no place in the sitemap, may as well not exist. That is the most common cause of a missing page.

Internal links are the part you control completely. A page linked from your main navigation is found quickly. A page linked once from a paragraph deep in the site waits much longer.

Internal links are the part you control completely, which is why they matter more than any submission. See internal linking.

What it reads

It fetches the page the way a browser would, then reads the markup. Modern crawlers can run scripts, and that step is slower and less certain.

Text in the markup is read first and most reliably. Text that only appears after scripting may be read later, or in part.

So the safest place for anything you want found is in the markup that arrives with the page.

Text that only appears after scripting may be read later, or in part, or not at all — which makes the markup that arrives with the page the only reliable place for anything you need found.

What stops it

Four things account for most of it. A robots file blocking a path. A menu that only works with scripting. A page with no links to it. A server too slow to respond.

Blocked paths are the sharpest. A single line in your robots file can hide an entire section, and it usually happens by accident during a redesign.

Broken links waste the effort too. A crawler that follows three dead links on a page learns to come back less often.

Infinite address patterns waste it too. Filters and sort options that generate a new address for every combination can absorb enormous crawling effort on a site that only has forty real pages.

• A robots file blocking a path
• A menu that only works with scripting
• A page with nothing linking to it
• A server too slow to respond
• Redirect chains that never resolve
• Endless filter and sort addresses absorbing the effort
• Broken links, which teach a crawler to come back less often

How often it comes back

It depends on how often your pages change and how important the crawler judges them. A busy news site is visited constantly. A small business site is visited far less.

You cannot buy more of it. You can earn it by publishing real changes and by making the site fast to fetch.

Do not expect a new page to appear the same day. Days to weeks is normal for a small site, and that is not a fault.

A page that has not changed in two years is checked rarely, which is correct. That is also why editing a page slightly to look fresh achieves nothing: the engine compares what changed, not when you saved it.

Crawl budget, honestly

There is a limit to how much a crawler will fetch from one site. For a site of a few hundred pages you will never reach it.

It becomes a real concern at tens of thousands of pages, or where a site generates endless filter and sort addresses.

If you have under a thousand pages, spend your attention on whether the right pages are linked rather than on this.

Below a thousand pages you will never reach the limit, and attention spent on it is attention not spent on whether the right pages are linked.

Helping it along

Keep a sitemap that lists what actually exists. Link every page from somewhere real. Fix links that return nothing.

Serve pages quickly, because a slow site is fetched less. Keep the structure shallow enough that nothing sits six clicks from the homepage.

Every page written by Website Builder Studio is checked before anything publishes, and the sitemap is rebuilt from what is really there rather than from a list somebody maintains.

Return the right response codes. A page that is gone should say so. A page that moved should redirect once, to the final address, rather than through a chain of two or three hops.

How to tell what happened

Search Console shows which pages were fetched, when, and what the server returned. It is the only honest source for this.

Your server logs show the same thing from your side, including crawlers Search Console does not report.

Guessing is the alternative, and guessing about crawling produces most of the bad advice in this field.

Look for patterns rather than single events. One failed fetch is noise. The same section failing every week is a fault worth an afternoon.

Look for patterns rather than single events. One failed fetch is noise; the same section failing weekly is a fault worth an afternoon.