Sitemaps, robots.txt and indexing, in plain words

What a sitemap is and is not

sitemap.xml is a plain list of addresses, one per page, with an optional lastmod date beside each. Crawlers read it to find pages and to see which ones changed. It is not a promise of indexing and not a ranking signal. It is a map.

It should list only pages you want indexed, at their clean canonical address, and it should be accurate. A sitemap that lists redirects, removed pages or two versions of the same page teaches the crawler to distrust it.

What robots.txt does

robots.txt sits at the root of your domain and tells crawlers which paths to leave alone. It is a courtesy the major engines honor, not a lock. Use it for things that should never be crawled: an admin area, a search-results page, a duplicate print version.

It is not the way to keep a page out of the index. A page blocked in robots.txt can still be indexed from links alone, with no description. To keep a page out, let it be crawled and mark it noindex.

Most business sites need a short robots.txt that allows everything and points to the sitemap. That is all.

Canonical addresses

One page can be reached at several addresses: with and without www, with a trailing slash, with a tracking code. The canonical link in the page head says which one counts. Crawlers merge the rest into it.

Without it, an engine may treat the versions as separate pages, split whatever they earn between them, and pick a version you did not intend. With it, there is one page.

What lastmod is for

lastmod is the date the page's content last changed. Google says it uses the value when it is accurate and consistent, and it ignores it when every page claims to have changed today. Update it only when the content actually changed, and let the boring pages keep their old dates.

Indexing is not ranking

Indexed means the engine fetched the page, read it and stored it. That is the ticket in. Ranking is what happens when someone searches and the engine sorts what it has stored. A page can be indexed and never shown, because for every search the engine found pages it judged more genuinely useful.

So a page missing from results is one of two problems. Either it was never indexed, which Search Console's Pages report will tell you, or it is indexed and not competitive, which is a content question. The fixes are different, so read the status first.

Why some pages are not indexed

Each of those has a plain fix, and only the first one is about the writing. Read the status before you rewrite anything.

• Crawled, currently not indexed: fetched and judged not to add enough. Add substance.
• Discovered, currently not indexed: known but not yet fetched. Wait.
• Duplicate, Google chose a different canonical: two versions, and it picked one.
• Excluded by noindex tag: you or your platform told it not to.
• Blocked by robots.txt: it could not look.

Submit once, then be patient

Submit the sitemap in Search Console once. Google re-reads it on its own visits. A new address in the sitemap does not need a new submission. A new domain is crawled lightly at first; weeks before the reports fill is normal, and nobody can give you an honest number.

Website Builder Studio generates a sitemap for every site and gives each page a clean address and a canonical link. Each page passes a check that runs before anything publishes, validated against Google Search Essentials and modern web standards. The in-app assistant walks you through submitting the sitemap to Search Console step by step.

What it cannot do is make a crawler come sooner, and it will not pretend to. See how to add your site to Google Search for the actual screens.