Whether to allow AI crawlers, and what robots.txt can do

What a crawler is and what robots.txt does

A crawler is a program that visits pages and reads them. Search engines use them to build an index, and many AI products use them for their own reasons.

Robots.txt is a plain text file at the root of your site. It lists which crawlers may visit which paths. Our guide to robots.txt covers the syntax.

It is a request that honest crawlers follow. It does not stop a program that ignores it, and it is not a way to hide private material.

The three roles an AI crawler can have

Think about what the visitor does with your page afterward. The purpose matters more than the name, and each product publishes which role its crawler serves.

Crawlers that collect AI training data are usually separate from search crawlers, and vendors typically document each crawler and its robots.txt token. Use that documentation as your source instead of a blog post, since details change.

• Training: collects text to teach a model. Your page may shape what it learns.
• Live fetching: reads a page at the moment a person asks an assistant about it.
• Search indexing: builds a searchable index that later links back to you.
• Some products use one crawler for more than one role.

What blocking each one can mean

Blocking a training crawler asks that model makers not use your pages to learn. It generally does not change whether you appear in ordinary search. Google, for example, offers the Google-Extended robots.txt token to opt out of Gemini training use without affecting Search.

Blocking a live-fetch or search crawler can mean the assistant cannot read your page when someone asks about you. Your business may then be absent from that answer.

That is the tradeoff. Being readable helps an answer engine describe you correctly. Being closed protects your text from reuse.

A way to decide

Start from what you want. A small local business usually wants to be found and described correctly, so most are comfortable leaving search and live-fetch crawlers allowed.

Training is the open question. Some owners allow it, and some prefer to opt out on principle. Neither choice is wrong, and you can change your mind later.

If your content is paid or private, put it behind a login. Robots.txt is the wrong tool for that.

Ask one more question: would a customer be better served if an assistant could read your prices, hours and service list? For most small businesses the honest answer is yes, and that favors leaving the doors open.

What Website Builder Studio gives you

You choose whether Google may use your content to train its AI. Search indexing is unaffected either way, so the choice does not take you out of ordinary search.

For how answer engines read a site, see how answer engines read a website. A file called llms.txt is a separate idea, explained in our page on llms.txt.

Either way, a genuinely useful page is what gives an answer engine something worth reading.

Check what you actually have

Add /robots.txt to the end of your web address in a browser and read the file. Many sites carry rules added long ago by a plugin or a developer, and the owner has never seen them.

Make sure nothing blocks your whole site by accident, and that your sitemap is listed. Test after every change, since a single misplaced line can hide a whole site.

Rules differ between vendors and change over time, so confirm any specific crawler against its maker's current documentation.

Keep your options open

Your choice is not permanent. A file can be edited in minutes, and a crawler that was blocked can be allowed later. Write down why you chose what you did, so the next person understands.

Review it once a year. New crawlers appear, old ones change purpose, and your own view may shift.