AI crawlers and permissions
A site can tell AI systems and search engines what they may use its content for. This is done on the Content-Signal line of the site's robots.txt file. The line comes from Cloudflare's Content Signals Policy and is related to the vocabulary the IETF AI Preferences (aipref) working group is standardising. Honouring it is voluntary for each AI operator.
By default, sites have no Content-Signal line. The site's robots.txt allows everything, so search engines and AI systems may use the content freely. For a marketing site this is usually the right choice, because its job is to be found.
If you want to restrict use, ask Kamu in Studio or the Assistant, or your own agent, to add the line, for example: "Block my content from being used for AI training."
To see which AI bots actually fetch your pages, check the bot card in Visitor analytics.
The three permissions
search: whether the content may appear in search results (traditional search engines and AI-based search engines like Perplexity).ai-input: whether an AI assistant (e.g. ChatGPT, Claude) may fetch your page and use it to answer a user's question. This is how people find businesses through AI assistants.ai-train: whether your content may end up in AI models' training data, i.e. become part of future models. This is different fromai-input: training happens once in advance, input happens every time someone asks.
Each value is either yes or no.
When to choose no
Most businesses have no reason to restrict any of these, but a restriction can be justified:
ai-train=no: if your content is unique expert writing you do not want absorbed into general AI models. People can still find you via an AI assistant.ai-input=no: if you do not want AI assistants to fetch your pages at all. This is a strong restriction and weakens your visibility in AI answers.search=no: effectively hides your site from search engines. A very rare choice for a public marketing site.
Changing the settings
Example requests:
- "Block training use but allow everything else."
- "Block AI search entirely."
- "Remove the AI restrictions from robots.txt." (restores the default)
After publishing, the change shows up in your site's robots.txt (at https://example.com/robots.txt), for example like this:
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
Sitemap: https://example.com/sitemap-index.xml
Notes
- Content-Signal is a declaration, not a technical barrier. It relies on AI operators reading and respecting the file, the same way traditional
robots.txtworks. - The settings only apply going forward. If your content is already in some model's training data,
ai-train=nodoes not remove it. - You can check your site's current state at isitagentready.com by entering your site's address.