
Refusing AI training with one line in robots.txt
Since 30 October, Google, OpenAI, Microsoft and Anthropic back one shared rule in robots.txt that lets you refuse AI training on your site. What the rule does, when it works, and why you should not just switch it on for an online store.
One rule instead of a list of bots
If you want to refuse AI training today, you keep a list in robots.txt: GPTBot for OpenAI, Google-Extended for Google, ClaudeBot for Anthropic, and every new bot on top. On 30 October, Google, OpenAI and Microsoft announced that they will follow one shared rule. The proposal comes from Anthropic and sits with the AI Preferences working group of the IETF, the organisation behind the standards of the internet.
The rule is called __usage_ai-training. It is not a bot but an agreement. Put that group in your robots.txt and you state in one go whether your content may be used to train generative AI models. It is still a proposal. It only becomes an official standard once the working group adopts it.
What the rule looks like
You add the rule as an ordinary group in robots.txt, with the Disallow and Allow lines you already know. The longest match wins, just as with search engines. This example refuses training on the whole site, except on the product pages:
# Only a preference about training: crawling and search stay as they were
User-agent: __usage_ai-training
Disallow: /
Allow: /products/What the rule does not do
- It is not a crawl ban. Bots can still fetch your pages. The rule only says what may happen to your content afterwards.
User-agent: *does not count. A general Disallow in your existing robots.txt does not count as a no to training. You have to name the group itself.- No rule means no preference. If the group is missing, you have said nothing. Microsoft reads that as permission.
- It works going forward, and it is voluntary. What is already in a model stays there. And a bot from a company that does not take part simply ignores the rule.
When it works
Nowhere yet. The companies have named a schedule, though:
- Microsoft: 1 February 2027. From then on, Bing reads the rule as soon as it crawls your site again.
- Google: early 2027. Google gives no exact date.
- OpenAI: in the coming months. OpenAI promises an update once it is ready.
- Anthropic: no date yet. Anthropic wrote the proposal, but we found no date for putting it into use.
Google and OpenAI say their own per-bot rules keep working. Microsoft does the opposite: after 1 March 2027 the old NoArchive and NoCache tags no longer count for training and AI answers.
Training is not the same as being found
For SEO, this is the most important part. Training means your content is used to build a model. That is separate from search, and separate from AI answers that use your page as a source at the moment they are written. But not every company draws that line in the same place.
- Google ties training to Gemini. If you say no with the new rule and do not name Google-Extended yourself, Google reads that as a no to Google-Extended. That rule covers two things: training Gemini, and using your pages as a source in the Gemini apps. So you disappear as a source there too. Google Search does not notice: Google-Extended does not count for inclusion or ranking.
- Microsoft keeps them apart. You refuse training through robots.txt or per page. Whether Copilot and the AI answers in Bing may use your page as a source is a separate setting per page. Your place in Bing's regular search results does not change.
- OpenAI only mentions training. At OpenAI the rule is about training foundation models. The announcement says nothing about search in ChatGPT.
If you want to stay a source in Gemini at Google, you have to allow Google-Extended yourself. Then Google may train as well. At Google it is still one switch.
And if you are on Shopify?
Shopify generates your robots.txt for you. You add a group of your own in the theme: create a file robots.txt.liquid in the Templates folder. It replaces Shopify's file completely. So start from Shopify's own example, which takes over the default rules from robots.default_groups, and put your group below it. Save a copy of your current /robots.txt first and compare it with the result. Shopify recommends this itself, because the template does not always give exactly the same file.
# Below Shopify's default groups: the blogs stay out of AI training
User-agent: __usage_ai-training
Disallow: /blogs/What this means for you
For most online stores, the question is not whether AI sees you, but which content you give away.
- Leave your product pages alone. They are there to be found, by AI too. Refusing gains you little there.
- Exclude content that is the product itself, per folder. An in-depth guide, your own recipes, research or a course. That is where training can cost you something: an answer that retells your work without anyone visiting your site.
- Do you use NoArchive or NoCache? From 1 March 2027 Bing no longer reads them as a preference. If you want to keep the same choice, set the new rules before that date.
- No rush, but make a choice. Nothing works yet. Microsoft starts on 1 February 2027, so decide before then.
On brandbrand.nl the rule is not there. Our articles are there to be read, by AI too.
Our position
An online store wants to be found, by AI too: we only exclude training for content that is the product itself.Mels, SEO Specialist at Brandbrand



