If you want search engines to find your website but have reservations about your content being used to train AI models, what can you control?
Until now, website owners have largely relied on instructions specific to individual AI providers. That may become simpler.
On 30 October 2026, Google announced plans to support a shared AI-training control through robots.txt, based on work by the AI Preferences (AIPREF) working group at the Internet Engineering Task Force (IETF).
The proposed approach would allow website owners to express an AI-training preference across participating providers without maintaining a separate instruction for every company.
It is a useful development, particularly for smaller websites. But before changing anything, it helps to understand what the new control does, what it doesn’t do and how it relates to existing settings.
What is the proposed AI-training control?
Robots.txt is a file that website owners use to communicate instructions to web crawlers.
Traditionally, these instructions have focused on which parts of a website particular crawlers may access.
AI training introduces another consideration: whether content collected from a website may be used to train generative AI models.
The proposed shared mechanism uses a special policy token in robots.txt to express this preference.
Its purpose is to reduce the need for website owners to maintain separate training-related instructions for every participating AI provider.
However, this is still a developing mechanism. Google’s implementation is planned for 2027, and the broader standardisation work is ongoing.
Website owners should not assume that every AI provider already recognises or honours the shared signal.
AI training, search crawling and AI-generated answers are different
This distinction is important because these activities are sometimes discussed as though they are interchangeable.
Search crawling allows search engines to discover and access website content.
AI training involves using content to develop or improve AI models.
AI grounding can involve retrieving or supplying information from sources when an AI system generates a response.
A website owner’s preference about AI training does not necessarily control all three activities.
Different providers may interpret and implement their controls differently.
For example, Google already offers Google-Extended, a robots.txt product token that controls certain uses of website content for Gemini model training and grounding in Gemini Apps.
Google has explained that when a website uses the proposed shared signal to opt out of training, and has not explicitly specified Google-Extended, it plans to treat the shared preference as a Google-Extended disallow instruction.
That matters because Google-Extended covers more than model training alone.
A website that specifically allows Google-Extended may also be eligible to appear in Gemini Apps responses through grounding.
In other words, website owners should understand the possible implications for AI-generated experiences before assuming the proposed control affects training alone.
Will the new control affect Google Search rankings?
The proposed shared signal is not a new Google Search ranking factor.
Google’s existing Google-Extended control does not determine whether a website can appear in ordinary Google Search results, nor is it used as a Google Search ranking signal.
Nevertheless, website owners should avoid treating all crawler instructions as equivalent.
A change intended to express an AI-training preference is different from a change that restricts ordinary search crawlers.
Understanding that difference is especially important for smaller websites that depend on organic search visibility.
Should you change your robots.txt file now?
For most website owners, the immediate priority is understanding their existing configuration rather than rushing to implement the proposed shared mechanism.
Three questions can help.
1. Which activity are you trying to control?
Are you concerned about AI model training, ordinary search crawling or the use of your content in AI-generated responses?
These are different objectives, and they may require different controls.
2. What instructions already exist?
Review your current robots.txt file.
Check whether it contains instructions for Googlebot, Google-Extended or other provider-specific crawlers.
If your website uses WordPress, remember that robots.txt may be generated virtually or managed through plugins or server configuration.
Avoid editing it without first understanding how your website currently manages the file.
3. What will change when the shared mechanism is supported?
Follow official announcements from the providers whose services matter to your website.
Check implementation dates, the interaction with existing instructions and any limitations before deciding whether to adopt the shared signal.
There is no immediate benefit in adding an instruction without understanding whether the relevant provider currently supports it.
What does this mean for smaller websites?
For a smaller website, technical decisions should serve a clear purpose.
The introduction of a shared AI-training preference may eventually make it easier to manage how content is used across participating providers.
But it does not replace the fundamentals of maintaining a searchable website.
Continue checking whether important pages can be crawled and indexed, whether your internal links help visitors discover relevant content and whether Google Search Console shows meaningful changes in search performance.
If your concern is AI training, review the available controls separately from your ordinary search optimisation work.
The important question is not simply whether a new control exists, but whether it addresses the outcome you want for your website.
Final thoughts
The proposed shared AI-training control is a step towards giving website owners a more consistent way to communicate their preferences.
Its practical value will depend on how widely it is adopted and how participating providers implement it.
For now, understanding your existing settings and following the development of the proposal is a sensible starting point.
You do not need to make every technical change as soon as it is announced.
Sometimes, the more useful first step is knowing exactly what a change would accomplish.
Further reading
- Google Search Central: Supporting a Shared AI Training Control for Websites (30 October 2026).
- Google Crawling Infrastructure: Google’s Common Crawlers — Google-Extended documentation.
- SEO Academy: Technical SEO, crawling and indexing lessons.
Practical next step
If you are reviewing your website’s technical settings, the SEO Best Practices SEO Audit Workbook can help you organise a broader review of your website’s technical and on-page SEO elements.
Keep learning with The SEO Note. Receive useful search updates, new SEO Academy lessons and practical resources when there is something worthwhile to share.