One line instead of a list
OpenAI and Google both said on 30 October that they will honour a single shared token in robots.txt through which a website can express whether it wants its content used for generative AI training. Until now a site owner had to name each crawler and keep a separate rule for each provider. OpenAI’s post puts the problem plainly: operators “would identify individual crawlers and maintain separate rules for each and every provider”, and a shared signal “reduces that work”.
The token itself is __usage_ai-training. It is defined in an individual Internet-Draft at the IETF, filed on 30 October by Kevin Kelley of Anthropic, and it is used as a User-Agent group key inside an ordinary robots.txt group — which means it inherits the path rules a publisher already knows how to write, and can be applied to one directory rather than a whole site.
The dates do not line up
Neither company has implemented it. OpenAI says it aims to do so “in the coming months” and will publish another update with implementation details when it does. Google says it aims to support the control “along with other AI providers in 2027”.
The draft they are both pointing at is a -00 individual submission. The IETF datatracker records it as Standards Track in intent but notes that an individual draft has no formal standing in the standards process and is not endorsed by the IETF. It expires on 3 May 2027. On the published timelines, the document defining the token can lapse before the larger of the two companies starts honouring it. That is normal for IETF drafts, which are routinely refiled, but it is worth stating plainly: publishers adding the line today are writing against a document with an expiry date and two implementation promises, not a standard.
What it does not do
Both companies are explicit that the new signal sits on top of the old controls rather than replacing them. OpenAI says its existing crawler controls “will continue to apply and be respected”. Google says existing company-specific controls will keep working alongside the joint signal.
Google adds the piece of interpretation that matters most for anyone who already maintains a robots.txt file. If a site uses the policy token to opt out of training and does not explicitly name Google-Extended, Google plans to read that as though the publisher had disallowed Google-Extended — the control it has offered since 2023. Sites that explicitly allow Google-Extended stay eligible to appear in Gemini Apps answers and receive clicks through grounding. So the broad signal and the per-company control remain separable, and a publisher who wants training blocked but grounding allowed still has to write both lines.
Neither announcement says anything about content already collected. A robots.txt change is forward-looking by construction; it does not reach into datasets that exist or models already trained.
Why now
Google ties the proposal to regulation directly, writing that it “is consistent with the EU Code of Practice for General Purpose AI (GPAI) Model Providers under the EU AI Act that recognizes robots.txt as a primary method for web publishers to express machine-readable preferences”. It also says the draft reflects feedback from publishers, civil society groups and regulators who have asked for industry-wide controls.
That is the honest framing. A voluntary token honoured by two companies on unspecified schedules is not enforcement. But it is the mechanism the EU’s own code of practice already names, which makes adoption cheaper than the alternative of arguing about what counts as a valid machine-readable preference.
What to watch
Three things. Whether the draft moves from an individual submission into the IETF’s AI Preferences working group as an adopted document, which is what would make it a standard rather than a proposal. Whether other providers follow — OpenAI closes its post by encouraging them to. And whether either company publishes a date, rather than a season, for turning the signal on.
