Google, Microsoft, OpenAI, and Anthropic say they will honor one robots. txt group as a cross-company opt-out from AI training
It governs how content is used rather than whether bots can crawl it, defaults to allowing training, and does nothing for content already scraped
Anthropic has started implementing it, OpenAI promises to do so in the coming months, Microsoft starts February 1, 2027, and Google targets 2027, while Meta is supportive but uncommitted
Google and Microsoft have joined OpenAI and Anthropic in promising to honor a single line in robots. txt that tells AI developers whether a website’s content can be used to train their models. Until now, a site that wanted to say no had to write a rule for each company’s crawler, and sometimes for several crawlers run by the same company. The shared signal is meant to replace that sprawl with a single entry that every participating provider reads the same way. That is genuine progress for overworked webmasters, but the promise is narrower than much of the coverage suggests: The new line states a preference about how content may be used rather than stopping bots from fetching it; it is not an approved standard, does nothing about content already collected, and none of the four major players fully honor it yet. Doing the bare minimum with differing definitions of what the commitment entails
OpenAI’s announcement drew much of the attention, but the company casts itself as one participant among several in an Internet Engineering Task Force (IETF) effort, and the mechanism isn’t its own. It comes from an Internet-Draft published the same day by Kevin Kelley of Anthropic, which names a policy token, "__usage_ai-training", in the User-Agent line of an otherwise ordinary robots. txt group. Training is defined by the IETF’s AI Preferences (AIPREF) working group as the process of changing a generative model’s learned parameters, excluding models used purely for tasks such as classification or ranking. OpenAI describes what it will honor as "foundational generative AI training," which sounds narrower than the draft’s definition. Calling it a standard may be premature: The IETF’s own tracker labels the document an individual draft that the IETF has not endorsed, and that has no formal standing, and it lapses on May 3, 2027 unless revised. Anthropic is the only one of the four major AI players to say it has started. OpenAI aims to implement the signal over the next few months but has given no date. Microsoft will begin honoring the shared preference as it crawls from February 1, 2027, while Google is aiming for 2027. Each company is also wiring the line into its own controls differently. Google says that when a site uses the shared token to opt out without separately specifying Google-Extended, it plans to treat that "as if the publisher had disallowed Google-Extended." Because Google-Extended also governs grounding in Gemini Apps, a site that only wants to stop training could lose its citations and clicks in Gemini answers too, and Search Engine Journal notes that Google’s documentation describes no way to separate the two. Interestingly, Microsoft goes the other way…