One robots.txt line now asks four AI companies not to train on your site. We tested it, and the order of your lines matters

On Oct 30, OpenAI, Google and Microsoft said they will honor one shared robots.txt signal for opting a website out of generative AI training, and Anthropic engineer Kevin Kelley, who wrote the draft spec, told the IETF's AI Preferences group that Anthropic has started building support. Until now a site had to write a separate rule for each company's crawler or token (GPTBot, Google-Extended, ClaudeBot and so on).

What it looks like. It is an ordinary robots.txt group, but the User-Agent line names a "policy token" instead of a crawler:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Per the draft, that means "no AI training" for the whole site, except files under /press-kit/. It covers training only. It does not tell any bot what it may crawl, and it does not affect search rankings.

Who reads it, and when. Nobody yet. Microsoft says it will honor it from Feb 1, 2027. Google says 2027 (its crawler docs say early 2027). OpenAI says "in the coming months." Anthropic gave no date. Posts on X saying the opt-out is already live overstate it.

What we tested. On our Linux test machine we fed the draft's example, and three variations, to Python 3.13's built-in robots.txt parser (urllib.robotparser), the kind of stock parser many site checkers and scripts use. We asked it what the token, Googlebot and GPTBot may do on /stories/x and /press-kit/logo.png.

• Your other bots are unaffected. Adding a __usage_ai-training group to an allow-all file changed nothing for Googlebot or GPTBot. That matches the draft's design goal.
• Trap 1: line order. With the draft's exact example, the parser said the token is disallowed on /press-kit/ too. Python's parser takes the first matching line; the draft uses RFC 9309's longest-match rule, under which the more specific Allow: /press-kit/ wins. When we put the Allow line first, both readings agree. Our fix: list your Allow exceptions above Disallow: /.
• Trap 2: wildcards. A file with only User-Agent: * and Disallow: / made the parser report the token as disallowed. The draft says the opposite: a wildcard group "MUST NOT" count, and a file with no __usage_ai-training line has expressed "no preference." If you check your opt-out with a generic robots tool, it can tell you that you are opted out when, under the draft, you are not.

The catches. It is a voluntary preference, not a lock: scrapers that ignore robots.txt will ignore this too. It does not remove anything already collected or trained on. The draft is an individual Internet-Draft that the IETF has not endorsed, and it expires May 3, 2027 unless revised. At Google, an opt-out with this token (when you do not mention Google-Extended) will be treated as a Google-Extended block, which per Google's docs also covers grounding in Gemini Apps. If you want Gemini citations, add an explicit Google-Extended allow group.

Our verdict. Worth adding now if you do not want your content used for AI training: it costs two lines and existing crawlers ignore it. Write it with the token named explicitly, put Allow lines first, and keep your existing GPTBot and Google-Extended rules, which all four companies say still apply.

We tested only how a standard parser reads these files. We did not test any company's crawler, because none of them reads the token yet. Test script and output are kept in our test notes.

MODEL Editors