Until now, a publisher who wanted to keep its work out of AI training had to tell each AI company separately, in whatever form that company preferred. OpenAI, Google, Microsoft and Anthropic now say they will support a single signal in robots.txt that reaches all of them at once. The catch is the calendar. One of the four has finished the work, and the rest have plans.
One signal instead of one per company
The proposal came from Kevin Kelly, a technical specialist at Anthropic who works on standards, and it is going through the IETF. The mechanism is simple in outline. A site owner adds a shared signal to robots.txt, and every participating AI provider reads it as the owner's preference about training.
Robots.txt is an old tool. Websites have long used it to tell search crawlers and other automated systems what they may and may not fetch. What it lacked was a common way to say "not for model training," so owners had to state that separately for each AI company.
All four companies say their AI crawlers will back off from a site when they detect the new signal. The report does not give the exact directive or syntax, so that detail will have to come from the standard itself.
Who ships when
Support is not simultaneous, and the schedule is worth reading closely.
| Company | Status |
|---|---|
| Anthropic | Support already in place |
| OpenAI | Planned for the coming months |
| Planned for 2027 | |
| Microsoft | Planned for 2027 |
So the company that proposed the standard has implemented it, and two of the four others will take until next year. A publisher who adds the signal today is making a request that, for now, only one of the four is certain to read.
Microsoft adds a second dial
Microsoft says it will offer finer control than the shared signal alone. Publishers will be able to forbid use of their material for training separately from use for grounding. Grounding is when a model pulls verified information from outside sources while building an answer.
The two are different bargains. Training takes content into the model. Grounding fetches it at the moment someone asks a question. A publisher might reasonably object to the first and tolerate the second, or the reverse.
Microsoft also says that sites will keep appearing in Bing search results whatever they choose. Opting out of AI use does not mean disappearing from search.
Google already offers something similar. Its existing controls cover how content is used in AI features of search and in model training, again without costing a site its place in ordinary search results.
Meta offers a polite maybe
Not every large AI developer has committed. Meta spokesperson Andy Stone backed the idea of a common mechanism and said the company is ready to work with other AI developers on the standard and its rollout. He stopped short of saying Meta will actually obey the bans.
That is a careful sentence. Supporting a shared mechanism and promising to honor it are separate acts, and only the second one keeps a crawler off your pages.
What the signal does not do
The new signal, by itself, offers no protection against crawlers run by companies that do not support it or do not follow it. Its force comes from the willingness of each company to comply, which is why Meta's answer matters even though Meta is only one name on a longer list.
Other controls are developing alongside it, and they work differently:
- Really Simple Licensing (RSL) lets a publisher charge crawlers for access to its material.
- Cloudflare blocks AI crawlers by default.
One of these puts a price on access and the other puts up a gate. The shared robots.txt signal is a sign on the door, and a useful one now that four companies have said they will read it.
For publishers, the practical gain is fewer separate notices to write. The practical limit is that the sign is only as good as the visitor's manners, and two of the four visitors will not be reading it until 2027.