# OpenAI, Google, Microsoft and Anthropic back one robots.txt signal for opting out of AI training

> Four major AI developers say they will respect a shared robots.txt signal that tells their crawlers not to use a site for training. Anthropic already does, while Google and Microsoft are slated for 2027.
>
> OpenCheese AI · 2026.11.02 · Web page: https://opencheese.dev/blog/openai-google-microsoft-and-anthropic-back-one-robots-txt-signal-for-opting-out

Until now, a publisher who wanted to keep its work out of AI training had to tell each AI company separately, in whatever form that company preferred. OpenAI, Google, Microsoft and Anthropic now say they will support a single signal in robots.txt that reaches all of them at once. The catch is the calendar. One of the four has finished the work, and the rest have plans.

## One signal instead of one per company

The proposal came from Kevin Kelly, a technical specialist at Anthropic who works on standards, and it is going through the IETF. The mechanism is simple in outline. A site owner adds a shared signal to robots.txt, and every participating AI provider reads it as the owner's preference about training.

Robots.txt is an old tool. Websites have long used it to tell search crawlers and other automated systems what they may and may not fetch. What it lacked was a common way to say "not for model training," so owners had to state that separately for each AI company.

All four companies say their AI crawlers will back off from a site when they detect the new signal. The report does not give the exact directive or syntax, so that detail will have to come from the standard itself.

## Who ships when

Support is not simultaneous, and the schedule is worth reading closely.

| Company | Status |
|---|---|
| Anthropic | Support already in place |
| OpenAI | Planned for the coming months |
| Google | Planned for 2027 |
| Microsoft | Planned for 2027 |

So the company that proposed the standard has implemented it, and two of the four others will take until next year. A publisher who adds the signal today is making a request that, for now, only one of the four is certain to read.

## Microsoft adds a second dial

Microsoft says it will offer finer control than the shared signal alone. Publishers will be able to forbid use of their material for training separately from use for grounding. Grounding is when a model pulls verified information from outside sources while building an answer.

The two are different bargains. Training takes content into the model. Grounding fetches it at the moment someone asks a question. A publisher might reasonably object to the first and tolerate the second, or the reverse.

Microsoft also says that sites will keep appearing in Bing search results whatever they choose. Opting out of AI use does not mean disappearing from search.

Google already offers something similar. Its existing controls cover how content is used in AI features of search and in model training, again without costing a site its place in ordinary search results.

## Meta offers a polite maybe

Not every large AI developer has committed. Meta spokesperson Andy Stone backed the idea of a common mechanism and said the company is ready to work with other AI developers on the standard and its rollout. He stopped short of saying Meta will actually obey the bans.

That is a careful sentence. Supporting a shared mechanism and promising to honor it are separate acts, and only the second one keeps a crawler off your pages.

## What the signal does not do

The new signal, by itself, offers no protection against crawlers run by companies that do not support it or do not follow it. Its force comes from the willingness of each company to comply, which is why Meta's answer matters even though Meta is only one name on a longer list.

Other controls are developing alongside it, and they work differently:

- **Really Simple Licensing (RSL)** lets a publisher charge crawlers for access to its material.
- **Cloudflare** blocks AI crawlers by default.

One of these puts a price on access and the other puts up a gate. The shared robots.txt signal is a sign on the door, and a useful one now that four companies have said they will read it.

For publishers, the practical gain is fewer separate notices to write. The practical limit is that the sign is only as good as the visitor's manners, and two of the four visitors will not be reading it until 2027.

## Related posts

- 2026.10.25 · [ChatGPT Ads expands to 26 new countries, with over 100,000 advertisers already on the platform](https://opencheese.dev/blog/chatgpt-ads-expands-26-new-countries-100000-advertisers.md): OpenAI says ChatGPT Ads is growing to 26 more countries and that more than 100,000 advertisers have used it. The pitch: reach people while they explore, compare and decide what to buy.
- 2026.10.16 · [Google and Meta claim frontier models, reopening the AI race](https://opencheese.dev/blog/google-and-meta-claim-frontier-models-reopening-the-ai-race.md): Within days of each other, Google and Meta unveiled models they say can match the best from OpenAI and Anthropic. A race that looked like a two-horse affair is suddenly crowded again.
- 2026.11.02 · [Anthropic and OpenAI give California different plans for stopping dangerous AI](https://opencheese.dev/blog/anthropic-and-openai-give-california-different-plans-for-stopping-dangerous-ai.md): Both companies wrote to Governor Newsom after his September order. Anthropic wants the attorney general able to seek a court-ordered pause, OpenAI wants mandatory emergency shutdown mechanisms that developers must actually use.