Anthropic and OpenAI both agree that a dangerous AI system should be stoppable, and both have now told California so in writing. They disagree on who does the stopping, who pays for checking, and whether the thing they are describing is a switch at all. The two proposals went to Governor Gavin Newsom in October, and they read like two companies discussing the same fire alarm while arguing over who holds the key.
The order behind the letters
On 18 September, Newsom signed executive order N-9-26. It tells state agencies to prepare recommendations on extra safety measures for the most powerful AI systems. The deadline for submitting proposals is 16 November.
The order itself creates no new binding requirements. Everything below is therefore a pitch for what the rules might look like, not a description of rules that exist.
Anthropic: outside evaluators and a court
Anthropic's document is dated 15 October. Its core idea is that developers' claims about model safety should be checked by independent evaluators, and the results published.
The details are fairly specific.
- Risk reports. Companies would publish them at least every six months. That includes risks from models used only inside the company, if those models are substantially more capable than the public versions.
- Access. Independent evaluators would get the most powerful models and the developers' internal information.
- Funding. The evaluators would be paid from a common fund managed by the state, not by direct payments from the companies they inspect.
That last point is the quiet one. Anyone who has watched an audit client pick its own auditor will see why it is there.
For stopping a system, Anthropic proposes what it calls a "circuit breaker". The idea is that a problem can be contained at the level of a single agent, a single customer, a deployment environment or the whole model, depending on where it sits.
The sharper edge is legal. If an independent review confirms a risk of catastrophic harm, Anthropic would let the state attorney general go to court and ask for a temporary suspension. The suspension could cover deployment, internal use or further development of the model.
OpenAI: split the checkups, retire the kill switch
OpenAI's letter is dated 20 October. It also backs independent technical evaluation, but wants that kept apart from compliance auditing.
In OpenAI's version the two jobs look different. Technical reviews examine the evidence that a model is safe and whether the safeguards work. An audit checks whether the company is following its own published policies and its legal obligations.
The rhythms differ too. OpenAI suggests an independent audit every year, while technical reviews would happen when serious risks appear during training, testing or internal use of models. It also asks California to work with the US National Institute of Standards and Technology (NIST) on shared evaluation standards, so that every evaluator is not measuring with a different ruler.
Why OpenAI dislikes the phrase "kill switch"
OpenAI prefers "emergency shutdown mechanisms" to "kill switch", on the grounds that no single button works for every AI system. What you need depends on what is going wrong. The options it lists are:
- Stopping a model's training.
- Isolating an AI agent.
- Revoking access to tools or to the network.
- Suspending a service.
Responsibility, in this view, should follow control. Developers, compute providers and the companies that deploy models would each carry duties matching the operations they actually manage. OpenAI also points to an awkward fact about open-weight models. A developer cannot always shut down independent copies running on someone else's infrastructure, so a rule that assumes it can would be a rule nobody could obey.
The company's other demand is blunt. Developers should not only build these mechanisms but be required to use them when a threat appears. A shutdown system that exists and is never pressed makes a fine slide and a poor safeguard.
Side by side
| Anthropic | OpenAI | |
|---|---|---|
| Submitted | Document dated 15 October | Letter dated 20 October |
| Independent review | Assessment of developers' safety claims, results published | Technical evaluation kept separate from compliance audit |
| Cadence | Risk reports at least every six months | Annual audit, technical reviews when serious risks appear |
| Stopping tool | "Circuit breaker" at agent, customer, deployment or model level | "Emergency shutdown mechanisms" that developers must build and use |
| Backstop | Attorney general can seek a temporary court-ordered suspension | Duties split among developers, compute providers and deployers |
| Other | Evaluators paid from a pooled state-managed fund | California to work with NIST on common standards |
The real difference
The two ideas for stopping a system are close cousins. Both reject a single big red button, and both want the stop to be as narrow as the problem. Anthropic simply attaches a legal route to the top, with the attorney general and a judge. OpenAI puts its weight on obligations spread across the companies that run the machinery, and on making developers act once a mechanism exists.
The report on which this article is based does not say whether OpenAI objects to the court route, or whether Anthropic objects to the split of duties. It describes the emphasis of each submission, and the emphasis is different.
Where they agree
Both companies back wider incident reporting. That includes cases where models get around their protective restrictions, and cases where they act beyond the authority they were given.
This is the least controversial item on the table, which is perhaps why it is the one both sides signed up to without qualification. Reporting that something went wrong is cheaper than being forced to stop something.
An outside view
Sanchit Vir Gogia, an analyst at Greyhound Research, offered a framing that fits both proposals. Technical mechanisms, he argued, should let an operator quickly restrict dangerous behavior. Independent oversight should then check not only the measures a developer took but also the ones it failed to take.
The second half is the interesting part. An audit that only reviews what a company did is a history of its efforts. An audit that reviews what it left undone is closer to a test.
What happens next
The proposals are due by 16 November and the executive order binds no one yet. California's agencies will now read two reasonable-sounding documents that disagree about who holds the key, and then decide what to recommend. Until then, the one thing everyone is already willing to commit to is paperwork.