The most awkward moment in this physics benchmark came when the graders marked the model wrong and the model turned out to be right. In a test of research-level theoretical physics problems, GPT-6-Astra solved 49 of 50 tasks. Several of the answers its human checkers first flagged as errors were correct, and the flaws sat in the reference solutions.
What the test actually was
The researchers wanted to know whether AI agents can handle problems at the level of real research work, not textbook exercises. They assembled 50 unpublished problems from different areas of theoretical physics. Unpublished matters here, because it makes it harder for a model to win by recalling a worked solution it has already seen.
The authors' own difficulty estimate is the interesting part. A world-class expert would typically need close to a full day to solve one of these problems. A specialist from an adjacent field would need considerably longer.
The tasks were designed to test the application of known methods. They were not designed to test whether a model can make a fundamentally new discovery. That distinction is the one the whole result hangs on, and we will come back to it.
The scoreboard
With access to the internet, scientific literature and computational tools, Astra reached 98% accuracy. It spent less than $3 per problem on average. For a task that eats a day of an expert's time, that is a price tag worth noticing.
The other models reported in the study trailed it:
| Model | Problems solved (of 50) |
|---|---|
| GPT-6-Astra | 49 |
| Opus 5.5 | 46 |
| Gemini 3.8 Flash | 34 |
| DeepSeek-V4.1-Flash | 27 |
| GLM 5.3 | 21 |
The report does not spell out whether the rivals' figures come from the same tool-enabled setup as Astra's 98%, so compare them with that caveat in mind. The spread is wide either way, from nearly perfect to fewer than half.
When the grader is the one who is wrong
Here the story gets more entertaining. The authors initially judged a few of Astra's answers to be incorrect. Benchmarks usually work on the assumption that the answer key is the fixed point and the model is the variable.
So they consulted experts. The experts found flaws in the authors' own solutions. Once those were corrected, Astra's calculations held up. The model had been right, and the humans had been confidently, officially wrong.
This is a useful reminder that evaluating a strong model gets harder as the model gets stronger. When the problems take an expert a day, the person checking the answer key is also working at the edge of what a day can buy.
The one problem nobody solved
Astra did fail on a single task, and so did every other model. The problem asked for a differential equation to be reconstructed from a given curve. The authors point to ambiguity in the problem statement as a complicating factor in the search. In other words, it is not obvious that this was a failure of physics reasoning at all. It may partly be a failure of the question.
Does it need the internet?
When the researchers switched off internet access, accuracy changed little. The authors read this as a possible sign that the models have absorbed a large share of the scientific literature and can reproduce the needed formulas without looking anything up.
They are careful about the limits of that reading. This study alone cannot rule out the influence of training data. The problems are unpublished, but that does not settle how close the underlying methods sit to material the models were trained on. An honest summary is that the result is consistent with strong internalised knowledge, and it does not prove it.
What this does not show
The authors are blunt about the ceiling of the finding. Solving problems well does not mean an AI can run a physics research program on its own. Models still do worse at the parts of research that come before and around the calculation:
- Choosing a worthwhile problem. Picking a question that matters is a different skill from answering one.
- Framing the right question. A well-posed question is half of a result, and the benchmark hands it over for free.
- Knowing when to change direction. Recognising that an approach is a dead end is, by the authors' account, harder for models than solving problems that have a checkable answer.
The benchmark rewards exactly the thing that is easy to verify. A problem with a definite answer is the friendliest possible terrain for a machine, and research mostly happens off that terrain.
What to take from it
The fair reading is a narrow one with real weight. On unpublished, day-long theoretical physics problems that apply known methods, the strongest model in the test was right 49 times out of 50, at a cost of a few dollars each. It was also right in places where experienced researchers thought it was wrong.
The unfair reading is that physicists are now optional. The authors themselves do not make that claim, and the test could not support it. A tool that checks your calculations and occasionally wins the argument is useful. It is still not the one deciding which argument is worth having.
One small irony to finish on. A benchmark built to measure whether machines can be trusted with expert-level work ended up needing outside experts to check the experts' own answer key.