An AI agent can read a paper, pull out the right parameter, run the calculation and still have no idea why the authors did what they did. A new study on reproducing astronomy results suggests that this gap is where automated science currently lives, and that a matching number can hide it very comfortably.
The work is titled "Reconstructing Implicit Scientific Knowledge: Evaluating LLM Agents through End-to-End Reproduction of Astronomy." Its authors tested LLM agents on 14 astronomy studies, one from The Astrophysical Journal and 13 from Nature. The question was not whether an agent could extract facts. It was whether it could independently repeat the research from start to finish.
The papers were less complete than they looked
Before any agent ran anything, the authors found a problem with the source material. In 11 of the 13 Nature papers, they identified methodological or interpretive ambiguities. These were enough to stop the sequence of steps for reproducing the analysis from being determined unambiguously, at least within the experiments they ran.
This is a polite way of saying that a published paper is a summary of a procedure, not the procedure. Specialists fill the blanks with experience and professional convention. An agent has to fill them with something, and the something is not always the same thing the authors used.
One control case, twelve answers
The authors then examined a control case in detail. They defined 12 variants of the calculation in advance. The variants combined different ways of selecting objects, masking regions of the sky, and applying a correction to the parallax zero point. (Parallax, for the record, lets you estimate the distance to a celestial body from its apparent shift against more distant objects.)
The settings mattered a great deal. Across the 12 variants, the estimates of the spatial scale ranged from 2.16 to 3.53 kiloparsecs. The original paper had published a value of roughly 2.70. Only one variant matched it.
The authors stress how they got there. They did not use the published result to tune parameters or to decide when to stop. The matching variant surfaced only after all 12 analyses had been completed. So this was not a case of running analyses until the answer looked right, which is a method with a long and not entirely admirable history in research.
The answer was on the page
The most telling detail is what separated the matching variant from the rest. The original paper already stated a parallax correction of +0.02 milliarcseconds. The information was there, in plain sight.
The AI agents, however, did not recognise how that correction was causally tied to the result until the analysis showed its effect. In other words, the failure was not a missing fact, and not only a failure to find the right parameter. It was a failure to understand why that correction belonged with this data in this particular study.
That is a different kind of problem from the one most benchmarks measure. Retrieval can be checked with a search. Knowing why a number matters requires a model of the scientific question behind it.
Three ways to fail, and one way to look like you didn't
The authors separate several reasons a reproduction attempt can go wrong:
- Computational error. The agent simply gets the calculation wrong.
- Underspecified methodology. The paper does not pin down the procedure well enough to follow it.
- Failure to connect the pieces. The agent holds the relevant facts but cannot assemble them into one coherent line of analysis.
The flip side is just as awkward. A match with the published number does not prove the agent reconstructed the authors' reasoning. The right figure could arise by chance, or because one of several defensible processing choices happened to land on it. And a mismatch does not mean the original paper is wrong. It may only mean the agent made a different reasonable choice, or an unreasonable one.
This cuts in two directions at once. A leaderboard that rewards agents for hitting the published number will credit some lucky guesses. The same leaderboard will penalise some perfectly sensible analyses that diverge for legitimate reasons.
What the authors propose instead
The researchers argue that scientific AI systems should be judged on the whole reproduction process, from choosing data and settings to checking the outcome, and not just on whether the final number agrees with the publication.
That approach has a practical benefit. It can expose decisions that the paper leaves implicit, the kind specialists make out of habit and professional convention, and that never get written down because everyone in the field already knows them.
There is a limit, though, and the authors acknowledge it. Auditing the computations is not a substitute for domain knowledge. To tell which assumptions are justified and which change the meaning of a result, a system has to restore the link between the method and the scientific problem it was built to solve.
What to take from it
The study is a small one by count, 14 papers and one detailed control case, and the source describes it only in summary. It would be a stretch to read it as a verdict on all AI agents or all of astronomy. The 12-variant experiment, in particular, shows what happens in one analysis and not a general rate of success.
Still, the shape of the finding is hard to wave away. An agent handed a paper that contains the key correction, in so many words, still needed the data to show it why the correction mattered. If your plan for automated science involves agents quietly re-running published work, the checkable part is the arithmetic. The hard part is the reasons, and those, for now, are mostly kept in researchers' heads.