A medical AI heat map can look like an explanation while mostly pointing to where disease usually appears. An audit of four systems that analyze chest X-rays found their attention maps often did no better than simple templates based on average anatomical patterns.
What the audit measured
Researchers at the International Institute of Information Technology Hyderabad compared heat maps from MAIRA-2, MedGemma-4B, LLaVA-Med-1.5, and LLaVA-1.5 with radiologists’ annotations. The evaluation covered three public datasets and an additional group of cases involving pneumothorax.
The comparison focused on localization: whether the areas marked by a model aligned with the regions identified by radiologists. It did not establish whether a model’s diagnosis was correct.
The ranking: MAIRA-2 led the automated comparison, followed by MedGemma. The two LLaVA models came behind them.
The average-case problem
On typical cases, the models’ attention maps were no more accurate than simple anatomical templates built from averaged data. That suggests a system may mark a region because pathology commonly appears there, rather than because it has located the finding in that particular X-ray.
That distinction matters. A heat map can appear specific while reflecting a general pattern about where disease tends to occur. Visual precision, it turns out, is not the same thing as image-specific evidence.
Atypical cases: MAIRA-2 and MedGemma showed more genuinely image-linked localization when a finding differed noticeably from the average pattern. The audit therefore points to a weakness that is easiest to miss in routine cases, where common patterns can make a generic map look plausible.
Why human review changes the score
A parallel experiment involving radiologists and 124 cases found that experts did not judge heat maps in the same way as the automated metric. A broader map that made clinical sense could seem more convincing to a radiologist than a tight match to the metric’s statistical boundaries.
That does not make either kind of assessment sufficient on its own. A numerical score offers consistency, but it may miss aspects of clinical plausibility. Human review can add context, though the audit does not treat expert judgment as a substitute for measurement.
The researchers recommend using evaluation methods that account for anatomical prior patterns and involving radiologists in the review. For medical AI, a colorful overlay is not proof that a model found the right thing. It is a claim that still needs checking.