# Comparative Evaluation of Generative AI Models for Chest Radiograph Report Generation in the Emergency Department

arXiv: https://arxiv.org/abs/2512.00271 · HTML: https://mlsanity.com/paper/2512.00271/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 30 references that are in the map.
- First posted on arXiv 29 Nov 2025.

## Builds on (most cited first, 6 of 6)

- [Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning](https://mlsanity.com/paper/2506.07044/) (2025)
- [MAIRA-2: Grounded Radiology Report Generation](https://mlsanity.com/paper/2406.04449/) (2024)
- [Improving Chest X-Ray Report Generation by Leveraging Warm Starting](https://mlsanity.com/paper/2201.09405/) (2022)
- [Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation](https://mlsanity.com/paper/2403.08002/) (2024)
- [MedVersa: A Generalist Foundation Model for Medical Image Interpretation](https://mlsanity.com/paper/2405.07988/) (2024)
- [Beyond Images: An Integrative Multi-modal Approach to Chest X-Ray Report Generation](https://mlsanity.com/paper/2311.11090/) (2023)

## Abstract (arXiv)

Purpose: To benchmark open-source or commercial medical image-specific VLMs against real-world radiologist-written reports. Methods: This retrospective study included adult patients who presented to the emergency department between January 2022 and April 2025 and underwent same-day CXR and CT for febrile or respiratory symptoms. Reports from five VLMs (AIRead, Lingshu, MAIRA-2, MedGemma, and MedVersa) and radiologist-written reports were randomly presented and blindly evaluated by three thoracic radiologists using four criteria: RADPEER, clinical acceptability, hallucination, and language clarity. Comparative performance was assessed using generalized linear mixed models, with radiologist-written reports treated as the reference. Finding-level analyses were also performed with CT as the reference. Results: A total of 478 patients (median age, 67 years [interquartile range, 50-78]; 282 men [59.0%]) were included. AIRead demonstrated the lowest RADPEER 3b rate (5.3% [76/1434] vs. radiologists 13.9% [200/1434]; P<.001), whereas other VLMs showed higher disagreement rates (16.8-43.0%; P<.05). Clinical acceptability was the highest with AIRead (84.5% [1212/1434] vs. radiologists 74.3% [1065/1434]; P<.001), while other VLMs performed worse (41.1-71.4%; P<.05). Hallucinations were rare with AIRead, comparable to radiologists (0.3% [4/1425]) vs. 0.1% [1/1425]; P=.21), but frequent with other models (5.4-17.4%; P<.05). Language clarity was higher with AIRead (82.9% [1189/1434]), Lingshu (88.0% [1262/1434]), and MedVersa (88.4% [1268/1434]) compared with radiologists (78.1% [1120/1434]; P<.05). Sensitivity varied substantially across VLMs for the common findings: AIRead, 15.5-86.7%; Lingshu, 2.4-86.7%; MAIRA-2, 6.0-72.0%; MedGemma, 4.8-76.7%; and MedVersa, 20.2-69.3%. Conclusion: Medical VLMs for CXR report generation exhibited variable performance in report quality and diagnostic measures.