# ReXVQA: A Large-scale Visual Question Answering Benchmark for Generalist Chest X-ray Understanding

arXiv: https://arxiv.org/abs/2506.04353 · HTML: https://mlsanity.com/paper/2506.04353/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 27 references that are in the map.
- First posted on arXiv 4 Jun 2025.

## Builds on (most cited first, 12 of 12)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025)
- [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://mlsanity.com/paper/2412.13663/) (2024)
- [MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiology](https://mlsanity.com/paper/2301.02228/) (2023)
- [Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Images](https://mlsanity.com/paper/2302.14042/) (2023)
- [Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models](https://mlsanity.com/paper/2501.14818/) (2025)
- [EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Images](https://mlsanity.com/paper/2310.18652/) (2023)
- [Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answering](https://mlsanity.com/paper/2307.11986/) (2023)
- [GEMeX: A Large-Scale, Groundable, and Explainable Medical VQA Benchmark for Chest X-ray Diagnosis](https://mlsanity.com/paper/2411.16778/) (2024)
- [Gemini Goes to Med School: Exploring the Capabilities of Multimodal Large Language Models on Medical Challenge Problems & Hallucinations](https://mlsanity.com/paper/2402.07023/) (2024)
- [ReXGradient-160K: A Large-Scale Publicly Available Dataset of Chest Radiographs with Free-text Reports](https://mlsanity.com/paper/2505.00228/) (2025)

## Built on by (most cited first, 20 of 20)

- [MIRAGE: The Illusion of Visual Understanding](https://mlsanity.com/paper/2603.21687/) (2026)
- [OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoning](https://mlsanity.com/paper/2511.23269/) (2025)
- [RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answering](https://mlsanity.com/paper/2512.17396/) (2025)
- [A Reasoning-Enabled Vision-Language Foundation Model for Chest X-ray Interpretation](https://mlsanity.com/paper/2604.00493/) (2026)
- [Picking the Right Specialist: Attentive Neural Process-based Selection of Task-Specialized Models as Tools for Agentic Healthcare Systems](https://mlsanity.com/paper/2602.14901/) (2026)
- [MedRCube: A Multidimensional Framework for Fine-Grained and In-Depth Evaluation of MLLMs in Medical Imaging](https://mlsanity.com/paper/2604.13756/) (2026)
- [Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRI](https://mlsanity.com/paper/2604.15808/) (2026)
- [Verification Mirage: Mapping the Reliability Boundary of Self-Verification in Medical VQA](https://mlsanity.com/paper/2605.10850/) (2026)
- [Vision-language models for chest radiography do not always need the image](https://mlsanity.com/paper/2606.17710/) (2026)
- [A Multi-Agent System for Complex Reasoning in Radiology Visual Question Answering](https://mlsanity.com/paper/2508.02841/) (2025)
- [Which Tool Response Should I Trust? Tool-Expertise-Aware Chest X-ray Agent with Multimodal Agentic Learning](https://mlsanity.com/paper/2602.21517/) (2026)
- [CheXpercept: A Benchmark for Evaluating Expert-Level Lesion Perception in Chest X-rays](https://mlsanity.com/paper/2606.21020/) (2026)
- [Performance Assessment Strategies for Language Model Applications in Healthcare](https://mlsanity.com/paper/2509.08087/) (2025)
- [ReXSonoVQA: A Video QA Benchmark for Procedure-Centric Ultrasound Understanding](https://mlsanity.com/paper/2604.10916/) (2026)
- [CXR-ContraBench: Benchmarking Negated-Option Attraction in Medical VLMs](https://mlsanity.com/paper/2605.05810/) (2026)
- [NeuroQA: A Large-Scale Image-Grounded Benchmark for 3D Brain MRI Understanding](https://mlsanity.com/paper/2605.20525/) (2026)
- [Teach-to-Reason: Competition-Guided Reasoning with a Self-Improving Teacher](https://mlsanity.com/paper/2606.25407/) (2026)
- [CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement](https://mlsanity.com/paper/2608.03890/) (2026)
- [MC-CXR: A Multi-Context Chest X-ray Benchmark for Context-Induced Disruption in Vision-Language Models](https://mlsanity.com/paper/2608.24118/) (2026)
- [Ranking-Aware Prompt Optimization for Multimodal Clinical Diagnosis](https://mlsanity.com/paper/2609.40361/) (2026)

## Abstract (arXiv)

We present ReXVQA, the largest and most comprehensive benchmark for visual question answering (VQA) in chest radiology, comprising approximately 696,000 questions paired with 160,000 chest X-rays studies across training, validation, and test sets. Unlike prior efforts that rely heavily on template based queries, ReXVQA introduces a diverse and clinically authentic task suite reflecting five core radiological reasoning skills: presence assessment, location analysis, negation detection, differential diagnosis, and geometric reasoning. We evaluate eight state-of-the-art multimodal large language models, including MedGemma-4B-it, Qwen2.5-VL, Janus-Pro-7B, and Eagle2-9B. The best-performing model (MedGemma) achieves 83.24% overall accuracy. To bridge the gap between AI performance and clinical expertise, we conducted a comprehensive human reader study involving 3 radiology residents on 200 randomly sampled cases. Our evaluation demonstrates that MedGemma achieved superior performance (83.84% accuracy) compared to human readers (best radiology resident: 77.27%), representing a significant milestone where AI performance exceeds expert human evaluation on chest X-ray interpretation. The reader study reveals distinct performance patterns between AI models and human experts, with strong inter-reader agreement among radiologists while showing more variable agreement patterns between human readers and AI models. ReXVQA establishes a new standard for evaluating generalist radiological AI systems, offering public leaderboards, fine-grained evaluation splits, structured explanations, and category-level breakdowns. This benchmark lays the foundation for next-generation AI systems capable of mimicking expert-level clinical reasoning beyond narrow pathology classification. Our dataset will be open-sourced at https://huggingface.co/datasets/rajpurkarlab/ReXVQA