# MedFrameQA: A Multi-Image Medical VQA Benchmark for Clinical Reasoning

arXiv: https://arxiv.org/abs/2505.16964 · HTML: https://mlsanity.com/paper/2505.16964/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 19 of its 46 references that are in the map.
- First posted on arXiv 22 May 2025.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 19 of 19)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [PathVQA: 30000+ Questions for Medical Visual Question Answering](https://mlsanity.com/paper/2003.10286/) (2020)
- [PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering](https://mlsanity.com/paper/2305.10415/) (2023)
- [Towards Generalist Biomedical AI](https://mlsanity.com/paper/2307.14334/) (2023)
- [Capabilities of Gemini Models in Medicine](https://mlsanity.com/paper/2404.18416/) (2024)
- [MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding](https://mlsanity.com/paper/2501.18362/) (2025)
- [SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models](https://mlsanity.com/paper/2504.11468/) (2025)
- [Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey](https://mlsanity.com/paper/2503.12605/) (2025)
- [MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learning](https://mlsanity.com/paper/2502.19634/) (2025)
- [Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning](https://mlsanity.com/paper/2401.06805/) (2024)
- [OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles](https://mlsanity.com/paper/2503.17352/) (2025)
- [MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphs](https://mlsanity.com/paper/2504.00993/) (2025)
- [m1: Unleash the Potential of Test-Time Scaling for Medical Reasoning with Large Language Models](https://mlsanity.com/paper/2504.00869/) (2025)
- [A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?](https://mlsanity.com/paper/2409.15277/) (2024)
- [Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQA](https://mlsanity.com/paper/2405.20421/) (2024)

## Built on by (most cited first, 30 of 34)

- [Kvasir-VQA-x1: A Multimodal Dataset for Medical Reasoning and Robust MedVQA in Gastrointestinal Endoscopy](https://mlsanity.com/paper/2506.09958/) (2025)
- [Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbations](https://mlsanity.com/paper/2510.11196/) (2025)
- [MedMMV: A Controllable Multimodal Multi-Agent Framework for Reliable and Verifiable Clinical Reasoning](https://mlsanity.com/paper/2509.24314/) (2025)
- [OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks](https://mlsanity.com/paper/2511.00846/) (2025)
- [Bridging the Gap in Ophthalmic AI: MM-Retinal-Reason Dataset and OphthaReason Model toward Dynamic Multimodal Reasoning](https://mlsanity.com/paper/2508.16129/) (2025)
- [Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Models](https://mlsanity.com/paper/2603.25155/) (2026)
- [Medical thinking with multiple images](https://mlsanity.com/paper/2604.16506/) (2026)
- [RadThinking: A Dataset for Longitudinal Clinical Reasoning in Radiology](https://mlsanity.com/paper/2605.10761/) (2026)
- [ClinSeekAgent: Automating Multimodal Evidence Seeking for Agentic Clinical Reasoning](https://mlsanity.com/paper/2605.20176/) (2026)
- [AutoMedBench: Towards Medical AutoResearch with Agentic AI Models](https://mlsanity.com/paper/2606.01961/) (2026)
- [Multimodal Retrieval-Augmented Generation with Large Language Models for Medical VQA](https://mlsanity.com/paper/2510.13856/) (2025)
- [ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding](https://mlsanity.com/paper/2607.24743/) (2026)
- [Architecting Clinical Collaboration: Multi-Agent Reasoning Systems for Multimodal Medical VQA](https://mlsanity.com/paper/2507.05520/) (2025)
- [MedRepBench: A Comprehensive Benchmark for Medical Report Interpretation](https://mlsanity.com/paper/2508.16674/) (2025)
- [TemMed-Bench: Evaluating Temporal Medical Image Reasoning in Vision-Language Models](https://mlsanity.com/paper/2509.25143/) (2025)
- [Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRI](https://mlsanity.com/paper/2604.15808/) (2026)
- [ClinHallu: A Benchmark for Diagnosing Stage-Wise Hallucinations in Medical MLLM Reasoning](https://mlsanity.com/paper/2606.14697/) (2026)
- [MMRareBench: A Rare-Disease Multimodal and Multi-Image Medical Benchmark](https://mlsanity.com/paper/2604.10755/) (2026)
- [Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models](https://mlsanity.com/paper/2606.26079/) (2026)
- [Unified Hallucination Fuzzing for Multimodal Large Language Models](https://mlsanity.com/paper/2608.07525/) (2026)
- [MultiMedEdit: A Scenario-Aware Benchmark for Evaluating Knowledge Editing in Medical VQA](https://mlsanity.com/paper/2508.07022/) (2025)
- [Lightweight Structured Multimodal Reasoning for Clinical Scene Understanding in Robotics](https://mlsanity.com/paper/2509.22014/) (2025)
- [MedGEN-Bench: A Contextually Entangled Benchmark for Open-ended Multimodal Medical Generation](https://mlsanity.com/paper/2511.13135/) (2025)
- [MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models](https://mlsanity.com/paper/2602.21950/) (2026)
- [MedGPT-oss: Training a General-Purpose Vision-Language Model for Biomedicine](https://mlsanity.com/paper/2603.00842/) (2026)
- [A Comparative Study in Surgical AI: Potential and Limitations of Data, Compute, and Scaling](https://mlsanity.com/paper/2603.27341/) (2026)
- [SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmark](https://mlsanity.com/paper/2604.20319/) (2026)
- [X-PCR: A Benchmark for Cross-modality Progressive Clinical Reasoning in Ophthalmic Diagnosis](https://mlsanity.com/paper/2604.20350/) (2026)
- [LiteMedCoT-VL: Parameter-Efficient Adaptation for Medical Visual Question Answering](https://mlsanity.com/paper/2605.09384/) (2026)
- [Inference-Time Agentic Decision Rules Beat Longer Evolving Search for Multi-Image Medical Reasoning](https://mlsanity.com/paper/2607.27564/) (2026)

## Abstract (arXiv)

Real-world clinical practice demands multi-image comparative reasoning, yet current medical benchmarks remain limited to single-frame interpretation. We present MedFrameQA, the first benchmark explicitly designed to test multi-image medical VQA through educationally-validated diagnostic sequences. To construct this dataset, we develop a scalable pipeline that leverages narrative transcripts from medical education videos to align visual frames with textual concepts, automatically producing 2,851 high-quality multi-image VQA pairs with explicit, transcript-grounded reasoning chains. Our evaluation of 11 advanced MLLMs (including reasoning models) exposes severe deficiencies in multi-image synthesis, where accuracies mostly fall below 50% and exhibit instability across varying image counts. Error analysis demonstrates that models often treat images as isolated instances, failing to track pathological progression or cross-reference anatomical shifts. MedFrameQA provides a rigorous standard for evaluating the next generation of MLLMs in handling complex, temporally grounded medical narratives.