# MMXU: A Multi-Modal and Multi-X-ray Understanding Dataset for Disease Progression

arXiv: https://arxiv.org/abs/2502.11651 · HTML: https://mlsanity.com/paper/2502.11651/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 22 of its 42 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 22 of 22)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration](https://mlsanity.com/paper/2311.04257/) (2023)
- [SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering](https://mlsanity.com/paper/2102.09542/) (2021)
- [Med-Flamingo: a Multimodal Medical Few-shot Learner](https://mlsanity.com/paper/2307.15189/) (2023)
- [ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge](https://mlsanity.com/paper/2303.14070/) (2023)
- [HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs](https://mlsanity.com/paper/2412.18925/) (2024)
- [CogVLM2: Visual Language Models for Image and Video Understanding](https://mlsanity.com/paper/2408.16500/) (2024)
- [HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale](https://mlsanity.com/paper/2406.19280/) (2024)
- [OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM](https://mlsanity.com/paper/2402.09181/) (2024)
- [Chest ImaGenome Dataset for Clinical Reasoning](https://mlsanity.com/paper/2108.00316/) (2021)
- [Weakly Supervised Lesion Localization With Probabilistic-CAM Pooling](https://mlsanity.com/paper/2005.14480/) (2020)
- [MiniGPT-Med: Large Language Model as a General Interface for Radiology Diagnosis](https://mlsanity.com/paper/2407.04106/) (2024)
- [STLLaVA-Med: Self-Training Large Language and Vision Assistant for Medical Question-Answering](https://mlsanity.com/paper/2406.19973/) (2024)
- [D-Rax: Domain-specific Radiologic assistant leveraging multi-modal data and eXpert model predictions](https://mlsanity.com/paper/2407.02604/) (2024)
- [MediFact at MEDIQA-M3G 2024: Medical Question Answering in Dermatology with Multimodal Learning](https://mlsanity.com/paper/2405.01583/) (2024)

## Built on by (most cited first, 9 of 9)

- [MedSG-Bench: A Benchmark for Medical Image Sequences Grounding](https://mlsanity.com/paper/2505.11852/) (2025)
- [CP-Env: Evaluating Large Language Models on Clinical Pathways in a Controllable Hospital Environment](https://mlsanity.com/paper/2512.10206/) (2025)
- [Multi Anatomy X-Ray Foundation Model](https://mlsanity.com/paper/2509.12146/) (2025)
- [MI-CXR: A Benchmark for Longitudinal Reasoning over Multi-Interval Chest X-rays](https://mlsanity.com/paper/2605.15574/) (2026)
- [A Vision-language Framework for Comparative Reasoning in Radiology](https://mlsanity.com/paper/2606.06407/) (2026)
- [FT-ARM: Fine-Tuned Agentic Reflection Multimodal Language Model for Pressure Ulcer Severity Classification with Reasoning](https://mlsanity.com/paper/2510.24980/) (2025)
- [MEDIC-AD: Towards Medical Vision-Language Model's Clinical Intelligence](https://mlsanity.com/paper/2603.27176/) (2026)
- [LoMeVQA: A Comprehensive Benchmark for Longitudinal Medical VQA](https://mlsanity.com/paper/2607.27806/) (2026)
- [Analyzing and Mitigating Cross-Lingual Degradation in Multilingual Medical VQA](https://mlsanity.com/paper/2608.22363/) (2026)

## Abstract (arXiv)

Large vision-language models (LVLMs) have shown great promise in medical applications, particularly in visual question answering (MedVQA) and diagnosis from medical images. However, existing datasets and models often fail to consider critical aspects of medical diagnostics, such as the integration of historical records and the analysis of disease progression over time. In this paper, we introduce MMXU (Multimodal and MultiX-ray Understanding), a novel dataset for MedVQA that focuses on identifying changes in specific regions between two patient visits. Unlike previous datasets that primarily address single-image questions, MMXU enables multi-image questions, incorporating both current and historical patient data. We demonstrate the limitations of current LVLMs in identifying disease progression on MMXU-textittest, even those that perform well on traditional benchmarks. To address this, we propose a MedRecord-Augmented Generation (MAG) approach, incorporating both global and regional historical records. Our experiments show that integrating historical records significantly enhances diagnostic accuracy by at least 20%, bridging the gap between current LVLMs and human expert performance. Additionally, we fine-tune models with MAG on MMXU-textitdev, which demonstrates notable improvements. We hope this work could illuminate the avenue of advancing the use of LVLMs in medical diagnostics by emphasizing the importance of historical context in interpreting medical images. Our dataset is released at github: https://github.com/linjiemu/MMXU.