# Medical Large Vision Language Models with Multi-Image Visual Ability

arXiv: https://arxiv.org/abs/2505.19031 · HTML: https://mlsanity.com/paper/2505.19031/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 25 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 16 of 16)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [Med-Flamingo: a Multimodal Medical Few-shot Learner](https://mlsanity.com/paper/2307.15189/) (2023)
- [PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering](https://mlsanity.com/paper/2305.10415/) (2023)
- [MANTIS: Interleaved Multi-Image Instruction Tuning](https://mlsanity.com/paper/2405.01483/) (2024)
- [Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data](https://mlsanity.com/paper/2308.02463/) (2023)
- [M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models](https://mlsanity.com/paper/2404.00578/) (2024)
- [MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs](https://mlsanity.com/paper/2406.11833/) (2024)
- [Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videos](https://mlsanity.com/paper/2312.04746/) (2023)
- [MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models](https://mlsanity.com/paper/2408.02718/) (2024)
- [Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation](https://mlsanity.com/paper/2410.00263/) (2024)
- [A Benchmark of Ocular Disease Intelligent Recognition: One Shot for Multi-disease Detection](https://mlsanity.com/paper/2102.07978/) (2021)
- [Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-training](https://mlsanity.com/paper/2405.19654/) (2024)

## Built on by (most cited first, 8 of 8)

- [MoMA: A Mixture-of-Multimodal-Agents Architecture for Enhancing Clinical Prediction Modelling](https://mlsanity.com/paper/2508.05492/) (2025)
- [Correcting Visual Blur Induced by Attention Distraction to Reduce Hallucinations: Algorithm and Theory](https://mlsanity.com/paper/2605.24602/) (2026)
- [See-in-Pairs: Reference Image-Guided Comparative Vision-Language Models for Medical Diagnosis](https://mlsanity.com/paper/2506.18140/) (2025)
- [Do All Individual Layers Help? An Empirical Study of Task-Interfering Layers in Vision-Language Models](https://mlsanity.com/paper/2602.01167/) (2026)
- [Location-Aware Pretraining for Medical Difference Visual Question Answering](https://mlsanity.com/paper/2603.04950/) (2026)
- [Robust Pre-Training of Medical Vision-and-Language Models with Domain-Invariant Multi-Modal Masked Reconstruction](https://mlsanity.com/paper/2602.17689/) (2026)
- [MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models](https://mlsanity.com/paper/2602.21950/) (2026)
- [Beyond Attention Imbalance: Mitigating Hallucinations via Spectral Surgery](https://mlsanity.com/paper/2609.37263/) (2026)

## Abstract (arXiv)

Medical large vision-language models (LVLMs) have demonstrated promising performance across various single-image question answering (QA) benchmarks, yet their capability in processing multi-image clinical scenarios remains underexplored. Unlike single image based tasks, medical tasks involving multiple images often demand sophisticated visual understanding capabilities, such as temporal reasoning and cross-modal analysis, which are poorly supported by current medical LVLMs. To bridge this critical gap, we present the Med-MIM instruction dataset, comprising 83.2K medical multi-image QA pairs that span four types of multi-image visual abilities (temporal understanding, reasoning, comparison, co-reference). Using this dataset, we fine-tune Mantis and LLaVA-Med, resulting in two specialized medical VLMs: MIM-LLaVA-Med and Med-Mantis, both optimized for multi-image analysis. Additionally, we develop the Med-MIM benchmark to comprehensively evaluate the medical multi-image understanding capabilities of LVLMs. We assess eight popular LVLMs, including our two models, on the Med-MIM benchmark. Experimental results show that both Med-Mantis and MIM-LLaVA-Med achieve superior performance on the held-in and held-out subsets of the Med-MIM benchmark, demonstrating that the Med-MIM instruction dataset effectively enhances LVLMs' multi-image understanding capabilities in the medical domain.