From Compound Figures to Composite Understanding: Developing a Multi-Modal LLM from Biomedical Literature with Medical Multiple-Image Benchmarking and Validation
On the map: Multimodal LLMs / vision-language models → Medical vision-language models
Abstract (arXiv)
Multimodal large language models (MLLMs) are increasingly capable in medical imaging, yet most focus on single-image settings. Clinical interpretation often requires integrating evidence across multiple images, such as different modalities, views, or time points. However, large-scale medical multi-image data and training strategies for such reasoning remain limited. We construct PMC-MI, a large-scale resource comprising 234,956 instruction instances derived from biomedical compound figures, with 10,555 multi-subimage instances structured for reinforcement learning and assessed by medical reviewers. We also introduce PMC-MI-Bench, a manually reviewed benchmark separated at the source-article level. We further propose a three-stage training framework, instantiated as M3LLM, combining supervised fine-tuning on multi-image instructions, selection-aware reinforcement learning for question-conditioned visual-evidence selection, and supervised consolidation on a broader instruction mixture. Evaluating M3LLM on PMC-MI-Bench, two public medical benchmarks, longitudinal chest radiographs, and a retrospective dermatology cohort reveals it achieves the highest performance among representative MLLMs. On PMC-MI-Bench, M3LLM scored 81.6, 84.6, and 85.2 on single-subimage, multi-subimage, and relative-position tasks, respectively, outperforming the strongest baselines (75.6, 79.1, and 77.6). It also set new records on the provenance-screened OmniMedVQA (88.6%) and MMMU-Med (68.0%), alongside top performance in two clinical settings after task-specific adaptation. These findings support using literature-derived multi-image supervision and targeted post-training for medical multi-image MLLMs. Data and code are publicly available.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 12 of its 160 references that are in the map. First posted on arXiv 27 Nov 2025.
Builds on
12 of 160 references.
Closest ancestors · 2
Same direction, newest first
- 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning8 Jun
- 2025Multimodal Large Language Models for Medicine: A Comprehensive Survey29 Apr
Other references · 10
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
- 2022Quantifying Memorization Across Neural Language Modelscited by 755
- 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
- 2025MedGemma Technical Reportcited by 398
- 2023ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoderscited by 49
- 2025Rethinking Retrieval-Augmented Generation for Medicine: A Large-Scale, Systematic Expert Evaluation and Practical Insightscited by 6
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025An Explainable Biomedical Foundation Model via Large-Scale Concept-Enhanced Vision-Language Pre-training
- 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review
- 2025Scaling Large Vision-Language Models for Enhanced Multimodal Comprehension In Biomedical Image Analysis
- 2025Fleming-VL: Towards Universal Medical Visual Reasoning with Multimodal LLMs
- 2025Open-PMC-18M: A High-Fidelity Large Scale Medical Dataset for Multimodal Representation Learning
- 2024M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models
- 2026MEDLAYXPLAIN: Benchmarking the Expert-Lay Gap in Medical Vision-Language Models
- 2025Medical Large Vision Language Models with Multi-Image Visual Ability