3D Modality-Aware Pre-training for Vision-Language Model in MRI Multi-organ Abnormality Detection
On the map: Multimodal LLMs / vision-language models → Medical vision-language models
Abstract (arXiv)
Vision-language models (VLMs) show strong potential for complex diagnostic tasks in medical imaging. However, applying VLMs to multi-organ medical imaging introduces two principal challenges: (1) modality-specific vision-language alignment and (2) cross-modal feature fusion. In this work, we propose MedMAP, a Medical Modality-Aware Pretraining framework that enhances vision-language representation learning in 3D MRI. MedMAP comprises a modality-aware vision-language alignment stage and a fine-tuning stage for multi-organ abnormality detection. During the pre-training stage, the modality-aware encoders implicitly capture the joint modality distribution and improve alignment between visual and textual representations. We then fine-tune the pre-trained vision encoders (while keeping the text encoder frozen) for downstream tasks. To this end, we curated MedMoM-MRI3D, comprising 7,392 3D MRI volume-report pairs spanning twelve MRI modalities and nine abnormalities tailored for various 3D medical analysis tasks. Extensive experiments on MedMoM-MRI3D demonstrate that MedMAP significantly outperforms existing VLMs in 3D MRI-based multi-organ abnormality detection. Our code is available at https://github.com/RomantiDr/MedMAP.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 13 of its 16 references that are in the map. First posted on arXiv 27 Feb 2026.
Builds on
13 of 16 references.
Closest ancestors · 1
Same direction, newest first
Other references · 12
- 2024GPT-4o System Cardcited by 4,467
- 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
- 2022Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Imagescited by 823
- 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
- 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
- 2024Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomographycited by 192
- 2021Modality-aware Mutual Learning for Multi-modal Medical Image Segmentationcited by 47
- 2024Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Modelscited by 24
- 2024AutoRG-Brain: Grounded Report Generation for Brain MRIcited by 23
- 2024SDR-Former: A Siamese Dual-Resolution Transformer for Liver Lesion Classification Using 3D Multi-Phase Imagingcited by 17
- 2024Towards a Comprehensive, Efficient and Promptable Anatomic Structure Segmentation Model using 3D Whole-body CT Scanscited by 10
- 2024Incorporating Clinical Guidelines through Adapting Multi-modal Large Language Model for Prostate Cancer PI-RADS Scoringcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Med3DVLM: An Efficient Vision-Language Model for 3D Medical Image Analysis
- 2026Mr3D-VL: A generalist vision language foundation model for Multiparametric 3D Magnetic Resonance Imaging
- 2025Enhancing 3D Medical Image Understanding with Pretraining Aided by 2D Multimodal Large Language Models
- 2025MedMoE: Modality-Specialized Mixture of Experts for Medical Vision-Language Understanding
- 2025Data-Efficient Fine-Tuning of Vision-Language Models for Diagnosis of Alzheimer's Disease
- 2024Medical Vision-Language Pre-Training for Brain Abnormalities
- 2025MedVista3D: Vision-Language Modeling for Reducing Diagnostic Errors in 3D CT Disease Detection, Understanding and Reporting
- 2024Med3DInsight: Enhancing 3D Medical Image Understanding with 2D Multi-Modal Large Language Models