mlsanityData to 14 Oct 2026

Med3DVLM: An Efficient Vision-Language Model for 3D Medical Image Analysis

arXiv 2503.20047 ↗First posted 25 Mar 2025cs.CV eess.IVcited by 46

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Vision-language models (VLMs) have shown promise in 2D medical image analysis, but extending them to 3D remains challenging due to the high computational demands of volumetric data and the difficulty of aligning 3D spatial features with clinical text. We present Med3DVLM, a 3D VLM designed to address these challenges through three key innovations: (1) DCFormer, an efficient encoder that uses decomposed 3D convolutions to capture fine-grained spatial features at scale; (2) SigLIP, a contrastive learning strategy with pairwise sigmoid loss that improves image-text alignment without relying on large negative batches; and (3) a dual-stream MLP-Mixer projector that fuses low- and high-level image features with text embeddings for richer multi-modal representations. We evaluate our model on the M3D dataset, which includes radiology reports and VQA data for 120,084 3D medical images. Results show that Med3DVLM achieves superior performance across multiple benchmarks. For image-text retrieval, it reaches 61.00% R@1 on 2,000 samples, significantly outperforming the current state-of-the-art M3D model (19.10%). For report generation, it achieves a METEOR score of 36.42% (vs. 14.38%). In open-ended visual question answering (VQA), it scores 36.76% METEOR (vs. 33.58%), and in closed-ended VQA, it achieves 79.95% accuracy (vs. 75.78%). These results highlight Med3DVLM's ability to bridge the gap between 3D imaging and language, enabling scalable, multi-task reasoning across clinical applications. Our code is publicly available at https://github.com/mirthAI/Med3DVLM.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 54 references that are in the map. First posted on arXiv 25 Mar 2025.

Builds on

34 of 54 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025DCFormer: Efficient 3D Vision-Language Modeling with Decomposed Convolutions7 Feb
  2. 2024VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge19 Nov
  3. 2024E3D-GPT: Enhanced 3D Visual Foundation for Medical Vision-Language Model18 Oct
  4. 2024M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models31 Mar
  5. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  9. 2024Qwen2.5 Technical Reportcited by 4,942
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  12. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
17 more
  1. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  2. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  3. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  4. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  5. 2021MetaFormer Is Actually What You Need for Visioncited by 524
  6. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
  7. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 339
  8. 2023Towards Generalist Biomedical AIcited by 289
  9. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  10. 2024Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomographycited by 199
  11. 2023R2GenGPT: Radiology Report Generation with Frozen LLMscited by 114
  12. 2024CT2Rep: Automated Radiology Report Generation for 3D Medical Imagingcited by 96
  13. 2019Overcoming Data Limitation in Medical Visual Question Answeringcited by 71
  14. 2019Automatic Radiology Report Generation based on Multi-view Image Fusion and Medical Concept Enrichmentcited by 60
  15. 2019Addressing Data Bias Problems for Chest X-ray Image Report Generationcited by 14
  16. 2024CAT: Coordinating Anatomical-Textual Prompts for Multi-Organ and Tumor Segmentationcited by 14
  17. 2020Hierarchical Deep Multi-modal Network for Medical Visual Question Answeringcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 31.

46 papers.

Sort:

By area:

  1. 2025Fleming-VL: Towards Universal Medical Visual Reasoning with Multimodal LLMscited by 11
  2. 2025Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generationcited by 7
  3. 2025Region-Aware Multimodal Large Language Model via SlowFast Tokenization and Pseudo-Mask Guidance for 3D CT Report Generationcited by 6
  4. 20253DReasonKnee: Advancing Grounded Reasoning in Medical Vision Language Modelscited by 6
  5. 2025RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answeringcited by 6
  6. 2025ARMs: Adaptive Red-Teaming Agent against Multimodal Models with Plug-and-Play Attackscited by 5
  7. 2025Fusion or Confusion? Multimodal Complexity Is Not All You Needcited by 5
  8. 2026MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentationcited by 5
  9. 2025PETAR: Localized Findings Generation with Mask-Aware Vision-Language Modeling for PET Automated Reportingcited by 4
  10. 2026Med3D-R1: Incentivizing Clinical Reasoning in 3D Medical Vision-Language Models for Abnormality Diagnosiscited by 4
  11. 2026Agentic Large Language Models for Training-Free Neuro-Radiological Image Analysiscited by 4
  12. 2026Lost in Volume: The CT-SpatialVQA Benchmark for Evaluating Semantic-Spatial Understanding of 3D Medical Vision-Language Modelscited by 4
34 more
  1. 2026Jolia: Concept-Level Vision-Language Alignment for 3D CT Contrastive Learningcited by 4
  2. 2025MedReason-R1: Learning to Reason for CT Diagnosis with Reinforcement Learning and Local Zoomcited by 3
  3. 2026TumorChain: Interleaved Multimodal Chain-of-Thought Reasoning for Traceable Clinical Tumor Analysiscited by 3
  4. 2026Beyond the Embedding Bottleneck: Adaptive Retrieval-Augmented 3D CT Report Generationcited by 3
  5. 2026Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compressioncited by 3
  6. 2025Multimodal LLM With Hierarchical Mixture-of-Experts for VQA on 3D Brain MRIcited by 2
  7. 2025Radiologist Copilot: An Agentic Framework Orchestrating Specialized Tools for Reliable Radiology Reportingcited by 2
  8. 2025R-GenIMA: Integrating Neuroimaging and Genetics with Interpretable Multimodal AI for Alzheimer's Disease Progressioncited by 2
  9. 2026CTest-Metric: A Unified Framework to Assess Clinical Validity of Metrics for CT Report Generationcited by 2
  10. 2026SigVLP: Sigmoid Volume-Language Pre-Training for Self-Supervised CT-Volume Adaptive Representation Learningcited by 2
  11. 2026Brain3D: Brain Report Automation via Inflated Vision Transformers in 3Dcited by 2
  12. 2026Regulating Anatomy-Aware Rewards via Trajectory-Integral Feedback for Volumetric Computed Tomography Analysiscited by 2
  13. 2025MedErr-CT: A Visual Question Answering Benchmark for Identifying and Correcting Errors in CT Reportscited by 1
  14. 2025Data-Efficient Fine-Tuning of Vision-Language Models for Diagnosis of Alzheimer's Diseasecited by 1
  15. 2026Adapting 2D Multi-Modal Large Language Model for 3D CT Image Analysiscited by 1
  16. 2026ASAP: Advancing Medical Volumetric Representation Learning with Anatomy-aware Semantically-adaptive Pre-trainingcited by 1
  17. 2026When Do Cheap Probes Predict Expensive Training? Probing 3D-CT Encoders for Text Generationcited by 1
  18. 2026Rad-JEPA 3D: Radiology Joint-Embedding Predictive Model for 3D Computed Tomographycited by 1
  19. 2026QwenVLConnector: A Fast, Unified Medical VLM Chatbot for Fine-Grained Clinical Perception and Text Generationcited by 1
  20. 2025SwinTF3D: A Lightweight Multimodal Fusion Approach for Text-Guided 3D Medical Image Segmentationcited by 0
  21. 2026Medical Image Spatial Grounding with Semantic Samplingcited by 0
  22. 2026YOLO with Kolmogorov-Arnold networks and vision-language foundation models for interpretable object detection with trustworthy multimodal AI in computer vision perceptioncited by 0
  23. 2026Anchoring the Eigengap: Cross-Modal Spectral Stabilization for Sample-Efficient Representation Learningcited by 0
  24. 2026UCSF-PDGM-VQA: Visual Question Answering dataset for brain tumor MRI interpretationcited by 0
  25. 2026Towards Enhancing 3D Spatial Reasoning in Medical Multimodal Large Language Modelscited by 0
  26. 2026CT-DeltaBench: A Benchmark for Longitudinal 3D Medical Imaging Difference Reporting with Vision-Language Modelscited by 0
  27. 2026Synthesizing Post-Acetazolamide Cerebral Blood Flow Maps from Baseline MRI in Moyamoya Using 3D Generative AIcited by 0
  28. 2026Entity-Constrained CBCT Retrieval for Low-Resource Dental Record Completioncited by 0
  29. 2026BrainDiff: Longitudinal Report Generation for Multimodal Brain MRIcited by 0
  30. 2026CoMLP: Cooperatively-Gated MLPs for Fine-Grained Cross-Modal Information Fusion in Medical Image Segmentationcited by 0
  31. 2026A Unified Vision-Language Model for PSMA PET/CT Report Generation, Visual Question Answering, and Lesion Segmentationcited by 0
  32. 2026GateSPINE: Gated Cross-View Fusion for Lumbar Spine MRI Report Generationcited by 0
  33. 2026Knee3DVLM: Dual-Sequence Full-Volume Vision-Language Modeling for Comprehensive Knee MRI Assessmentcited by 0
  34. 2026Prior-Fitted and Frozen Scorers for Multimodal Contrastive Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing 3D Medical Image Understanding with Pretraining Aided by 2D Multimodal Large Language Models
  2. 2024Read Like a Radiologist: Efficient Vision-Language Model for 3D Medical Imaging Interpretation
  3. 2024Med3DInsight: Enhancing 3D Medical Image Understanding with 2D Multi-Modal Large Language Models
  4. 2024Med-2E3: A 2D-Enhanced 3D Medical Multimodal Large Language Model
  5. 2023T3D: Advancing 3D Medical Vision-Language Pre-training by Learning Multi-View Visual Consistency
  6. 2025Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging
  7. 2025MedM-VL: What Makes a Good Medical LVLM?
  8. 2025Comprehensive language-image pre-training for 3D medical image understanding