mlsanityData to 14 Oct 2026

M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models

arXiv 2404.00578 ↗First posted 31 Mar 2024cs.CVcited by 162

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Medical image analysis is essential to clinical diagnosis and treatment, which is increasingly supported by multi-modal large language models (MLLMs). However, previous research has primarily focused on 2D medical images, leaving 3D images under-explored, despite their richer spatial information. This paper aims to advance 3D medical image analysis with MLLMs. To this end, we present a large-scale 3D multi-modal medical dataset, M3D-Data, comprising 120K image-text pairs and 662K instruction-response pairs specifically tailored for various 3D medical tasks, such as image-text retrieval, report generation, visual question answering, positioning, and segmentation. Additionally, we propose M3D-LaMed, a versatile multi-modal large language model for 3D medical image analysis. Furthermore, we introduce a new 3D multi-modal medical benchmark, M3D-Bench, which facilitates automatic evaluation across eight tasks. Through comprehensive evaluation, our method proves to be a robust model for 3D medical image analysis, outperforming existing solutions. All code, data, and models are publicly available at: https://github.com/BAAI-DCAI/M3D.

Key facts. Cited by 162 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 68 references that are in the map. First posted on arXiv 31 Mar 2024.

Builds on

36 of 68 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2023Qwen Technical Reportcited by 3,344
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  12. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
23 more
  1. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  2. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  3. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  4. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  5. 2022TotalSegmentator: robust segmentation of 104 anatomical structures in CT imagescited by 613
  6. 2019The Liver Tumor Segmentation Benchmark (LiTS)cited by 509
  7. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
  8. 2019A large annotated medical image dataset for the development and evaluation of segmentation algorithmscited by 373
  9. 2022AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentationcited by 351
  10. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 338
  11. 2020CHAOS Challenge -- Combined (CT-MR) Healthy Abdominal Organ Segmentationcited by 309
  12. 2023Towards Generalist Biomedical AIcited by 289
  13. 2020AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?cited by 224
  14. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  15. 2016Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challengecited by 201
  16. 2019The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 Challengecited by 190
  17. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 179
  18. 2015DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentationcited by 159
  19. 2020VerSe: A Vertebrae Labelling and Segmentation Benchmark for Multi-detector CT Imagescited by 111
  20. 2023The KiTS21 Challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase CTcited by 103
  21. 2023Unleashing the Strengths of Unlabeled Data in Pan-cancer Abdominal Organ Quantification: the FLARE22 Challengecited by 50
  22. 2021A Computed Tomography Vertebral Segmentation Dataset with Anatomical Variations and Multi-Vendor Scanner Datacited by 28
  23. 2018CT organ segmentation using GPU data augmentation, unsupervised labels and IOU losscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 19; 2025: 62; 2026: 81.

162 papers.

Sort:

By area:

  1. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 82
  2. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  3. 2025Med3DVLM: An Efficient Vision-Language Model for 3D Medical Image Analysiscited by 46
  4. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  5. 2025How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLMcited by 42
  6. 2025RadGPT: Constructing 3D Image-Text Tumor Datasetscited by 36
  7. 2024Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?cited by 31
  8. 2024Large Language Model with Region-guided Referring and Grounding for CT Report Generationcited by 28
  9. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  10. 2024Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attackscited by 26
  11. 20253D-RAD: A Comprehensive 3D Radiology Med-VQA Dataset with Multi-Temporal Analysis and Diverse Diagnostic Taskscited by 26
  12. 2025MedSAM3: Delving into Segment Anything with Medical Conceptscited by 26
38 more
  1. 2024Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Taskscited by 24
  2. 2025MedGround-R1: Advancing Medical Image Grounding via Spatial-Semantic Rewarded Group Relative Policy Optimizationcited by 23
  3. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Modelscited by 22
  4. 2025CT-Agent: A Multimodal-LLM Agent for 3D CT Radiology Question Answeringcited by 21
  5. 2024Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Surveycited by 20
  6. 2025A co-evolving agentic AI system for medical imaging analysiscited by 17
  7. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  8. 2024Med-2E3: A 2D-Enhanced 3D Medical Multimodal Large Language Modelcited by 15
  9. 2025Are Vision Language Models Ready for Clinical Diagnosis? A 3D Medical Benchmark for Tumor-centric Visual Question Answeringcited by 14
  10. 2025Towards Scalable Language-Image Pre-training for 3D Medical Imagingcited by 14
  11. 2025UMIT: Unifying Medical Imaging Tasks via Vision-Language Modelscited by 12
  12. 2024FMBench: Benchmarking Fairness in Multimodal Large Language Models on Medical Taskscited by 11
  13. 2025SegAnyPET: Universal Promptable Segmentation from Positron Emission Tomography Imagescited by 11
  14. 2025Fleming-VL: Towards Universal Medical Visual Reasoning with Multimodal LLMscited by 11
  15. 2024Medical MLLM is Vulnerable: Cross-Modality Jailbreak and Mismatched Attacks on Medical Multimodal Large Language Modelscited by 10
  16. 2024CT-AGRG: Automated Abnormality-Guided Report Generation from 3D Chest CT Volumescited by 10
  17. 2025UniBiomed: A Universal Foundation Model for Grounded Biomedical Image Interpretationcited by 10
  18. 2025Ophiuchus: Incentivizing Tool-augmented "Think with Images" for Joint Medical Segmentation, Understanding and Reasoningcited by 10
  19. 2026IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentationcited by 10
  20. 2026OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysiscited by 10
  21. 2024MG-3D: Multi-Grained Knowledge-Enhanced 3D Medical Vision-Language Pre-trainingcited by 8
  22. 2025Medical Large Vision Language Models with Multi-Image Visual Abilitycited by 8
  23. 2025Foundation Models in Medical Imaging: A Review and Outlookcited by 8
  24. 2025CT-GRAPH: Hierarchical Graph Attention Network for Anatomy-Guided CT Report Generationcited by 8
  25. 2025Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoningcited by 8
  26. 2025OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Taskscited by 8
  27. 2024MAIRA-Seg: Enhancing Radiology Report Generation with Segmentation-Aware Multimodal Large Language Modelscited by 7
  28. 2025Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generationcited by 7
  29. 2024ImageRAG: Enhancing Ultra High Resolution Remote Sensing Imagery Analysis with ImageRAGcited by 6
  30. 2025Brain-Adapter: Enhancing Neurological Disorder Analysis with Adapter-Tuning Multimodal Large Language Modelscited by 6
  31. 2025MEXA: Towards General Multimodal Reasoning with Dynamic Multi-Expert Aggregationcited by 6
  32. 2025Region-Aware Multimodal Large Language Model via SlowFast Tokenization and Pseudo-Mask Guidance for 3D CT Report Generationcited by 6
  33. 20253DReasonKnee: Advancing Grounded Reasoning in Medical Vision Language Modelscited by 6
  34. 2025RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answeringcited by 6
  35. 2026Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Modelscited by 6
  36. 2026RadAgent: A tool-using AI agent for stepwise interpretation of chest computed tomographycited by 6
  37. 2025From large language models to multimodal AI: A scoping review on the potential of generative AI in medicinecited by 5
  38. 2025VCRBench: Exploring Long-form Causal Reasoning Capabilities of Large Video Language Modelscited by 5

50 most cited of 162 All 162 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Med3DInsight: Enhancing 3D Medical Image Understanding with 2D Multi-Modal Large Language Models
  2. 2025MedVision: Benchmarking Quantitative Medical Image Analysis
  3. 2025From Compound Figures to Composite Understanding: Developing a Multi-Modal LLM from Biomedical Literature with Medical Multiple-Image Benchmarking and Validation
  4. 2024E3D-GPT: Enhanced 3D Visual Foundation for Medical Vision-Language Model
  5. 2026MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentation
  6. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM
  7. 2026MMRareBench: A Rare-Disease Multimodal and Multi-Image Medical Benchmark
  8. 2024MultiViewDx: Evidence-Linked Multi-View Clinical Diagnosis