mlsanityData to 14 Oct 2026

Med-2E3: A 2D-Enhanced 3D Medical Multimodal Large Language Model

arXiv 2411.12783 ↗First posted 19 Nov 2024cs.CVcited by 15

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

3D medical image analysis is essential for modern healthcare, yet traditional task-specific models are inadequate due to limited generalizability across diverse clinical scenarios. Multimodal large language models (MLLMs) offer a promising solution to these challenges. However, existing MLLMs have limitations in fully leveraging the rich, hierarchical information embedded in 3D medical images. Inspired by clinical practice, where radiologists focus on both 3D spatial structure and 2D planar content, we propose Med-2E3, a 3D medical MLLM that integrates a dual 3D-2D encoder architecture. To aggregate 2D features effectively, we design a Text-Guided Inter-Slice (TG-IS) scoring module, which scores the attention of each 2D slice based on slice contents and task instructions. To the best of our knowledge, Med-2E3 is the first MLLM to integrate both 3D and 2D features for 3D medical image analysis. Experiments on large-scale, open-source 3D medical multimodal datasets demonstrate that TG-IS exhibits task-specific attention distribution and significantly outperforms current state-of-the-art models. The code is available at: https://github.com/MSIIP/Med-2E3

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 59 references that are in the map. First posted on arXiv 19 Nov 2024.

Builds on

50 of 59 references.

Closest ancestors · 9

Same direction, newest first

  1. 20243D-CT-GPT: Generating 3D Radiology Reports through Integration of Large Vision-Language Models28 Sep
  2. 2024Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoE26 Sep
  3. 2024MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine6 Aug
  4. 2024Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation2 Jul
  5. 2024M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models31 Mar
  6. 2024Med3DInsight: Enhancing 3D Medical Image Understanding with 2D Multi-Modal Large Language Models8 Mar
  7. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review4 Mar
  8. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
  9. 2023MedBLIP: Bootstrapping Language-Image Pre-training from 3D Medical Images and Texts18 May

Other references · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2023Qwen Technical Reportcited by 3,344
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  11. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  12. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
29 more
  1. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  2. 2019Deep Learning for 3D Point Clouds: A Surveycited by 558
  3. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
  4. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  5. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  6. 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 422
  7. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
  8. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 338
  9. 2023Towards Generalist Biomedical AIcited by 289
  10. 2019Med3D: Transfer Learning for 3D Medical Image Analysiscited by 288
  11. 2024Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomographycited by 199
  12. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 179
  13. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  14. 2019Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysiscited by 159
  15. 2017Surgical Data Science: Enabling Next-Generation Surgerycited by 126
  16. 2022Recovering 3D Human Mesh from Monocular Images: A Surveycited by 103
  17. 2024CT2Rep: Automated Radiology Report Generation for 3D Medical Imagingcited by 96
  18. 2021Medical Visual Question Answering: A Surveycited by 93
  19. 2024Advancing Multimodal Medical Capabilities of Geminicited by 90
  20. 2023SegVol: Universal and Interactive Volumetric Medical Image Segmentationcited by 87
  21. 20203D Deep Learning on Medical Images: A Reviewcited by 76
  22. 2024VoCo: A Simple-yet-Effective Volume Contrastive Learning Framework for 3D Medical Image Analysiscited by 75
  23. 2020A Survey on Deep Learning and Explainability for Automatic Report Generation from Medical Imagescited by 35
  24. 2023Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learningcited by 34
  25. 2024Dia-LLaMA: Towards Large Language Model-driven CT Report Generationcited by 32
  26. 2017Automatic Liver Lesion Segmentation Using A Deep Convolutional Neural Network Methodcited by 19
  27. 2020Knee Injury Detection using MRI with Efficiently-Layered Network (ELNet)cited by 7
  28. 2024TinyLLaVA Factory: A Modularized Codebase for Small-scale Large Multimodal Modelscited by 7
  29. 2021Optimising Knee Injury Detection with Spatial Attention and Validating Localisation Abilitycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 10.

15 papers.

Sort:

By area:

  1. 2026OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysiscited by 10
  2. 2025Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoningcited by 8
  3. 2025Region-Aware Multimodal Large Language Model via SlowFast Tokenization and Pseudo-Mask Guidance for 3D CT Report Generationcited by 6
  4. 2026MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentationcited by 5
  5. 2025MedM-VL: What Makes a Good Medical LVLM?cited by 4
  6. 2026Med3D-R1: Incentivizing Clinical Reasoning in 3D Medical Vision-Language Models for Abnormality Diagnosiscited by 4
  7. 2026Beyond Medical Diagnostics: How Medical Multimodal Large Language Models Think in Spacecited by 3
  8. 2026Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compressioncited by 3
  9. 2026CTest-Metric: A Unified Framework to Assess Clinical Validity of Metrics for CT Report Generationcited by 2
  10. 2026Regulating Anatomy-Aware Rewards via Trajectory-Integral Feedback for Volumetric Computed Tomography Analysiscited by 2
  11. 2025HSENet: Hybrid Spatial Encoding Network for 3D Medical Vision-Language Understandingcited by 1
  12. 2025Data-Efficient Fine-Tuning of Vision-Language Models for Diagnosis of Alzheimer's Diseasecited by 1
3 more
  1. 2026Revisiting LLM Adaptation for 3D CT Report Generation: A Study of Scaling and Diagnostic Priorscited by 1
  2. 2026Towards Enhancing 3D Spatial Reasoning in Medical Multimodal Large Language Modelscited by 0
  3. 2026HounsWorld: A Multimodal World Model for Hidden Patient-State Readout, Reconstruction, and Simulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing 3D Medical Image Understanding with Pretraining Aided by 2D Multimodal Large Language Models
  2. 2025Med3DVLM: An Efficient Vision-Language Model for 3D Medical Image Analysis
  3. 2026Adapting 2D Multi-Modal Large Language Model for 3D CT Image Analysis
  4. 2024Read Like a Radiologist: Efficient Vision-Language Model for 3D Medical Imaging Interpretation
  5. 2026ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding
  6. 2024MG-3D: Multi-Grained Knowledge-Enhanced 3D Medical Vision-Language Pre-training
  7. 2025Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging
  8. 20263DMedAgent: Unified Perception-to-Understanding for 3D Medical Analysis