mlsanityData to 14 Oct 2026

E3D-GPT: Enhanced 3D Visual Foundation for Medical Vision-Language Model

arXiv 2410.14200 ↗First posted 18 Oct 2024eess.IV cs.CL cs.CVcited by 19

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

The development of 3D medical vision-language models holds significant potential for disease diagnosis and patient treatment. However, compared to 2D medical images, 3D medical images, such as CT scans, face challenges related to limited training data and high dimension, which severely restrict the progress of 3D medical vision-language models. To address these issues, we collect a large amount of unlabeled 3D CT data and utilize self-supervised learning to construct a 3D visual foundation model for extracting 3D visual features. Then, we apply 3D spatial convolutions to aggregate and project high-level image features, reducing computational complexity while preserving spatial information. We also construct two instruction-tuning datasets based on BIMCV-R and CT-RATE to fine-tune the 3D vision-language model. Our model demonstrates superior performance compared to existing methods in report generation, visual question answering, and disease diagnosis. Code and data will be made publicly available soon.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 21 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

16 of 21 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024STLLaVA-Med: Self-Training Large Language and Vision Assistant for Medical Question-Answering28 Jun
  2. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale27 Jun
  3. 2024Argus: Benchmarking and Enhancing Vision-Language Models for 3D Radiology Report Generation11 Jun
  4. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 12

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  7. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  8. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  9. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
  10. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 179
  11. 2022Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Trainingcited by 104
  12. 2024MiniGPT-Med: Large Language Model as a General Interface for Radiology Diagnosiscited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 12.

19 papers.

Sort:

By area:

  1. 2025Med3DVLM: An Efficient Vision-Language Model for 3D Medical Image Analysiscited by 46
  2. 2025Raptor: Scalable Train-Free Embeddings for 3D Medical Volumes Leveraging Pretrained 2D Foundation Modelscited by 13
  3. 2025Unified Supervision For Vision-Language Modeling in 3D Computed Tomographycited by 4
  4. 2026Med3D-R1: Incentivizing Clinical Reasoning in 3D Medical Vision-Language Models for Abnormality Diagnosiscited by 4
  5. 2026Agentic Large Language Models for Training-Free Neuro-Radiological Image Analysiscited by 4
  6. 2026Lost in Volume: The CT-SpatialVQA Benchmark for Evaluating Semantic-Spatial Understanding of 3D Medical Vision-Language Modelscited by 4
  7. 2025Vision-Language Models for Automated 3D PET/CT Report Generationcited by 2
  8. 2026Learning Clinical Representations Under Systematic Distribution Shiftcited by 2
  9. 2026Discriminative Representation Learning for Clinical Predictioncited by 2
  10. 2025OWT: A Foundational Organ-Wise Tokenization Framework for Medical Imagingcited by 1
  11. 2025Zero-Shot Multi-modal Large Language Model v.s. Supervised Deep Learning: A Comparative Study on CT-Based Intracranial Hemorrhage Subtypingcited by 1
  12. 2025HSENet: Hybrid Spatial Encoding Network for 3D Medical Vision-Language Understandingcited by 1
7 more
  1. 2026Revisiting LLM Adaptation for 3D CT Report Generation: A Study of Scaling and Diagnostic Priorscited by 1
  2. 2026Event Fields: Learning Latent Event Structure for Waveform Foundation Modelscited by 0
  3. 2026WISTERIA: Learning Clinical Representations from Noisy Supervision via Multi-View Consistency in Electronic Health Recordscited by 0
  4. 2026AURORA: Contextual Orthogonalization for Geometric Representation Learning in Healthcare Foundation Modelscited by 0
  5. 2026MorphologyFM: A Foundation Model for Morphology-Aware Representation Learning from ECG and Pulse Oximetry Waveformscited by 0
  6. 2026Multi-LLM Collaborative MRI Report Generation for Visual Instruction Tuning in Brain Oncologycited by 0
  7. 2026Volumetric Radiology AI in the Era of Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Comprehensive language-image pre-training for 3D medical image understanding
  2. 2026Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography
  3. 2025Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging
  4. 20243D-CT-GPT: Generating 3D Radiology Reports through Integration of Large Vision-Language Models
  5. 2023T3D: Advancing 3D Medical Vision-Language Pre-training by Learning Multi-View Visual Consistency
  6. 2023Generative Text-Guided 3D Vision-Language Pretraining for Unified Medical Image Segmentation
  7. 2026Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT
  8. 2025Data-Efficient Fine-Tuning of Vision-Language Models for Diagnosis of Alzheimer's Disease