mlsanityData to 7 Oct 2026

MedBLIP: Bootstrapping Language-Image Pre-training from 3D Medical Images and Texts

arXiv 2305.10799 ↗First posted 18 May 2023cs.CVcited by 51

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

also in 1 more
Abstract (arXiv)

Vision-language pre-training (VLP) models have been demonstrated to be effective in many computer vision applications. In this paper, we consider developing a VLP model in the medical domain for making computer-aided diagnoses (CAD) based on image scans and text descriptions in electronic health records, as done in practice. To achieve our goal, we present a lightweight CAD system MedBLIP, a new paradigm for bootstrapping VLP from off-the-shelf frozen pre-trained image encoders and frozen large language models. We design a MedQFormer module to bridge the gap between 3D medical images and 2D pre-trained image encoders and language models as well. To evaluate the effectiveness of our MedBLIP, we collect more than 30,000 image volumes from five public Alzheimer's disease (AD) datasets, i.e., ADNI, NACC, OASIS, AIBL, and MIRIAD. On this largest AD dataset we know, our model achieves the SOTA performance on the zero-shot classification of healthy, mild cognitive impairment (MCI), and AD subjects, and shows its capability of making medical visual question answering (VQA). The code and pre-trained models is available online: https://github.com/Qybc/MedBLIP.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 41 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

28 of 41 references.

References · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
16 more
  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  2. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  4. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  5. 2021Scaling Vision Transformerscited by 993
  6. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  7. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  8. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  9. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  10. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  11. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  12. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  13. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 210
  14. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  15. 2023ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Modelscited by 116
  16. 2023Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Modelscited by 58

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 12; 2025: 24; 2026: 9.

51 papers.

Sort:

By area:

  1. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 176
  2. 2023Foundational Models in Medical Imaging: A Comprehensive Survey and Future Visioncited by 74
  3. 2023Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcarecited by 62
  4. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  5. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  6. 2024CadVLM: Bridging Language and Vision in the Generation of Parametric CAD Sketchescited by 28
  7. 2024Towards a Personal Health Large Language Modelcited by 25
  8. 2023Medical Vision Language Pretraining: A surveycited by 24
  9. 2024Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Surveycited by 20
  10. 2024Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQAcited by 19
  11. 2024Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directionscited by 17
  12. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
38 more
  1. 2024Med-2E3: A 2D-Enhanced 3D Medical Multimodal Large Language Modelcited by 15
  2. 2023T3D: Advancing 3D Medical Vision-Language Pre-training by Learning Multi-View Visual Consistencycited by 14
  3. 2025Localizing Before Answering: A Hallucination Evaluation Benchmark for Grounded Medical Multimodal LLMscited by 14
  4. 2025Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoningcited by 8
  5. 2025Foundation Models in Medical Imaging: A Review and Outlookcited by 7
  6. 2024VoxelPrompt: A Vision Agent for End-to-End Medical Image Analysiscited by 6
  7. 2026Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMscited by 6
  8. 2024Med3DInsight: Enhancing 3D Medical Image Understanding with 2D Multi-Modal Large Language Modelscited by 5
  9. 2024Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answeringcited by 5
  10. 2025SilVar-Med: A Speech-Driven Visual Language Model for Explainable Abnormality Detection in Medical Imagingcited by 5
  11. 2023A Survey on Image-text Multimodal Modelscited by 4
  12. 2025A Narrative Review on Large AI Models in Lung Cancer Screening, Diagnosis, and Treatment Planningcited by 4
  13. 2025ESTR-CoT: Towards Explainable and Accurate Event Stream based Scene Text Recognition with Chain-of-Thought Reasoningcited by 3
  14. 2025Adaptation of Foundation Models for Medical Image Analysis: Strategies, Challenges, and Future Directionscited by 3
  15. 2024ZALM3: Zero-Shot Enhancement of Vision-Language Alignment via In-Context Information in Multi-Turn Multimodal Medical Dialoguecited by 2
  16. 2025Abn-BLIP: Abnormality-aligned Bootstrapping Language-Image Pre-training for Pulmonary Embolism Diagnosis and Report Generation from CTPAcited by 2
  17. 2025Recent Advances in Medical Image Classificationcited by 2
  18. 2025Multimodal LLM With Hierarchical Mixture-of-Experts for VQA on 3D Brain MRIcited by 2
  19. 2024Medical Vision-Language Pre-Training for Brain Abnormalitiescited by 1
  20. 2025Enhancing Multi-task Learning Capability of Medical Generalist Foundation Model via Image-centric Multi-annotation Datacited by 1
  21. 2025MedBLIP: Fine-tuning BLIP for Medical Image Captioningcited by 1
  22. 2025HSENet: Hybrid Spatial Encoding Network for 3D Medical Vision-Language Understandingcited by 1
  23. 2025Cross-modal Causal Intervention for Alzheimer's Disease Predictioncited by 1
  24. 2025CLARIFY: A Specialist-Generalist Framework for Accurate and Lightweight Dermatological Visual Question Answeringcited by 1
  25. 2025Data-Efficient Fine-Tuning of Vision-Language Models for Diagnosis of Alzheimer's Diseasecited by 1
  26. 2025Enhancing 3D Medical Image Understanding with Pretraining Aided by 2D Multimodal Large Language Modelscited by 1
  27. 2026LLaVA-LE: Large Language-and-Vision Assistant for Lunar Explorationcited by 1
  28. 2026ARC-CT: Anatomy-Routed Contrastive Vision-Language Learning for 3D Chest CTcited by 1
  29. 2025Progressive Local Alignment for Medical Multimodal Pre-trainingcited by 0
  30. 2025Brain Imaging Foundation Models, Are We There Yet? A Systematic Review of Foundation Models for Brain Imaging and Biomedical Researchcited by 0
  31. 2025SurgWound-Bench: A Benchmark for Surgical Wound Diagnosiscited by 0
  32. 2025NeuroRAD-FM: A Foundation Model for Neuro-Oncology with Distributionally Robust Trainingcited by 0
  33. 2025Regional Attention-Enhanced Swin Transformer for Clinically Relevant Medical Image Captioningcited by 0
  34. 2026EMAD: Evidence-Centric Grounded Multimodal Diagnosis for Alzheimer's Diseasecited by 0
  35. 2026PRIMA: Pre-Training with Risk-Integrated Image--Metadata Alignment for Medical Diagnosis with LLM-Based Feature Aggregationcited by 0
  36. 2026Toward Clinically Ready Foundation Models in Medical Image Analysis: Adaptation Mechanisms and Deployment Trade-offscited by 0
  37. 2026BLEG: LLM Functions as Powerful fMRI Graph-Enhancer for Brain Network Analysiscited by 0
  38. 2026TaDA: Calibrated Probe Gating for Task-Domain LoRA Mergingcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Multi-modal Pre-training for Medical Vision-language Understanding and Generation: An Empirical Study with A New Benchmark
  2. 2024Can Medical Vision-Language Pre-training Succeed with Purely Synthetic Data?
  3. 2023Utilizing Synthetic Data for Medical Vision-Language Pre-training: Bypassing the Need for Real Images
  4. 2025MedBLINK: Probing Basic Perception in Multimodal Language Models for Medicine
  5. 2025VELVET-Med: Vision and Efficient Language Pre-training for Volumetric Imaging Tasks in Medicine
  6. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
  7. 2025Comprehensive language-image pre-training for 3D medical image understanding
  8. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review