mlsanityData to 14 Oct 2026

Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training

arXiv 2105.11333 ↗First posted 24 May 2021cs.CVcited by 96

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Recently a number of studies demonstrated impressive performance on diverse vision-language multi-modal tasks such as image captioning and visual question answering by extending the BERT architecture with multi-modal pre-training objectives. In this work we explore a broad set of multi-modal representation learning tasks in the medical domain, specifically using radiology images and the unstructured report. We propose Medical Vision Language Learner (MedViLL), which adopts a BERT-based architecture combined with a novel multi-modal attention masking scheme to maximize generalization performance for both vision-language understanding tasks (diagnosis classification, medical image-report retrieval, medical visual question answering) and vision-language generation task (radiology report generation). By statistically and rigorously evaluating the proposed model on four downstream tasks with three radiographic image-report datasets (MIMIC-CXR, Open-I, and VQA-RAD), we empirically demonstrate the superior downstream task performance of MedViLL against various baselines, including task-specific architectures. The source code is publicly available at: https://github.com/SuperSupermoon/MedViLL

Key facts. Cited by 96 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 59 references that are in the map. First posted on arXiv 24 May 2021.

Builds on

37 of 59 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2016Densely Connected Convolutional Networkscited by 6,720
  7. 2015VQA: Visual Question Answeringcited by 2,506
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  9. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  11. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  12. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
25 more
  1. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  2. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  3. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  4. 2017A Survey on Deep Learning in Medical Image Analysiscited by 1,186
  5. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  6. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  7. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  8. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  9. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  10. 2019Transfusion: Understanding Transfer Learning for Medical Imagingcited by 368
  11. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 337
  12. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  13. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 228
  14. 2018Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysiscited by 179
  15. 2018TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rayscited by 139
  16. 2019Clinically Accurate Chest X-Ray Report Generationcited by 139
  17. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 137
  18. 2019Overcoming Data Limitation in Medical Visual Question Answeringcited by 71
  19. 2020Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTscited by 71
  20. 2020X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformerscited by 59
  21. 2020XGPT: Cross-modal Generative Pre-Training for Image Captioningcited by 51
  22. 2020A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reportscited by 35
  23. 2018Unsupervised Multimodal Representation Learning across Medical Images and Reportscited by 15
  24. 2020A Multimodal Memes Classification: A Survey and Open Research Issuescited by 14
  25. 2020EMIXER: End-to-end Multimodal X-ray Generation via Self-supervisioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 8; 2023: 32; 2024: 31; 2025: 20; 2026: 4.

96 papers.

Sort:

By area:

  1. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 211
  2. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
  3. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 125
  4. 2022Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Trainingcited by 104
  5. 2024Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directionscited by 97
  6. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  7. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  8. 2023Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-traincited by 75
  9. 2022Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Studycited by 66
  10. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  11. 2023ECG-QA: A Comprehensive Question Answering Dataset Combined With Electrocardiogramcited by 54
  12. 2023EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Imagescited by 53
38 more
  1. 2022Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledgecited by 51
  2. 2024Automated Radiology Report Generation: A Review of Recent Advancescited by 43
  3. 2023CLIP in Medical Imaging: A Surveycited by 40
  4. 2023Towards Unifying Medical Vision-and-Language Pre-training via Soft Promptscited by 36
  5. 2023Utilizing Longitudinal Chest X-Rays and Reports to Pre-Fill Radiology Reportscited by 35
  6. 2023eP-ALM: Efficient Perceptual Augmentation of Language Modelscited by 34
  7. 2023RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-trainingcited by 31
  8. 2024GEMeX: A Large-Scale, Groundable, and Explainable Medical VQA Benchmark for Chest X-ray Diagnosiscited by 29
  9. 2023Fine-Grained Image-Text Alignment in Medical Imaging Enables Explainable Cyclic Image-Report Generationcited by 26
  10. 2024MIntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversationscited by 26
  11. 2023Medical Vision Language Pretraining: A surveycited by 24
  12. 2024Large Language Models in Drug Discovery and Development: From Disease Mechanisms to Clinical Trialscited by 24
  13. 2022CAMANet: Class Activation Map Guided Attention Network for Radiology Report Generationcited by 22
  14. 2023Towards long-tailed, multi-label disease classification from chest X-ray: Overview of the CXR-LT challengecited by 22
  15. 2023UniChest: Conquer-and-Divide Pre-training for Multi-Source Chest X-Ray Classificationcited by 21
  16. 2023LLM-driven Multimodal Target Volume Contouring in Radiation Oncologycited by 20
  17. 2025A Survey on Efficient Vision-Language Modelscited by 19
  18. 2024Read Like a Radiologist: Efficient Vision-Language Model for 3D Medical Imaging Interpretationcited by 18
  19. 2022DrugEHRQA: A Question Answering Dataset on Structured and Unstructured Electronic Health Records For Medicine Related Queriescited by 17
  20. 2023Vision-Language Generative Model for View-Specific Chest X-ray Generationcited by 14
  21. 2025MM-Skin: Enhancing Dermatology Vision-Language Model with an Image-Text Dataset Derived from Textbookscited by 14
  22. 2023LIMITR: Leveraging Local Information for Medical Image-Text Representationcited by 13
  23. 2023A scoping review on multimodal deep learning in biomedical images and textscited by 13
  24. 2024Uncertainty-aware Medical Diagnostic Phrase Identification and Groundingcited by 13
  25. 2024Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-trainingcited by 13
  26. 2024Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcarecited by 12
  27. 2025Vision Language Models in Medicinecited by 12
  28. 2025DenseGrounding: Improving Dense Language-Vision Semantics for Ego-Centric 3D Visual Groundingcited by 12
  29. 2023Towards Medical Artificial General Intelligence via Knowledge-Enhanced Multimodal Pretrainingcited by 11
  30. 2024Memory-based Cross-modal Semantic Alignment Network for Radiology Report Generationcited by 11
  31. 2023Bi-VLGM : Bi-Level Class-Severity-Aware Vision-Language Graph Matching for Text Guided Medical Image Segmentationcited by 10
  32. 2024RespLLM: Unifying Audio and Text with Multimodal LLMs for Generalized Respiratory Health Predictioncited by 10
  33. 2025anyECG-chat: A Generalist ECG-MLLM for Flexible ECG Input and Multi-Task Understandingcited by 10
  34. 2025CXR-LT 2024: A MICCAI challenge on long-tailed, multi-label, and zero-shot disease classification from chest X-raycited by 10
  35. 2024VR-GPT: Visual Language Model for Intelligent Virtual Reality Applicationscited by 9
  36. 2023Leveraging Summary Guidance on Medical Report Summarizationcited by 8
  37. 2023Missing-modality Enabled Multi-modal Fusion Architecture for Medical Datacited by 8
  38. 2024Examining Modality Incongruity in Multimodal Federated Learning for Medical Vision and Language-based Disease Detectioncited by 8

50 most cited of 96 All 96 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representations
  2. 2024Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering
  3. 2025A Reality Check of Vision-Language Pre-training in Radiology: Have We Progressed Using Text?
  4. 2024Advancing High Resolution Vision-Language Models in Biomedicine
  5. 2025MedGEN-Bench: A Contextually Entangled Benchmark for Open-ended Multimodal Medical Generation
  6. 2023Effectively Fine-tune to Improve Large Multimodal Models for Radiology Report Generation
  7. 2024Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology
  8. 2025Medical Large Vision Language Models with Multi-Image Visual Ability