mlsanityData to 7 Oct 2026

Multi-modal Pre-training for Medical Vision-language Understanding and Generation: An Empirical Study with A New Benchmark

arXiv 2306.06494 ↗First posted 10 Jun 2023cs.CV cs.AIcited by 5

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

With the availability of large-scale, comprehensive, and general-purpose vision-language (VL) datasets such as MSCOCO, vision-language pre-training (VLP) has become an active area of research and proven to be effective for various VL tasks such as visual-question answering. However, studies on VLP in the medical domain have so far been scanty. To provide a comprehensive perspective on VLP for medical VL tasks, we conduct a thorough experimental analysis to study key factors that may affect the performance of VLP with a unified vision-language Transformer. To allow making sound and quick pre-training decisions, we propose RadioGraphy Captions (RGC), a high-quality, multi-modality radiographic dataset containing 18,434 image-caption pairs collected from an open-access online database MedPix. RGC can be used as a pre-training dataset or a new benchmark for medical report generation and medical image-text retrieval. By utilizing RGC and other available datasets for pre-training, we develop several key insights that can guide future medical VLP research and new strong baselines for various medical VL tasks.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 70 references that are in the map. First posted on arXiv 10 Jun 2023.

Builds on

52 of 70 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training24 May

Other references · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2017Decoupled Weight Decay Regularizationcited by 1,722
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
38 more
  1. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  2. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  3. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  4. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  5. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  6. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  7. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  8. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  9. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  10. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  11. 2020Generating Radiology Reports via Memory-driven Transformercited by 363
  12. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  13. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 310
  14. 2019PadChest: A large chest x-ray image dataset with multi-label annotated reportscited by 308
  15. 2018Bilinear Attention Networkscited by 277
  16. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  17. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  18. 2016Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Groundingcited by 243
  19. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  20. 2017On the Automatic Generation of Medical Imaging Reportscited by 215
  21. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  22. 2021Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generationcited by 160
  23. 2018Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generationcited by 141
  24. 2017MUTAN: Multimodal Tucker Fusion for Visual Question Answeringcited by 137
  25. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 136
  26. 2022Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Trainingcited by 104
  27. 2019Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generationcited by 101
  28. 2020BIMCV COVID-19+: a large annotated dataset of RX and CT images from COVID-19 patientscited by 81
  29. 2017Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answeringcited by 79
  30. 2022Competence-based Multimodal Curriculum Learning for Medical Report Generationcited by 74
  31. 2019Overcoming Data Limitation in Medical Visual Question Answeringcited by 71
  32. 2020MedICaT: A Dataset of Medical Images, Captions, and Textual Referencescited by 71
  33. 2021MMBERT: Multimodal BERT Pretraining for Improved Medical VQAcited by 68
  34. 2020Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-Ray Reportscited by 58
  35. 2022Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledgecited by 51
  36. 2021Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generationcited by 47
  37. 2021Cross-Modal Self-Attention with Multi-Task Pre-Training for Medical Visual Question Answeringcited by 26
  38. 2021Self-supervised Image-text Pre-training With Mixed Data In Chest X-rayscited by 16

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 0; 2026: 1.

5 papers.

  1. 2024GEMeX: A Large-Scale, Groundable, and Explainable Medical VQA Benchmark for Chest X-ray Diagnosiscited by 28
  2. 2023Medical Vision Language Pretraining: A surveycited by 24
  3. 2024RJUA-MedDQA: A Multimodal Benchmark for Medical Document Question Answering and Clinical Reasoningcited by 8
  4. 2024Daily Physical Activity Monitoring -- Adaptive Learning from Multi-source Motion Sensor Datacited by 2
  5. 2026Robust Pre-Training of Medical Vision-and-Language Models with Domain-Invariant Multi-Modal Masked Reconstructioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Reality Check of Vision-Language Pre-training in Radiology: Have We Progressed Using Text?
  2. 2025Comprehensive language-image pre-training for 3D medical image understanding
  3. 2023Utilizing Synthetic Data for Medical Vision-Language Pre-training: Bypassing the Need for Real Images
  4. 2023Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime
  5. 2024Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering
  6. 2024Can Medical Vision-Language Pre-training Succeed with Purely Synthetic Data?
  7. 2024Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology
  8. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study