mlsanityData to 7 Oct 2026

Self-supervised vision-language pretraining for Medical visual question answering

arXiv 2211.13594 ↗First posted 24 Nov 2022cs.CV cs.AIcited by 30

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Medical image visual question answering (VQA) is a task to answer clinical questions, given a radiographic image, which is a challenging problem that requires a model to integrate both vision and language information. To solve medical VQA problems with a limited number of training data, pretrain-finetune paradigm is widely used to improve the model generalization. In this paper, we propose a self-supervised method that applies Masked image modeling, Masked language modeling, Image text matching and Image text alignment via contrastive learning (M2I2) for pretraining on medical image caption dataset, and finetunes to downstream medical VQA tasks. The proposed method achieves state-of-the-art performance on all the three public medical VQA datasets. Our codes and models are available at https://github.com/pengfeiliHEU/M2I2.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 18 of its 26 references that are in the map. First posted on arXiv 24 Nov 2022.

Builds on

18 of 26 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  10. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  12. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
6 more
  1. 2021Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?cited by 85
  2. 2019Overcoming Data Limitation in Medical Visual Question Answeringcited by 71
  3. 2021MMBERT: Multimodal BERT Pretraining for Improved Medical VQAcited by 68
  4. 2021Multiple Meta-model Quantifying for Medical Visual Question Answeringcited by 54
  5. 2021Cross-Modal Self-Attention with Multi-Task Pre-Training for Medical Visual Question Answeringcited by 26
  6. 2020Hierarchical Deep Multi-modal Network for Medical Visual Question Answeringcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 13; 2025: 8; 2026: 4.

30 papers.

Sort:

By area:

  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  2. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  3. 2024MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicinecited by 79
  4. 2023OphGLM: Training an Ophthalmology Large Language-and-Vision Assistant based on Instructions and Dialoguecited by 34
  5. 2023Medical Vision Language Pretraining: A surveycited by 24
  6. 2024Biomedical Visual Instruction Tuning with Clinician Preference Alignmentcited by 21
  7. 2024R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interestcited by 21
  8. 2024PA-LLaVA: A Large Language-Vision Assistant for Human Pathology Image Understandingcited by 15
  9. 2023A scoping review on multimodal deep learning in biomedical images and textscited by 13
  10. 2024LaPA: Latent Prompt Assist Model For Medical Visual Question Answeringcited by 12
  11. 2025LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understandingcited by 12
  12. 2024Parameter-Efficient Fine-Tuning Medical Multimodal Large Language Models for Medical Visual Groundingcited by 9
18 more
  1. 2024Cost-effective Instruction Learning for Pathology Vision and Language Analysiscited by 7
  2. 2024Accurate Explanation Model for Image Classifiers using Class Association Embeddingcited by 6
  3. 2024ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Modelscited by 6
  4. 2025Structure Causal Models and LLMs Integration in Medical Visual Question Answeringcited by 6
  5. 2024Advancing High Resolution Vision-Language Models in Biomedicinecited by 5
  6. 2025SilVar-Med: A Speech-Driven Visual Language Model for Explainable Abnormality Detection in Medical Imagingcited by 5
  7. 2024Prompt-based Personalized Federated Learning for Medical Visual Question Answeringcited by 4
  8. 2025Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Modelscited by 4
  9. 2025Knowing or Guessing? Robust Medical Visual Question Answering via Joint Consistency and Contrastive Learningcited by 2
  10. 2024Enhancing Generalization in Medical Visual Question Answering Tasks via Gradient-Guided Model Perturbationcited by 1
  11. 2024Which Client is Reliable?: A Reliable and Personalized Prompt-based Federated Learning for Medical Image Question Answeringcited by 1
  12. 2025Alignment, Mining and Fusion: Representation Alignment with Hard Negative Mining and Selective Knowledge Fusion for Medical Visual Question Answeringcited by 1
  13. 2026Dual Causal Inference: Integrating Backdoor Adjustment and Instrumental Variable Learning for Medical VQAcited by 1
  14. 2025Bridging the Semantic Gaps: Improving Medical VQA Consistency with LLM-Augmented Question Setscited by 0
  15. 2025CMI-MTL: Cross-Mamba interaction based multi-task learning for medical visual question answeringcited by 0
  16. 2026Learning to Trim: End-to-End Causal Graph Pruning with Dynamic Anatomical Feature Banks for Medical VQAcited by 0
  17. 2026KG-CMI: Knowledge graph enhanced cross-Mamba interaction for medical visual question answeringcited by 0
  18. 2026CheXGround: Anatomical Region Tokens for Grounded Longitudinal Chest X-ray Interpretationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MISS: A Generative Pretraining and Finetuning Approach for Med-VQA
  2. 2023Visual Question Answering in the Medical Domain
  3. 2024Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering
  4. 2023RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-training
  5. 2025Adapting Lightweight Vision Language Models for Radiological Visual Question Answering
  6. 2023Interpretable Medical Image Visual Question Answering via Multi-Modal Relationship Graph Learning
  7. 2024STLLaVA-Med: Self-Training Large Language and Vision Assistant for Medical Question-Answering
  8. 2026Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence