mlsanityData to 14 Oct 2026

MISS: A Generative Pretraining and Finetuning Approach for Med-VQA

arXiv 2401.05163 ↗First posted 10 Jan 2024cs.CV cs.AIcited by 14

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Medical visual question answering (VQA) is a challenging multimodal task, where Vision-Language Pre-training (VLP) models can effectively improve the generalization performance. However, most methods in the medical field treat VQA as an answer classification task which is difficult to transfer to practical application scenarios. Additionally, due to the privacy of medical images and the expensive annotation process, large-scale medical image-text pairs datasets for pretraining are severely lacking. In this paper, we propose a large-scale MultI-task Self-Supervised learning based framework (MISS) for medical VQA tasks. Unlike existing methods, we treat medical VQA as a generative task. We unify the text encoder and multimodal encoder and align image-text features through multi-task learning. Furthermore, we propose a Transfer-and-Caption method that extends the feature space of single-modal image datasets using Large Language Models (LLMs), enabling those traditional medical vision field task data to be applied to VLP. Experiments show that our method achieves excellent results with fewer multimodal datasets and demonstrates the advantages of generative VQA models.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 32 references that are in the map. First posted on arXiv 10 Jan 2024.

Builds on

20 of 32 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  10. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  11. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  12. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
7 more
  1. 2022Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Trainingcited by 104
  2. 2021Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?cited by 85
  3. 2019Overcoming Data Limitation in Medical Visual Question Answeringcited by 71
  4. 2020MedICaT: A Dataset of Medical Images, Captions, and Textual Referencescited by 71
  5. 2021Multiple Meta-model Quantifying for Medical Visual Question Answeringcited by 54
  6. 2023Masked Vision and Language Pre-training with Unimodal and Multimodal Contrastive Losses for Medical Visual Question Answeringcited by 29
  7. 2023Text-oriented Modality Reinforcement Network for Multimodal Sentiment Analysis from Unaligned Multimodal Sequencescited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 2; 2026: 3.

14 papers.

Sort:

By area:

  1. 2024Detecting and Evaluating Medical Hallucinations in Large Vision Language Modelscited by 57
  2. 2024Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalitiescited by 32
  3. 2024Towards Multimodal Sentiment Analysis Debiasing via Bias Purificationcited by 27
  4. 2024CoMT: Chain-of-Medical-Thought Reduces Hallucination in Medical Report Generationcited by 25
  5. 2024Robust Emotion Recognition in Context Debiasingcited by 24
  6. 2024PediatricsGPT: Large Language Models as Chinese Medical Assistants for Pediatric Applicationscited by 16
  7. 2024Can LLMs' Tuning Methods Work in Medical Multimodal Domain?cited by 12
  8. 2024From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practicecited by 8
  9. 2025Computed Tomography Visual Question Answering with Cross-modal Feature Graphingcited by 1
  10. 2025Barriers in Integrating Medical Visual Question Answering into Radiology Workflows: A Scoping Review and Clinicians' Insightscited by 1
  11. 2026Dual Causal Inference: Integrating Backdoor Adjustment and Instrumental Variable Learning for Medical VQAcited by 1
  12. 2024Skip and Skip: Segmenting Medical Images with Promptscited by 0
2 more
  1. 2026Learning to Trim: End-to-End Causal Graph Pruning with Dynamic Anatomical Feature Banks for Medical VQAcited by 0
  2. 2026KG-CMI: Knowledge graph enhanced cross-Mamba interaction for medical visual question answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Self-supervised vision-language pretraining for Medical visual question answering
  2. 2023Visual Question Answering in the Medical Domain
  3. 2024Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering
  4. 2025A Lightweight Large Vision-language Model for Multimodal Medical Images
  5. 2026MedProb: Probing Internal Representations of Vision-Language Models for Medical Question Answering
  6. 2025Med-GRIM: Enhanced Zero-Shot Medical VQA using prompt-embedded Multimodal Graph RAG
  7. 2026MedFG-VQA: Low-Frequency Memory and Graph Attention for Lightweight Medical VQA
  8. 2025Multimodal Retrieval-Augmented Generation with Large Language Models for Medical VQA