mlsanityData to 7 Oct 2026

Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Models

arXiv 2303.05977 ↗First posted 10 Mar 2023cs.CVcited by 58

Not on the map. Why?

Cited by 24 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Medical Visual Question Answering (VQA) is an important challenge, as it would lead to faster and more accurate diagnoses and treatment decisions. Most existing methods approach it as a multi-class classification problem, which restricts the outcome to a predefined closed-set of curated answers. We focus on open-ended VQA and motivated by the recent advances in language models consider it as a generative task. Leveraging pre-trained language models, we introduce a novel method particularly suited for small, domain-specific, medical datasets. To properly communicate the medical images to the language model, we develop a network that maps the extracted visual features to a set of learnable tokens. Then, alongside the question, these learnable tokens directly prompt the language model. We explore recent parameter-efficient fine-tuning strategies for language models, which allow for resource- and data-efficient fine-tuning. We evaluate our approach on the prime medical VQA benchmarks, namely, Slake, OVQA and PathVQA. The results demonstrate that our approach outperforms existing methods across various training settings while also being computationally efficient.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 21 of its 35 references that are in the map. First posted on arXiv 10 Mar 2023.

Builds on

21 of 35 references.

References · 21

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  10. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  11. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  12. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
9 more
  1. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  2. 2021Medical Visual Question Answering: A Surveycited by 93
  3. 2021Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?cited by 85
  4. 2019Overcoming Data Limitation in Medical Visual Question Answeringcited by 71
  5. 2021MMBERT: Multimodal BERT Pretraining for Improved Medical VQAcited by 68
  6. 2021Multiple Meta-model Quantifying for Medical Visual Question Answeringcited by 54
  7. 2021Cross-Modal Self-Attention with Multi-Task Pre-Training for Medical Visual Question Answeringcited by 26
  8. 2023Meta Learning to Bridge Vision and Language Models for Multimodal Few-Shot Learningcited by 20
  9. 2022Clinical Prompt Learning with Frozen Language Modelscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 19; 2025: 19; 2026: 8.

58 papers.

Sort:

By area:

  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  2. 2023Towards Generalist Biomedical AIcited by 286
  3. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  4. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 176
  5. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 85
  6. 2024MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicinecited by 79
  7. 2023PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathologycited by 57
  8. 2023Prompt Engineering for Healthcare: Methodologies and Applicationscited by 52
  9. 2023KnowledGPT: Enhancing Large Language Models with Retrieval and Storage Access on Knowledge Basescited by 52
  10. 2023MedBLIP: Bootstrapping Language-Image Pre-training from 3D Medical Images and Textscited by 51
  11. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  12. 2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Modelscited by 43
38 more
  1. 2023CLIP in Medical Imaging: A Surveycited by 40
  2. 2023OphGLM: Training an Ophthalmology Large Language-and-Vision Assistant based on Instructions and Dialoguecited by 34
  3. 20253D-RAD: A Comprehensive 3D Radiology Med-VQA Dataset with Multi-Temporal Analysis and Diverse Diagnostic Taskscited by 24
  4. 2024PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imagingcited by 21
  5. 2024Biomedical Visual Instruction Tuning with Clinician Preference Alignmentcited by 21
  6. 2024MC-CoT: A Modular Collaborative CoT Framework for Zero-shot Medical-VQA with LLM and MLLM Integrationcited by 21
  7. 2024R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interestcited by 21
  8. 2025A Survey on Efficient Vision-Language Modelscited by 19
  9. 2024ProtChatGPT: Towards Understanding Proteins with Large Language Modelscited by 18
  10. 2024ChEX: Interactive Localization and Region Description in Chest X-rayscited by 18
  11. 2025OmniV-Med: Scaling Medical Vision-Language Model for Universal Visual Understandingcited by 17
  12. 2024A Survey on Trustworthiness in Foundation Models for Medical Image Analysiscited by 16
  13. 2024PA-LLaVA: A Large Language-Vision Assistant for Human Pathology Image Understandingcited by 15
  14. 2024MedThink: Explaining Medical Visual Question Answering via Multimodal Decision-Making Rationalecited by 14
  15. 2025LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understandingcited by 12
  16. 2026MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolutioncited by 12
  17. 2024A Survey of Medical Vision-and-Language Applications and Their Techniquescited by 11
  18. 2025MedSeg-R: Reasoning Segmentation in Medical Images with Multimodal Large Language Modelscited by 10
  19. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Reviewcited by 9
  20. 2025CaReAQA: A Cardiac and Respiratory Audio Question Answering Model for Open-Ended Diagnostic Reasoningcited by 8
  21. 2024Cost-effective Instruction Learning for Pathology Vision and Language Analysiscited by 7
  22. 2025Structure Causal Models and LLMs Integration in Medical Visual Question Answeringcited by 6
  23. 2024Advancing High Resolution Vision-Language Models in Biomedicinecited by 5
  24. 2023A Survey on Image-text Multimodal Modelscited by 4
  25. 2025Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Modelscited by 4
  26. 2023Effectively Fine-tune to Improve Large Multimodal Models for Radiology Report Generationcited by 3
  27. 2024Language Models Meet Anomaly Detection for Better Interpretability and Generalizabilitycited by 3
  28. 2025Grounding Chest X-Ray Visual Question Answering with Generated Radiology Reportscited by 3
  29. 2025Adaptation of Foundation Models for Medical Image Analysis: Strategies, Challenges, and Future Directionscited by 3
  30. 2025Knowing or Guessing? Robust Medical Visual Question Answering via Joint Consistency and Contrastive Learningcited by 2
  31. 2024Design as Desired: Utilizing Visual Question Answering for Multimodal Pre-trainingcited by 1
  32. 2024Which Client is Reliable?: A Reliable and Personalized Prompt-based Federated Learning for Medical Image Question Answeringcited by 1
  33. 2025HSCR: Hierarchical Self-Contrastive Rewarding for Aligning Medical Vision Language Modelscited by 1
  34. 2025Computed Tomography Visual Question Answering with Cross-modal Feature Graphingcited by 1
  35. 2026Dual Causal Inference: Integrating Backdoor Adjustment and Instrumental Variable Learning for Medical VQAcited by 1
  36. 2026Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Modelscited by 1
  37. 2026OpenMedQ: Broad Open Pretraining for Medical Vision-Language Modelscited by 1
  38. 2025Brain Imaging Foundation Models, Are We There Yet? A Systematic Review of Foundation Models for Brain Imaging and Biomedical Researchcited by 0

50 most cited of 58 All 58 (JSON)