mlsanityData to 7 Oct 2026

Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review

arXiv 2403.02469 ↗First posted 4 Mar 2024cs.CV cs.LGcited by 121

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Medical vision-language models (VLMs) combine computer vision (CV) and natural language processing (NLP) to analyze visual and textual medical data. Our paper reviews recent advancements in developing VLMs specialized for healthcare, focusing on models designed for medical report generation and visual question answering (VQA). We provide background on NLP and CV, explaining how techniques from both fields are integrated into VLMs to enable learning from multimodal data. Key areas we address include the exploration of medical vision-language datasets, in-depth analyses of architectures and pre-training strategies employed in recent noteworthy medical VLMs, and comprehensive discussion on evaluation metrics for assessing VLMs' performance in medical report generation and VQA. We also highlight current challenges and propose future directions, including enhancing clinical validity and addressing patient privacy concerns. Overall, our review summarizes recent progress in developing VLMs to harness multimodal medical data for improved healthcare applications.

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 134 of its 217 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

134 of 217 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Medical Vision Language Pretraining: A survey11 Dec
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
  3. 2021Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training24 May

Other references · 131

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  6. 2020Language Models are Few-Shot Learnerscited by 32,199
  7. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  8. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  9. 2022Training language models to follow instructions with human feedbackcited by 15,712
  10. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  11. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  12. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
38 more
  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2017Mask R-CNNcited by 7,259
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  10. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  13. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  14. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  15. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  16. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  17. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  18. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  19. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  20. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  21. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  22. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  23. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  24. 2023Mistral 7Bcited by 3,065
  25. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  26. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  27. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  28. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  29. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  30. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  31. 2015VQA: Visual Question Answeringcited by 2,489
  32. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  33. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  34. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  35. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  36. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  37. 2021Vision Transformers for Dense Predictioncited by 1,789
  38. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612

50 most cited of 131 All 131 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 59; 2026: 39.

121 papers.

Sort:

By area:

  1. 2025MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learningcited by 172
  2. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  3. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  4. 2025Sparse Autoencoders Learn Monosemantic Features in Vision-Language Modelscited by 70
  5. 2024VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledgecited by 50
  6. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  7. 2024A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directionscited by 34
  8. 2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineeringcited by 34
  9. 2024CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Datasetcited by 33
  10. 2025Membership Inference Attacks Against Vision-Language Modelscited by 26
  11. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  12. 2025A Survey on Efficient Vision-Language Modelscited by 19
38 more
  1. 2025Vid-SME: Membership Inference Attacks against Large Video Understanding Modelscited by 17
  2. 2024Seeing Through Their Eyes: Evaluating Visual Perspective Taking in Vision Language Modelscited by 15
  3. 2024Med-2E3: A 2D-Enhanced 3D Medical Multimodal Large Language Modelcited by 15
  4. 2025Are Vision Language Models Ready for Clinical Diagnosis? A 3D Medical Benchmark for Tumor-centric Visual Question Answeringcited by 14
  5. 2025SCAM: A Real-World Typographic Robustness Evaluation for Multimodal Foundation Modelscited by 12
  6. 2025Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Imagescited by 12
  7. 2024A Survey of Medical Vision-and-Language Applications and Their Techniquescited by 11
  8. 2025MORALISE: A Structured Benchmark for Moral Alignment in Visual Language Modelscited by 10
  9. 2025Mitigating Hallucination in Large Vision-Language Models via Adaptive Attention Calibrationcited by 10
  10. 2025DrVD-Bench: Do Vision-Language Models Reason Like Human Doctors in Medical Image Diagnosis?cited by 10
  11. 2024Boosting Vision-Language Models for Histopathology Classification: Predict all at oncecited by 8
  12. 2024From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practicecited by 8
  13. 2025EchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Modelscited by 8
  14. 2025ReEvalMed: Rethinking Medical Report Evaluation by Aligning Metrics with Real-World Clinical Judgmentcited by 8
  15. 2025V3LMA: Visual 3D-enhanced Language Model for Autonomous Drivingcited by 7
  16. 2025Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Modelscited by 7
  17. 2026Uncertainty Quantification for Multimodal Large Language Models with Incoherence-adjusted Semantic Volumecited by 7
  18. 2024HoneyBee: A Scalable Modular Framework for Creating Multimodal Oncology Datasets with Foundational Embedding Modelscited by 6
  19. 2025Applications of Large Models in Medicinecited by 6
  20. 2025Video Panels for Long Video Understandingcited by 6
  21. 2025LongWriter-V: Enabling Ultra-Long and High-Fidelity Generation in Vision-Language Modelscited by 5
  22. 2025A LongFormer-Based Framework for Accurate and Efficient Medical Text Summarizationcited by 5
  23. 2025UniVRSE: Unified Vision-conditioned Response Semantic Entropy for Hallucination Detection in Medical Vision-Language Modelscited by 5
  24. 2025MOTOR: Multimodal Optimal Transport via Grounded Retrieval in Medical Visual Question Answeringcited by 5
  25. 2024Exploring Vision Language Models for Facial Attribute Recognition: Emotion, Race, Gender, and Agecited by 4
  26. 2024Semantic Consistency-Based Uncertainty Quantification for Factuality in Radiology Report Generationcited by 4
  27. 2025StreamingRAG: Real-time Contextual Retrieval and Generation Frameworkcited by 4
  28. 2025MedM-VL: What Makes a Good Medical LVLM?cited by 4
  29. 2025X-Ray-CoT: Interpretable Chest X-ray Diagnosis with Vision-Language Models via Chain-of-Thought Reasoningcited by 4
  30. 2025Evolution of meta's llama models and parameter-efficient fine-tuning of large language models: a surveycited by 4
  31. 2024Exploring Foundation Models in Remote Sensing Image Change Detection: A Comprehensive Surveycited by 3
  32. 2025MEENA (PersianMMMU): Multimodal-Multilingual Educational Exams for N-level Assessmentcited by 3
  33. 2025Eye Gaze Tells You Where to Compute: Gaze-Driven Efficient VLMscited by 3
  34. 2025RAU: Reference-based Anatomical Understanding with Vision Language Modelscited by 3
  35. 2025Black-Box Membership Inference Attack for LVLMs via Prior Knowledge-Calibrated Memory Probingcited by 3
  36. 2025Semantic Document Derendering: SVG Reconstruction via Vision-Language Modelingcited by 3
  37. 2025On the Utility of Foundation Models for Fast MRI: Vision-Language-Guided Image Reconstructioncited by 3
  38. 2026Do Multimodal RAG Systems Leak Data? A Comprehensive Evaluation of Membership Inference and Image Caption Retrieval Attackscited by 3

50 most cited of 121 All 121 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering
  2. 2025Vision Language Models in Medicine
  3. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM
  4. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
  5. 2025Multimodal Retrieval-Augmented Generation with Large Language Models for Medical VQA
  6. 2024SURE-VQA: Systematic Understanding of Robustness Evaluation in Medical VQA Tasks
  7. 2023Visual Question Answering in the Medical Domain
  8. 2024Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology