mlsanityData to 7 Oct 2026

Prompting Medical Large Vision-Language Models to Diagnose Pathologies by Visual Question Answering

arXiv 2407.21368 ↗First posted 31 Jul 2024cs.CV cs.AI cs.CLcited by 5

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Large Vision-Language Models (LVLMs) have achieved significant success in recent years, and they have been extended to the medical domain. Although demonstrating satisfactory performance on medical Visual Question Answering (VQA) tasks, Medical LVLMs (MLVLMs) suffer from the hallucination problem, which makes them fail to diagnose complex pathologies. Moreover, they readily fail to learn minority pathologies due to imbalanced training data. We propose two prompting strategies for MLVLMs that reduce hallucination and improve VQA performance. In the first strategy, we provide a detailed explanation of the queried pathology. In the second strategy, we fine-tune a cheap, weak learner to achieve high performance on a specific metric, and textually provide its judgment to the MLVLM. Tested on the MIMIC-CXR-JPG and Chexpert datasets, our methods significantly improve the diagnostic F1 score, with the highest increase being 0.27. We also demonstrate that our prompting strategies can be extended to general LVLM domains. Based on POPE metrics, it effectively suppresses the false negative predictions of existing LVLMs and improves Recall by approximately 0.07.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 53 references that are in the map. First posted on arXiv 31 Jul 2024.

Builds on

41 of 53 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models13 Jun
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2015Fast R-CNNcited by 2,930
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  11. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
27 more
  1. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  2. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  3. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  5. 2018Object Hallucination in Image Captioningcited by 661
  6. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  7. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  8. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  9. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  10. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  11. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  12. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  13. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  14. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  15. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  16. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
  17. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  18. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 196
  19. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Modelscited by 124
  20. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  21. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMscited by 79
  22. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuningcited by 69
  23. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Promptscited by 47
  24. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Modelscited by 42
  25. 2022Breaking with Fixed Set Pathology Recognition through Report-Guided Contrastive Trainingcited by 26
  26. 2023Xplainer: From X-Ray Observations to Explainable Zero-Shot Diagnosiscited by 23
  27. 2022Significantly improving zero-shot X-ray pathology classification via fine-tuning pre-trained image-text encoderscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understandingcited by 12
  2. 2025MoMA: A Mixture-of-Multimodal-Agents Architecture for Enhancing Clinical Prediction Modellingcited by 7
  3. 2025MRGAgents: A Multi-Agent Framework for Improved Medical Report Generation with Med-LVLMscited by 5
  4. 2025Adaptive Diagnostic Reasoning Framework for Pathology with Multimodal Large Language Modelscited by 2
  5. 2026Med-R2: An Adversarial Benchmark for Evidence-Grounded Reasoning in Medical VLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decoding
  2. 2025Medical Large Vision Language Models with Multi-Image Visual Ability
  3. 2026V-Loop: Visual Logical Loop Verification for Hallucination Detection in Medical Visual Question Answering
  4. 2024Targeted Visual Prompting for Medical Visual Question Answering
  5. 2025MedHallTune: An Instruction-Tuning Benchmark for Mitigating Medical Hallucination in Vision-Language Models
  6. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM
  7. 2026To Agree or To Be Right? The Grounding-Sycophancy Tradeoff in Medical Vision-Language Models
  8. 2026Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs