mlsanityData to 7 Oct 2026

MedThink: Explaining Medical Visual Question Answering via Multimodal Decision-Making Rationale

arXiv 2404.12372 ↗First posted 18 Apr 2024cs.CVcited by 14

On the map: LLM reasoning → Clinical and biomedical applications

Abstract (arXiv)

Medical Visual Question Answering (MedVQA), which offers language responses to image-based medical inquiries, represents a challenging task and significant advancement in healthcare. It assists medical experts to swiftly interpret medical images, thereby enabling faster and more accurate diagnoses. However, the model interpretability and transparency of existing MedVQA solutions are often limited, posing challenges in understanding their decision-making processes. To address this issue, we devise a semi-automated annotation process to streamline data preparation and build new benchmark MedVQA datasets R-RAD, R-SLAKE and R-Path. These datasets provide intermediate medical decision-making rationales generated by multimodal large language models and human annotations for question-answering pairs in existing MedVQA datasets, i.e., VQA-RAD, SLAKE and PathVQA. Moreover, we design a novel framework, MedThink, which finetunes lightweight pretrained generative models by incorporating medical decision-making rationales. MedThink includes three distinct strategies to generate decision outcomes and corresponding rationales, thereby clearly showcasing the medical decision-making process during reasoning. Our comprehensive experiments show that our method achieves an accuracy of 83.5% on R-RAD, 86.3% on R-SLAKE and 87.2% on R-Path. These results significantly exceed those of existing state-of-the-art models with comparable parameters. Datasets and code will be released.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 36 of its 51 references that are in the map. First posted on arXiv 18 Apr 2024.

Builds on

36 of 51 references.

References · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  10. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  11. 2017A Survey on Deep Learning in Medical Image Analysiscited by 1,182
  12. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
24 more
  1. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  2. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  3. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  4. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  5. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  6. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  7. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  8. 2019A large annotated medical image dataset for the development and evaluation of segmentation algorithmscited by 371
  9. 2020CHAOS Challenge -- Combined (CT-MR) Healthy Abdominal Organ Segmentationcited by 307
  10. 2018Bilinear Attention Networkscited by 277
  11. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  12. 2023HuatuoGPT, towards Taming Language Model to Be a Doctorcited by 211
  13. 2017Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answeringcited by 117
  14. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  15. 2019Overcoming Data Limitation in Medical Visual Question Answeringcited by 71
  16. 2023Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Modelscited by 58
  17. 2021Multiple Meta-model Quantifying for Medical Visual Question Answeringcited by 54
  18. 2023A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertisecited by 49
  19. 2023Huatuo-26M, a Large-scale Chinese Medical QA Datasetcited by 46
  20. 2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Modelscited by 43
  21. 2023A ChatGPT Aided Explainable Framework for Zero-Shot Medical Image Diagnosiscited by 30
  22. 2024Residual-based Language Models are Free Boosters for Biomedical Imagingcited by 23
  23. 2024Adaptive Ensembles of Fine-Tuned Transformers for LLM-Generated Text Detectioncited by 22
  24. 2023Gemini vs GPT-4V: A Preliminary Comparison and Combination of Vision-Language Models Through Qualitative Casescited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 5; 2026: 5.

14 papers.

Sort:

By area:

  1. 2024MedCoT: Medical Chain of Thought via Hierarchical Expertcited by 46
  2. 2025A Comprehensive Survey on the Trustworthiness of Large Language Models in Healthcarecited by 27
  3. 2024R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interestcited by 21
  4. 2024WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Drivingcited by 19
  5. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  6. 2026MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolutioncited by 12
  7. 2025KPL: Training-Free Medical Knowledge Mining of Vision-Language Modelscited by 10
  8. 2025Fair-MoE: Fairness-Oriented Mixture of Experts in Vision-Language Modelscited by 6
  9. 2025Anatomical Region-Guided Contrastive Decoding: A Plug-and-Play Strategy for Mitigating Hallucinations in Medical VLMscited by 2
  10. 2026MM-NeuroOnco: A Multimodal Benchmark and Instruction Dataset for MRI-Based Brain Tumor Diagnosiscited by 2
  11. 2025From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Modelscited by 1
  12. 2026Breaking Failure Cascades: Step-Aware Reinforcement Learning for Medical Multimodal Reasoningcited by 1
2 more
  1. 2026X-PCR: A Benchmark for Cross-modality Progressive Clinical Reasoning in Ophthalmic Diagnosiscited by 0
  2. 2026CheXGround: Anatomical Region Tokens for Grounded Longitudinal Chest X-ray Interpretationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Visual Question Answering in the Medical Domain
  2. 2026MedProb: Probing Internal Representations of Vision-Language Models for Medical Question Answering
  3. 2026MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering
  4. 2025Multimodal Retrieval-Augmented Generation with Large Language Models for Medical VQA
  5. 2026Medical thinking with multiple images
  6. 2026MedQA-MM: Shortcuts Behind Medical Visual Reasoning
  7. 2025Knowing or Guessing? Robust Medical Visual Question Answering via Joint Consistency and Contrastive Learning
  8. 2025MedXplain-VQA: Multi-Component Explainable Medical Visual Question Answering