mlsanityData to 14 Oct 2026

Why Text Prevails: Vision May Undermine Multimodal Medical Decision Making

arXiv 2512.13747 ↗First posted 15 Dec 2025cs.CV cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

With the rapid progress of large language models (LLMs), advanced multimodal large language models (MLLMs) have demonstrated impressive zero-shot capabilities on vision-language tasks. In the biomedical domain, however, even state-of-the-art MLLMs struggle with basic Medical Decision Making (MDM) tasks. We investigate this limitation using two challenging datasets: (1) three-stage Alzheimer's disease (AD) classification (normal, mild cognitive impairment, dementia), where category differences are visually subtle, and (2) MIMIC-CXR chest radiograph classification with 14 non-mutually exclusive conditions. Our empirical study shows that text-only reasoning consistently outperforms vision-only or vision-text settings, with multimodal inputs often performing worse than text alone. To mitigate this, we explore three strategies: (1) in-context learning with reason-annotated exemplars, (2) vision captioning followed by text-only inference, and (3) few-shot fine-tuning of the vision tower with classification supervision. These findings reveal that current MLLMs lack grounded visual understanding and point to promising directions for improving multimodal decision making in healthcare.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 42 references that are in the map. First posted on arXiv 15 Dec 2025.

Builds on

28 of 42 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology16 Dec
  2. 2024MMedAgent: Learning to Use Medical Tools with Multi-modal Agent2 Jul
  3. 2023XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models13 Jun
  4. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  8. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  11. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
  12. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 316
12 more
  1. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 163
  2. 2022REFUGE2 Challenge: A Treasure Trove for Multi-Dimension Analysis and Evaluation in Glaucoma Screeningcited by 38
  3. 2024SLIDE: A Framework Integrating Small and Large Language Models for Open-Domain Dialogues Evaluationcited by 11
  4. 2024X-ray Made Simple: Lay Radiology Report Generation and Robust Evaluationcited by 10
  5. 2024SouLLMate: An Adaptive LLM-Driven System for Advanced Mental Health Support and Assessment, Based on a Systematic Application Surveycited by 8
  6. 2024SouLLMate: An Application Enhancing Diverse Mental Health Support with Adaptive LLMs, Prompt Engineering, and RAG Techniquescited by 8
  7. 2023Evaluating Open-Domain Dialogues in Latent Space with Next Sentence Prediction and Mutual Informationcited by 6
  8. 2025Zeus: Zero-shot LLM Instruction for Union Segmentation in Multimodal Medical Imagingcited by 4
  9. 2024Constrained Multiview Representation for Self-supervised Contrastive Learningcited by 3
  10. 2024Interpretable Spatio-Temporal Embedding for Brain Structural-Effective Network with Ordinary Differential Equationcited by 3
  11. 2024A Heterogeneous Graph Neural Network Fusing Functional and Structural Connectivity for MCI Diagnosiscited by 3
  12. 2025DRE: An Effective Dual-Refined Method for Integrating Small and Large Language Models in Open-Domain Dialogue Evaluationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Physics-based phenomenological characterization of cross-modal bias in multimodal modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Models
  2. 2026When Prompts Mislead: Textual Dominance and Diagnostic Bias in MLLMs
  3. 2026NeuroVLM-Bench: Evaluation of Vision-Enabled Large Language Models for Clinical Reasoning in Neurological Disorders
  4. 2026Medical Context Distorts Decisions in Clinical Vision Language Models
  5. 2025Benchmarking the Thinking Mode of Multimodal Large Language Models in Clinical Tasks
  6. 2026Lost in the Hype: Revealing and Dissecting the Performance Degradation of Medical Multimodal Large Language Models in Image Classification
  7. 2025MedBLINK: Probing Basic Perception in Multimodal Language Models for Medicine
  8. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study