mlsanityData to 7 Oct 2026

MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models

arXiv 2602.21950 ↗First posted 25 Feb 2026cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Multimodal large language models (MLLMs) have shown great potential in medical applications, yet existing benchmarks inadequately capture real-world clinical complexity. We introduce MEDSYN, a multilingual, multimodal benchmark of highly complex clinical cases with up to 7 distinct visual clinical evidence (CE) types per case. Mirroring clinical workflow, we evaluate 18 MLLMs on differential diagnosis (DDx) generation and final diagnosis (FDx) selection. While top models often match or even outperform human experts on DDx generation, all MLLMs exhibit a much larger DDx--FDx performance gap compared to expert clinicians, indicating a failure mode in synthesis of heterogeneous CE types. Ablations attribute this failure to (i) overreliance on less discriminative textual CE (ite.g., medical history) and (ii) a cross-modal CE utilization gap. We introduce Evidence Sensitivity to quantify the latter and show that a smaller gap correlates with higher diagnostic accuracy. Finally, we demonstrate how it can be used to guide interventions to improve model performance. We will open-source our benchmark and code.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 68 references that are in the map. First posted on arXiv 25 Feb 2026.

Builds on

15 of 68 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning8 Jun
  2. 2025Medical Large Vision Language Models with Multi-Image Visual Ability25 May
  3. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM14 Feb

Other references · 12

  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  2. 2022A Survey on In-context Learningcited by 717
  3. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  4. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  5. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  6. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  7. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 212
  8. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 86
  9. 2023Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videoscited by 84
  10. 2025MedFrameQA: A Multi-Image Medical VQA Benchmark for Clinical Reasoningcited by 33
  11. 2025MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Researchcited by 30
  12. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MedReaMM: Evaluating Large Multimodal Models on Expert-Level Clinical Diagnostic Synthesis
  2. 2026CURE: A Multimodal Benchmark for Clinical Understanding and Retrieval Evaluation
  3. 2026Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases
  4. 2024MultiViewDx: Evidence-Linked Multi-View Clinical Diagnosis
  5. 2025BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text
  6. 2025DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models
  7. 2026MMRareBench: A Rare-Disease Multimodal and Multi-Image Medical Benchmark
  8. 2024A Spectrum Evaluation Benchmark for Medical Multi-Modal Large Language Models