# IMCBench: A benchmark for multimodal LLMs in Image-grounded Medical Conversations

arXiv: https://arxiv.org/abs/2606.28556 · HTML: https://mlsanity.com/paper/2606.28556/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 26 Jun 2026.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)
- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 12 of 12)

- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset](https://mlsanity.com/paper/2104.09957/) (2021)
- [OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM](https://mlsanity.com/paper/2402.09181/) (2024)
- [Disparities in Dermatology AI: Assessments Using Diverse Clinical Images](https://mlsanity.com/paper/2111.08006/) (2021)
- [Evaluating LLM -- Generated Multimodal Diagnosis from Medical Images and Symptom Analysis](https://mlsanity.com/paper/2402.01730/) (2024)
- [3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark](https://mlsanity.com/paper/2504.13861/) (2025)
- [MidMed: Towards Mixed-Type Dialogues for Medical Consultation](https://mlsanity.com/paper/2306.02923/) (2023)
- [WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation](https://mlsanity.com/paper/2410.12722/) (2024)
- [Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Grounding](https://mlsanity.com/paper/2405.19567/) (2024)
- [MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text](https://mlsanity.com/paper/2508.10947/) (2025)
- [ZALM3: Zero-Shot Enhancement of Vision-Language Alignment via In-Context Information in Multi-Turn Multimodal Medical Dialogue](https://mlsanity.com/paper/2409.17610/) (2024)

## Abstract (arXiv)

Recent advances in large language models and vision-language models have enabled reasoning over multimodal data, offering opportunities for clinical applications such as decision support and triaging. However, existing medical AI benchmarks are fragmented: some support multi-turn dialogues but lack images, while others provide multimodal inputs but focus on single-turn QA tasks. To address this gap, we introduce IMCBench, an image-grounded, multi-turn medical conversation benchmark that pairs real, publicly available clinical images with synthetic patient profiles to simulate realistic patient-clinician interactions. Each conversation is evaluated across three clinical dimensions: safety, accuracy, and appropriate use of uncertainty in diagnosis. We benchmark eight multimodal frontier models across four model families (Claude, GPT, Nova, and Llama), scoring each on a 1-5 scale using LLM-as-Jury scoring calibrated against expert clinician annotations. Our results show that Claude Opus 4.6 achieves the highest overall score (3.61), followed by Claude Sonnet 4.6 (3.30) and GPT-5.2 (3.29), though no model dominates all dimensions and safety degrades for both malignant and rare conditions (Delta = -0.27 each). Ablation studies further reveal that both visual input and EHR context contribute to safe guidance (safety drops of 0.18 and 0.23 on average when each is removed), with stronger models leveraging visual features more effectively. Together, these findings demonstrate that accurate clinical description does not guarantee safe patient guidance, motivating the need for multi-dimensional evaluation frameworks in medical AI.