mlsanityData to 14 Oct 2026

MVC-Bench: Benchmarking Calibration of Medical Vision-Language Models

arXiv 2608.27004 ↗First posted 27 Aug 2026cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Reliable evaluation of vision-language models (VLMs) and medical vision-language models (Medical-VLMs) requires calibrated confidence, particularly under realistic clinical conditions. However, existing efforts mainly focused on improving accuracy, leaving calibration in the medical domain underexplored. To this end, we propose MVC-Bench, a calibration-centric benchmark for medical image classification with VLMs and Medical-VLMs. MVC-Bench assesses the calibration across three axes: (i) robustness to modality, backbone, and domain shift (ii) effectiveness of calibration strategies and prompt-tuning methods (iii) stability under prompt-template and random-seed variations. The benchmark covers eight different backbones, three medical modalities, including fundus imaging, histopathology, and chest X-ray under in-domain and domain shift settings. It compares post-hoc calibration, train-time calibration, and zero-shot inference methods, together with six prompt-tuning methods. Across more than 1638 controlled experiments, we report accuracy and Expected Calibration Error (ECE) as primary metrics, and further report results with complementary calibration measures, including Maximum Calibration Error (MCE) and Adaptive Calibration Error (ACE). We further investigate the underlying causes of miscalibration in VLMs and Medical-VLMs and propose a simple train-time calibration method, Multi-Class Margin (MCM) regularization, which achieves lowest ECE on 10 out of 12 settings in in-domain and remains competitive under domain shifts. Collectively, MVC-Bench provides a structured evaluation framework and actionable guidance for improving calibration in safety-critical medical workflows.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 59 references that are in the map. First posted on arXiv 27 Aug 2026.

Builds on

29 of 59 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024PromptSmooth: Certifying Robustness of Medical Vision-Language Models via Prompt Learning29 Aug

Other references · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  4. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  5. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  6. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,245
  7. 2019When Does Label Smoothing Help?cited by 988
  8. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  9. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  10. 2017A systematic study of the class imbalance problem in convolutional neural networkscited by 513
  11. 2019Measuring Calibration in Deep Learningcited by 446
  12. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
16 more
  1. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 379
  2. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 364
  3. 2022Prompt-aligned Gradient for Prompt Tuningcited by 341
  4. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 308
  5. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 305
  6. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  7. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 194
  8. 2019Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathologycited by 173
  9. 2021The Devil is in the Margin: Margin-based Label Smoothing for Network Calibrationcited by 91
  10. 2024C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersioncited by 83
  11. 2022A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved Neural Network Calibrationcited by 49
  12. 2024Robust Calibration of Large Vision-Language Adapterscited by 28
  13. 2015Binary Classifier Calibration using an Ensemble of Near Isotonic Regression Modelscited by 21
  14. 2023Two Sides of Miscalibration: Identifying Over and Under-Confidence Prediction for Network Calibrationcited by 14
  15. 2020Improving Calibration and Out-of-Distribution Detection in Medical Image Segmentation with Convolutional Neural Networkscited by 6
  16. 2025A-TPT: Angular Diversity Calibration Properties for Test-Time Prompt Tuning of Vision-Language Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026OmniMed-Jev: Calibrating LVLM Confidence for Trustworthy Medical Multimodal Decisions via System Onecited by 0
  2. 2026Calibration in One Go: Jointly Learning Accurate and Calibrated Neural Networkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Calibration-Aware Prompt Learning for Medical Vision-Language Models
  2. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
  3. 2025EchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Models
  4. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review
  5. 2024MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Models
  6. 2025On the Robustness of Medical Vision-Language Models: Are they Truly Generalizable?
  7. 2026Evaluating and Understanding Model Editing for Medical Vision Language Models
  8. 2025MedVision: Benchmarking Quantitative Medical Image Analysis