mlsanityData to 14 Oct 2026

Perceive and Calibrate: Analyzing and Enhancing Robustness of Medical Multi-Modal Large Language Models

arXiv 2512.21964 ↗First posted 26 Dec 2025cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Medical Multi-modal Large Language Models (MLLMs) have shown promising clinical performance. However, their sensitivity to real-world input perturbations, such as imaging artifacts and textual errors, critically undermines their clinical applicability. Systematic analysis of such noise impact on medical MLLMs remains largely unexplored. Furthermore, while several works have investigated the MLLMs' robustness in general domains, they primarily focus on text modality and rely on costly fine-tuning. They are inadequate to address the complex noise patterns and fulfill the strict safety standards in medicine. To bridge this gap, this work systematically analyzes the impact of various perturbations on medical MLLMs across both visual and textual modalities. Building on our findings, we introduce a training-free Inherent-enhanced Multi-modal Calibration (IMC) framework that leverages MLLMs' inherent denoising capabilities following the perceive-and-calibrate principle for cross-modal robustness enhancement. For the visual modality, we propose a Perturbation-aware Denoising Calibration (PDC) which leverages MLLMs' own vision encoder to identify noise patterns and perform prototype-guided feature calibration. For text denoising, we design a Self-instantiated Multi-agent System (SMS) that exploits the MLLMs' self-assessment capabilities to refine noisy text through a cooperative hierarchy of agents. We construct a benchmark containing 11 types of noise across both image and text modalities on 2 datasets. Experimental results demonstrate our method achieves the state-of-the-art performance across multiple modalities, showing potential to enhance MLLMs' robustness in real clinical scenarios.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 57 references that are in the map. First posted on arXiv 26 Dec 2025.

Builds on

31 of 57 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning8 Jun
  2. 2024MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation29 Sep
  3. 2024Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports8 Jul
  4. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine14 May
  5. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM14 Feb

Other references · 26

  1. 2025Qwen2.5-VL Technical Reportcited by 5,587
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  4. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  5. 2024Efficient multi-prompt evaluation of LLMscited by 73
  6. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steeringcited by 59
  7. 2023Deep Learning Approaches for Data Augmentation in Medical Imaging: A Reviewcited by 48
  8. 2018Deep-neural-network based sinogram synthesis for sparse-view CT image reconstructioncited by 47
  9. 2024A safety realignment framework via subspace-oriented model fusion for large language modelscited by 41
  10. 2025Benchmarking Prompt Sensitivity in Large Language Modelscited by 38
  11. 2024Improving the Robustness of Large Language Models via Consistency Alignmentcited by 36
  12. 2022LMentry: A Language Model Benchmark of Elementary Language Taskscited by 22
14 more
  1. 2025Enhancing LLM Robustness to Perturbed Instructions: An Empirical Studycited by 17
  2. 2021Deformed2Self: Self-Supervised Denoising for Dynamic Medical Imagingcited by 15
  3. 2024Mitigating Boundary Ambiguity and Inherent Bias for Text Classification in the Era of Large Language Modelscited by 9
  4. 2025Enhancing Robustness of LLM-Driven Multi-Agent Systems through Randomized Smoothingcited by 9
  5. 2025MedLA: A Logic-Driven Multi-Agent Framework for Complex Medical Reasoning with Large Language Modelscited by 9
  6. 2024Real-Time Multimodal Cognitive Assistant for Emergency Medical Servicescited by 8
  7. 2024Enhancing the Capability and Robustness of Large Language Models through Reinforcement Learning-Driven Query Refinementcited by 5
  8. 2020MRI Banding Removal via Adversarial Trainingcited by 4
  9. 2023A task-specific deep-learning-based denoising approach for myocardial perfusion SPECTcited by 4
  10. 2024Neighboring Slice Noise2Noise: Self-Supervised Medical Image Denoising from Single Noisy Image Volumecited by 3
  11. 2025Robustness of Prompting: Enhancing Robustness of Large Language Models Against Prompting Attackscited by 3
  12. 2024Unsupervised Training of a Dynamic Context-Aware Deep Denoising Framework for Low-Dose Fluoroscopic Imagingcited by 2
  13. 2024Robust image classification with multi-modal large language modelscited by 2
  14. 2025Certifying Language Model Robustness with Fuzzed Randomized Smoothing: An Efficient Defense Against Backdoor Attackscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Med-Evo: Test-time Self-evolution for Medical Multimodal Large Language Modelscited by 0
  2. 2026MedQ-Deg: A Multidimensional Benchmark for Evaluating MLLMs Across Medical Image Quality Degradationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025On the Robustness of Medical Vision-Language Models: Are they Truly Generalizable?
  2. 2026Just how sure are you? Improving Verbalized Uncertainty Calibration in Medical VQA
  3. 2025On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AI
  4. 2025EchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Models
  5. 2025Calibration-Aware Prompt Learning for Medical Vision-Language Models
  6. 2024MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Models
  7. 2025Analysis of Image-and-Text Uncertainty Propagation in Multimodal Large Language Models with Cardiac MR-Based Applications
  8. 2026Lost in the Hype: Revealing and Dissecting the Performance Degradation of Medical Multimodal Large Language Models in Image Classification