# Perceive and Calibrate: Analyzing and Enhancing Robustness of Medical Multi-Modal Large Language Models

arXiv: https://arxiv.org/abs/2512.21964 · HTML: https://mlsanity.com/paper/2512.21964/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 31 of its 57 references that are in the map.
- First posted on arXiv 26 Dec 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 30 of 31)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering](https://mlsanity.com/paper/2102.09542/) (2021)
- [Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning](https://mlsanity.com/paper/2506.07044/) (2025)
- [OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM](https://mlsanity.com/paper/2402.09181/) (2024)
- [Efficient multi-prompt evaluation of LLMs](https://mlsanity.com/paper/2405.17202/) (2024)
- [Reducing Hallucinations in Vision-Language Models via Latent Space Steering](https://mlsanity.com/paper/2410.15778/) (2024)
- [Deep Learning Approaches for Data Augmentation in Medical Imaging: A Review](https://mlsanity.com/paper/2307.13125/) (2023)
- [Deep-neural-network based sinogram synthesis for sparse-view CT image reconstruction](https://mlsanity.com/paper/1803.00694/) (2018)
- [A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine](https://mlsanity.com/paper/2405.08603/) (2024)
- [A safety realignment framework via subspace-oriented model fusion for large language models](https://mlsanity.com/paper/2405.09055/) (2024)
- [Benchmarking Prompt Sensitivity in Large Language Models](https://mlsanity.com/paper/2502.06065/) (2025)
- [Improving the Robustness of Large Language Models via Consistency Alignment](https://mlsanity.com/paper/2403.14221/) (2024)
- [LMentry: A Language Model Benchmark of Elementary Language Tasks](https://mlsanity.com/paper/2211.02069/) (2022)
- [MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation](https://mlsanity.com/paper/2409.19684/) (2024)
- [Enhancing LLM Robustness to Perturbed Instructions: An Empirical Study](https://mlsanity.com/paper/2504.02733/) (2025)
- [Deformed2Self: Self-Supervised Denoising for Dynamic Medical Imaging](https://mlsanity.com/paper/2106.12175/) (2021)
- [Mitigating Boundary Ambiguity and Inherent Bias for Text Classification in the Era of Large Language Models](https://mlsanity.com/paper/2406.07001/) (2024)
- [Enhancing Robustness of LLM-Driven Multi-Agent Systems through Randomized Smoothing](https://mlsanity.com/paper/2507.04105/) (2025)
- [MedLA: A Logic-Driven Multi-Agent Framework for Complex Medical Reasoning with Large Language Models](https://mlsanity.com/paper/2509.23725/) (2025)
- [Real-Time Multimodal Cognitive Assistant for Emergency Medical Services](https://mlsanity.com/paper/2403.06734/) (2024)
- [Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports](https://mlsanity.com/paper/2407.05758/) (2024)
- [Enhancing the Capability and Robustness of Large Language Models through Reinforcement Learning-Driven Query Refinement](https://mlsanity.com/paper/2407.01461/) (2024)
- [MRI Banding Removal via Adversarial Training](https://mlsanity.com/paper/2001.08699/) (2020)
- [A task-specific deep-learning-based denoising approach for myocardial perfusion SPECT](https://mlsanity.com/paper/2303.00212/) (2023)
- [Neighboring Slice Noise2Noise: Self-Supervised Medical Image Denoising from Single Noisy Image Volume](https://mlsanity.com/paper/2411.10831/) (2024)
- [Robustness of Prompting: Enhancing Robustness of Large Language Models Against Prompting Attacks](https://mlsanity.com/paper/2506.03627/) (2025)
- [Unsupervised Training of a Dynamic Context-Aware Deep Denoising Framework for Low-Dose Fluoroscopic Imaging](https://mlsanity.com/paper/2411.00830/) (2024)
- [Robust image classification with multi-modal large language models](https://mlsanity.com/paper/2412.10353/) (2024)

## Built on by (most cited first, 2 of 2)

- [Med-Evo: Test-time Self-evolution for Medical Multimodal Large Language Models](https://mlsanity.com/paper/2603.07443/) (2026)
- [MedQ-Deg: A Multidimensional Benchmark for Evaluating MLLMs Across Medical Image Quality Degradations](https://mlsanity.com/paper/2603.07769/) (2026)

## Abstract (arXiv)

Medical Multi-modal Large Language Models (MLLMs) have shown promising clinical performance. However, their sensitivity to real-world input perturbations, such as imaging artifacts and textual errors, critically undermines their clinical applicability. Systematic analysis of such noise impact on medical MLLMs remains largely unexplored. Furthermore, while several works have investigated the MLLMs' robustness in general domains, they primarily focus on text modality and rely on costly fine-tuning. They are inadequate to address the complex noise patterns and fulfill the strict safety standards in medicine. To bridge this gap, this work systematically analyzes the impact of various perturbations on medical MLLMs across both visual and textual modalities. Building on our findings, we introduce a training-free Inherent-enhanced Multi-modal Calibration (IMC) framework that leverages MLLMs' inherent denoising capabilities following the perceive-and-calibrate principle for cross-modal robustness enhancement. For the visual modality, we propose a Perturbation-aware Denoising Calibration (PDC) which leverages MLLMs' own vision encoder to identify noise patterns and perform prototype-guided feature calibration. For text denoising, we design a Self-instantiated Multi-agent System (SMS) that exploits the MLLMs' self-assessment capabilities to refine noisy text through a cooperative hierarchy of agents. We construct a benchmark containing 11 types of noise across both image and text modalities on 2 datasets. Experimental results demonstrate our method achieves the state-of-the-art performance across multiple modalities, showing potential to enhance MLLMs' robustness in real clinical scenarios.