mlsanityData to 7 Oct 2026

The Forgotten Shield: Safety Grafting in Parameter-Space for Medical MLLMs

arXiv 2601.04199 ↗First posted 5 Dec 2025cs.LG cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Medical Multimodal Large Language Models (Medical MLLMs) have achieved remarkable progress in specialized medical tasks; however, research into their safety has lagged, posing potential risks for real-world deployment. In this paper, we first establish a multidimensional evaluation framework to systematically benchmark the safety of current SOTA Medical MLLMs. Our empirical analysis reveals pervasive vulnerabilities across both general and medical-specific safety dimensions in existing models, particularly highlighting their fragility against cross-modality jailbreak attacks. Furthermore, we find that the medical fine-tuning process frequently induces catastrophic forgetting of the model's original safety alignment. To address this challenge, we propose a novel "Parameter-Space Intervention" approach for efficient safety re-alignment. This method extracts intrinsic safety knowledge representations from original base models and concurrently injects them into the target model during the construction of medical capabilities. Additionally, we design a fine-grained parameter search algorithm to achieve an optimal trade-off between safety and medical performance. Experimental results demonstrate that our approach significantly bolsters the safety guardrails of Medical MLLMs without relying on additional domain-specific safety data, while minimizing degradation to core medical performance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 27 references that are in the map. First posted on arXiv 5 Dec 2025.

Builds on

25 of 27 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Medical MLLM is Vulnerable: Cross-Modality Jailbreak and Mismatched Attacks on Medical Multimodal Large Language Models26 May
  2. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models29 Nov
  3. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov

Other references · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2025Qwen2.5-VL Technical Reportcited by 5,418
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  11. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  12. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
10 more
  1. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  2. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scalecited by 194
  3. 2025Qwen3Guard Technical Reportcited by 125
  4. 2025HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptationcited by 112
  5. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 105
  6. 2025Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understandingcited by 96
  7. 2023HuatuoGPT-II, One-stage Training for Medical Adaption of LLMscited by 91
  8. 2024MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Modelscited by 43
  9. 2025CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMscited by 26
  10. 2024Safeguard Fine-Tuned LLMs Through Pre- and Post-Tuning Model Mergingcited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine
  2. 2026When Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical Queries
  3. 2025Black-Box Behavioral Distillation Breaks Safety Alignment in Medical LLMs
  4. 2026MME-Safety: A Fine-grained Benchmark for Safety Evaluation of MLLMs
  5. 2025Towards Safe AI Clinicians: A Comprehensive Study on Large Language Model Jailbreaking in Healthcare
  6. 2025Benchmarking Chinese Medical LLMs: A Medbench-based Analysis of Performance Gaps and Hierarchical Optimization Strategies
  7. 2025SafeMLRM: Demystifying Safety in Multi-modal Large Reasoning Models
  8. 2026Neurosymbolic Alignment for Physiologically-Safe Clinical Language Models