mlsanityData to 14 Oct 2026

Understanding and Rectifying Safety Perception Distortion in VLMs

arXiv 2502.13095 ↗First posted 18 Feb 2025cs.CV cs.CL cs.LGcited by 17

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Recent studies reveal that vision-language models (VLMs) become more susceptible to harmful requests and jailbreak attacks after integrating the vision modality, exhibiting greater vulnerability than their text-only LLM backbones. To uncover the root cause of this phenomenon, we conduct an in-depth analysis and identify a key issue: multimodal inputs introduce an modality-induced activation shift toward a "safer" direction compared to their text-only counterparts, leading VLMs to systematically overestimate the safety of harmful inputs. We refer to this issue as safety perception distortion. To mitigate such distortion, we propose Activation Shift Disentanglement and Calibration (ShiftDC), a training-free method that decomposes and calibrates the modality-induced activation shift to reduce the impact of modality on safety. By isolating and removing the safety-relevant component, ShiftDC restores the inherent safety alignment of the LLM backbone while preserving the vision-language capabilities of VLMs. Empirical results demonstrate that ShiftDC significantly enhances alignment performance on safety benchmarks without impairing model utility.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 53 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

46 of 53 references.

Closest ancestors · 18

Same direction, newest first

  1. 2024Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks23 Nov
  2. 2024UniGuard: Towards Universal Safety Guardrails for Jailbreak Attacks on Multimodal Large Language Models3 Nov
  3. 2024BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attacks28 Oct
  4. 2024Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models11 Oct
  5. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time9 Oct
  6. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Model17 Jun
  7. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks3 Apr
  8. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models14 Mar
  9. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation14 Mar
  10. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting14 Mar
  11. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance5 Jan
  12. 2023OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimization7 Dec
6 more
  1. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models29 Nov
  2. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  3. 2023How Robust is Google's Bard to Adversarial Image Attacks?21 Sep
  4. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul
  5. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun
  6. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May

Other references · 28

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  7. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  8. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  9. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  10. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  11. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  12. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
16 more
  1. 2023Steering Language Models With Activation Engineeringcited by 922
  2. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  3. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  4. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  5. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  6. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 185
  7. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 167
  8. 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 135
  9. 2024Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimizationcited by 118
  10. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 103
  11. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidancecited by 83
  12. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editingcited by 79
  13. 2024On the Role of Attention Heads in Large Language Model Safetycited by 72
  14. 2024Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicitycited by 35
  15. 2024The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defensecited by 21
  16. 2025DELMAN: Dynamic Defense Against Large Language Model Jailbreaking with Model Editingcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 13.

17 papers.

Sort:

By area:

  1. 2025Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Modelscited by 18
  2. 2025Activation Steering Meets Preference Optimization: Defense Against Jailbreaks in Vision Language Modelscited by 5
  3. 2026Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Imagescited by 3
  4. 2026Risk Awareness Injection: Calibrating Vision-Language Models for Safety without Compromising Utilitycited by 2
  5. 2026Diagnosing and Repairing Unsafe Channels in Vision-Language Models via Causal Discovery and Dual-Modal Safety Subspace Projectioncited by 2
  6. 2025Understanding and Mitigating Toxicity in Image-Text Pretraining Datasets: A Case Study on LLaVAcited by 1
  7. 2025Dynamic Token Reweighting for Robust Vision-Language Modelscited by 1
  8. 2026GuardAlign: Test-time Safety Alignment in Multimodal Large Language Modelscited by 1
  9. 2026Two Birds, One Projection: Harmonizing Safety and Utility in LVLMs via Inference-time Feature Projectioncited by 1
  10. 2026Safety Geometry Collapse in Multimodal LLMs and Adaptive Drift Correctioncited by 1
  11. 2026SafeCap: Improving LVLM Safety with Image Captioning Reinforcement Learningcited by 1
  12. 2026Directional Embedding Smoothing for Robust Vision Language Modelscited by 0
5 more
  1. 2026A Cross-Architecture Audit of Direction-Based Inference-Time Defences in Vision-Language Modelscited by 0
  2. 2026SafeNexus: Discovering and Steering Modality-Universal Safety Neurons in MLLMscited by 0
  3. 2026Do VLMs Share Safety Neurons Across Modalities?cited by 0
  4. 2026Transfer Safety Awareness for Cross-Modal Safety Drift in Multimodal Large Language Modelscited by 0
  5. 2026The Uncontrolled Variable: Vision-Language Model Refusal Responds to Image Presence in Ways Risk Cannot Explaincited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gap
  2. 2025Bootstrapping LLM Robustness for VLM Safety via Reducing the Pretraining Modality Gap
  3. 2024Safety Alignment for Vision Language Models
  4. 2025VisuoAlign: Safety Alignment of LVLMs with Multimodal Tree Search
  5. 2025The Safety Reminder: A Soft Prompt to Reactivate Delayed Safety Awareness in Vision-Language Models
  6. 2024Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Model
  7. 2024PSA-VLM: Enhancing Vision-Language Model Safety through Progressive Concept-Bottleneck-Driven Alignment
  8. 2025SIA: Enhancing Safety via Intent Awareness for Vision-Language Models