# Diagnosing and Repairing Unsafe Channels in Vision-Language Models via Causal Discovery and Dual-Modal Safety Subspace Projection

arXiv: https://arxiv.org/abs/2603.27240 · HTML: https://mlsanity.com/paper/2603.27240/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 29 of its 57 references that are in the map.
- First posted on arXiv 28 Mar 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)

## Builds on (most cited first, 29 of 29)

- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [Adversarial Machine Learning at Scale](https://mlsanity.com/paper/1611.01236/) (2016)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [Diffusion Models for Adversarial Purification](https://mlsanity.com/paper/2205.07460/) (2022)
- [MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models](https://mlsanity.com/paper/2311.17600/) (2023)
- [Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models](https://mlsanity.com/paper/2307.14539/) (2023)
- [OR-Bench: An Over-Refusal Benchmark for Large Language Models](https://mlsanity.com/paper/2405.20947/) (2024)
- [JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks](https://mlsanity.com/paper/2404.03027/) (2024)
- [Programming Refusal with Conditional Activation Steering](https://mlsanity.com/paper/2409.05907/) (2024)
- [A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity](https://mlsanity.com/paper/2401.01967/) (2024)
- [Jailbreaking Attack against Multimodal Large Language Model](https://mlsanity.com/paper/2402.02309/) (2024)
- [Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs](https://mlsanity.com/paper/2307.10490/) (2023)
- [JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Models](https://mlsanity.com/paper/2407.01599/) (2024)
- [Red Teaming Visual Language Models](https://mlsanity.com/paper/2401.12915/) (2024)
- [JailGuard: A Universal Detection Framework for LLM Prompt-based Attacks](https://mlsanity.com/paper/2312.10766/) (2023)
- [Eigenvalue and Generalized Eigenvalue Problems: Tutorial](https://mlsanity.com/paper/1903.11240/) (2019)
- [On the Trustworthiness of Generative Foundation Models: Guideline, Assessment, and Perspective](https://mlsanity.com/paper/2502.14296/) (2025)
- [Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicity](https://mlsanity.com/paper/2405.13967/) (2024)
- [Understanding and Rectifying Safety Perception Distortion in VLMs](https://mlsanity.com/paper/2502.13095/) (2025)
- [Universal Adversarial Attack on Aligned Multimodal LLMs](https://mlsanity.com/paper/2502.07987/) (2025)
- [Activation Steering Meets Preference Optimization: Defense Against Jailbreaks in Vision Language Models](https://mlsanity.com/paper/2509.00373/) (2025)
- [Why Pre-trained Models Fail: Feature Entanglement in Multi-modal Depression Detection](https://mlsanity.com/paper/2503.06620/) (2025)

## Built on by (most cited first, 2 of 2)

- [Learning to Edit Knowledge via Instruction-based Chain-of-Thought Prompting](https://mlsanity.com/paper/2604.05540/) (2026)
- [From Local Evidence to Safety Verdicts: Causal Tracing in Vision-Language Models](https://mlsanity.com/paper/2610.07514/) (2026)

## Abstract (arXiv)

Large Vision-Language Models (LVLMs) have achieved impressive performance across multimodal understanding and reasoning tasks, yet their internal safety mechanisms remain opaque and poorly controlled. In this work, we present a comprehensive framework for diagnosing and repairing unsafe channels within LVLMs (CARE). We first perform causal mediation analysis to identify neurons and layers that are causally responsible for unsafe behaviors. Based on these findings, we introduce a dual-modal safety subspace projection method that learns generalized safety subspaces for both visual and textual modalities through generalized eigen-decomposition between benign and malicious activations. During inference, activations are dynamically projected toward these safety subspaces via a hybrid fusion mechanism that adaptively balances visual and textual corrections, effectively suppressing unsafe features while preserving semantic fidelity. Extensive experiments on multiple safety benchmarks demonstrate that our causal-subspace repair framework significantly enhances safety robustness without degrading general multimodal capabilities, outperforming prior activation steering and alignment-based baselines. Additionally, our method exhibits good transferability, defending against unseen attacks.