mlsanityData to 7 Oct 2026

Transfer Safety Awareness for Cross-Modal Safety Drift in Multimodal Large Language Models

arXiv 2609.02082 ↗First posted 2 Sep 2026cs.MM cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Visual modality enhances the capabilities of multimodal large language models (MLLMs) but also introduces a safety concern: a benign textual query may convey harmful intent when grounded in a visual image. We term this cross-modal safety drift and our pilot studies show that the safety response rate for such requests is substantially lower than that for requests containing explicitly unsafe text. This paper aims to systematically study this issue. First, we conduct an empirical analysis to identify representative unsafe response patterns. Building on these, we interpret model representations and attentions, revealing that visually risky cues receive limited attention and weakly trigger refusal. Motivated by the observation that safety signals from unsafe text processing can be transferred, we propose safety-awareness representation transfer (SRT), a lightweight direction-refinement method that mitigates cross-modal safety drift with a frozen MLLM backbone. Experiments across multiple benchmarks and models show that SRT effectively improves safety in diverse cross-modal settings while preserving utility. Code is available at https://github.com/cucu220123/safety-awareness.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 21 references that are in the map. First posted on arXiv 2 Sep 2026.

Builds on

17 of 21 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026Risk Awareness Injection: Calibrating Vision-Language Models for Safety without Compromising Utility3 Feb
  2. 2025Automating Steering for Safe Multimodal Large Language Models17 Jul
  3. 2025Understanding and Rectifying Safety Perception Distortion in VLMs18 Feb
  4. 2024VLSBench: Unveiling Visual Leakage in Multimodal Safety29 Nov
  5. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time9 Oct
  6. 2024CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibration17 Sep
  7. 2024Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Model21 Jun
  8. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation14 Mar
  9. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting14 Mar
  10. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov

Other references · 7

  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  2. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  3. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  4. 2023Steering Language Models With Activation Engineeringcited by 824
  5. 2026LatentLens: Revealing Highly Interpretable Visual Tokens in LLMscited by 9
  6. 2025VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safetycited by 7
  7. 2025Attention Misses Visual Risk: Risk-Adaptive Steering for Multimodal Safety Alignmentcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Can't See the Forest for the Trees: Benchmarking Multimodal Safety Awareness for Multimodal LLMs
  2. 2026CSR-Bench: A Benchmark for Evaluating the Cross-modal Safety and Reliability of MLLMs
  3. 2024Multimodal Situational Safety
  4. 2025Steering Multimodal Large Language Models Decoding for Context-Aware Safety
  5. 2026MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models
  6. 2026Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguards
  7. 2026MTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialogues
  8. 2025SafeMT: Multi-turn Safety for Multimodal Language Models