mlsanityData to 14 Oct 2026

Security Tensors as a Cross-Modal Bridge: Extending Text-Aligned Safety to Vision in LVLM

arXiv 2507.20994 ↗First posted 28 Jul 2025cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Large visual-language models (LVLMs) integrate aligned large language models (LLMs) with visual modules to process multimodal inputs. However, the safety mechanisms developed for text-based LLMs do not naturally extend to visual modalities, leaving LVLMs vulnerable to harmful image inputs. To address this cross-modal safety gap, we introduce security tensors - trainable input vectors applied during inference through either the textual or visual modality. These tensors transfer textual safety alignment to visual processing without modifying the model's parameters. They are optimized using a curated dataset containing (i) malicious image-text pairs requiring rejection, (ii) contrastive benign pairs with text structurally similar to malicious queries, with the purpose of being contrastive examples to guide visual reliance, and (iii) general benign samples preserving model functionality. Experimental results demonstrate that both textual and visual security tensors significantly enhance LVLMs' ability to reject diverse harmful visual inputs while maintaining near-identical performance on benign tasks. Further internal analysis towards hidden-layer representations reveals that security tensors successfully activate the language module's textual "safety layers" in visual inputs, thereby effectively extending text-based safety to the visual modality.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 27 references that are in the map. First posted on arXiv 28 Jul 2025.

Builds on

17 of 27 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Images23 Feb
  2. 2025Spot Risks Before Speaking! Unraveling Safety Attention Heads in Large Vision-Language Models3 Jan
  3. 2024Cross-Modal Safety Mechanism Transfer in Large Vision-Language Models16 Oct
  4. 2024How Does Vision-Language Adaptation Impact the Safety of Vision Language Models?10 Oct
  5. 2024Break the Visual Perception: Adversarial Attacks Targeting Encoded Visual Tokens of Large Vision-Language Models9 Oct
  6. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt6 Jun

Other references · 11

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  6. 2022Teaching Small Language Models to Reasoncited by 284
  7. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 167
  8. 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 145
  9. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 134
  10. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 103
  11. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvementcited by 74

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gap
  2. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time
  3. 2026Jailbreaking Vision-Language Models Through the Visual Modality
  4. 2025DAVSP: Safety Alignment for Large Vision-Language Models via Deep Aligned Visual Safety Prompt
  5. 2025Reimagining Safety Alignment with An Image
  6. 2026Beyond Visual Safety: Jailbreaking Multimodal Large Language Models for Harmful Image Generation via Semantic-Agnostic Inputs
  7. 2025VisuoAlign: Safety Alignment of LVLMs with Multimodal Tree Search
  8. 2026Risk Awareness Injection: Calibrating Vision-Language Models for Safety without Compromising Utility