mlsanityData to 7 Oct 2026

VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gap

arXiv 2502.10486 ↗First posted 14 Feb 2025cs.CR cs.AI cs.CVcited by 16

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 2 more
Abstract (arXiv)

The emergence of vision language models (VLMs) comes with increased safety concerns, as the incorporation of multiple modalities heightens vulnerability to attacks. Although VLMs can be built upon LLMs that have textual safety alignment, it is easily undermined when the vision modality is integrated. We attribute this safety challenge to the modality gap, a separation of image and text in the shared representation space, which blurs the distinction between harmful and harmless queries that is evident in LLMs but weakened in VLMs. To avoid safety decay and fulfill the safety alignment gap, we propose VLM-Guard, an inference-time intervention strategy that leverages the LLM component of a VLM as supervision for the safety alignment of the VLM. VLM-Guard projects the representations of VLM into the subspace that is orthogonal to the safety steering direction that is extracted from the safety-aligned LLM. Experimental results on three malicious instruction settings show the effectiveness of VLM-Guard in safeguarding VLM and fulfilling the safety alignment gap between VLM and its LLM component.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 33 references that are in the map. First posted on arXiv 14 Feb 2025.

Builds on

14 of 33 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation14 Mar
  2. 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models21 Feb
  3. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  4. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun

Other references · 10

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  2. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  3. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  4. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 165
  5. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 156
  6. 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 128
  7. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 99
  8. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidancecited by 78
  9. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Modelscited by 55
  10. 2024Debiasing Multimodal Large Language Models via Penalization of Language Priorscited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 11.

16 papers.

Sort:

By area:

  1. 2025Safety in Large Reasoning Models: A Surveycited by 58
  2. 2025GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoningcited by 38
  3. 2025EcoAlign: An Economically Rational Framework for Efficient LVLM Alignmentcited by 5
  4. 2025CrossGuard: Safeguarding MLLMs against Joint-Modal Implicit Malicious Attackscited by 4
  5. 2026WARD: Adversarially Robust Defense of Web Agents Against Prompt Injectionscited by 3
  6. 2025A Survey on Training-free Alignment of Large Language Modelscited by 2
  7. 2026Jailbreaks on Vision Language Model via Multimodal Reasoningcited by 2
  8. 2026Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Imagescited by 2
  9. 2026Reading Between the Pixels: Linking Text-Image Embedding Alignment to Typographic Attack Success on Vision-Language Modelscited by 1
  10. 2026Sparse Autoencoders as Plug-and-Play Firewalls for Adversarial Attack Detection in VLMscited by 1
  11. 2026Safety Geometry Collapse in Multimodal LLMs and Adaptive Drift Correctioncited by 1
  12. 2026SALLIE: Generation-Free Hidden-State Detection of Jailbreaks and Prompt Injections Across Text and Visioncited by 0
4 more
  1. 2026V-DEAL: Diagnosing Video Safety De-Calibration as an Understanding-Refusal Coupling Failurecited by 0
  2. 2026A Cross-Architecture Audit of Direction-Based Inference-Time Defences in Vision-Language Modelscited by 0
  3. 2026SPARK: Representation-Level KV Memory Alignment for Safer Vision-Language Modelscited by 0
  4. 2026SafeMol: Dual-Modality Safety Alignment for Molecular Multimodal Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VLMGuard-R1: Proactive Safety Alignment for VLMs via Reasoning-Driven Prompt Optimization
  2. 2024Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models
  3. 2024PSA-VLM: Enhancing Vision-Language Model Safety through Progressive Concept-Bottleneck-Driven Alignment
  4. 2026GuardAlign: Test-time Safety Alignment in Multimodal Large Language Models
  5. 2025Security Tensors as a Cross-Modal Bridge: Extending Text-Aligned Safety to Vision in LVLM
  6. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models
  7. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time
  8. 2024Safety Alignment for Vision Language Models