mlsanityData to 14 Oct 2026

SafeSteer: Adaptive Subspace Steering for Efficient Jailbreak Defense in Vision-Language Models

arXiv 2509.21400 ↗First posted 24 Sep 2025cs.CRcited by 0

On the map: LLM safety and security → Vision-language and text-to-image safety

Abstract (arXiv)

As the capabilities of Vision Language Models (VLMs) continue to improve, they are increasingly targeted by jailbreak attacks. Existing defense methods face two major limitations: (1) they struggle to ensure safety without compromising the model's utility; and (2) many defense mechanisms significantly reduce the model's inference efficiency. To address these challenges, we propose SafeSteer, a lightweight, inference-time steering framework that effectively defends against diverse jailbreak attacks without modifying model weights. At the core of SafeSteer is the innovative use of Singular Value Decomposition to construct a low-dimensional "safety subspace." By projecting and reconstructing the raw steering vector into this subspace during inference, SafeSteer adaptively removes harmful generation signals while preserving the model's ability to handle benign inputs. The entire process is executed in a single inference pass, introducing negligible overhead. Extensive experiments show that SafeSteer reduces the attack success rate by over 60% and improves accuracy on normal tasks by 1-2%, without introducing significant inference latency. These results demonstrate that robust and practical jailbreak defense can be achieved through simple, efficient inference-time control.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Vision-language and text-to-image safety, one of 12 directions in LLM safety and security. Builds on 25 of its 44 references that are in the map. First posted on arXiv 24 Sep 2025.

Builds on

25 of 44 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 202514 Jun
  2. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injection19 Apr
  3. 2025Jailbreaking Multimodal Large Language Models via Shuffle Inconsistency9 Jan
  4. 2024Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks23 Nov
  5. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt6 Jun
  6. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  7. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun

Other references · 18

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  7. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  8. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  9. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  10. 2022Relative representations enable zero-shot latent space communicationcited by 191
  11. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 175
  12. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 124
6 more
  1. 2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learningcited by 115
  2. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidancecited by 83
  3. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Timecited by 60
  4. 2023X-Adv: Physical Adversarial Object Attacks against X-ray Prohibited Item Detectioncited by 59
  5. 2024SCANS: Mitigating the Exaggerated Safety for LLMs via Safety-Conscious Activation Steeringcited by 50
  6. 2024Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shiftcited by 49

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Principled Steering via Null-space Projection for Jailbreak Defense in Vision-Language Models
  2. 2025TAIJI: Textual Anchoring for Immunizing Jailbreak Images in Vision Language Models
  3. 2025JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Models
  4. 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden States
  5. 2026Jailbreaking the Matrix: Nullspace Steering for Controlled Model Subversion
  6. 2025Dynamic Token Reweighting for Robust Vision-Language Models
  7. 2025JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language Model
  8. 2025Simulated Ensemble Attack: Transferring Jailbreaks Across Fine-tuned Vision-Language Models