mlsanityData to 7 Oct 2026

Steering to Say No: Configurable Refusal via Activation Steering in Vision Language Models

arXiv 2602.07013 ↗First posted 31 Jan 2026cs.CV cs.AI cs.LGcited by 3

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

With the rapid advancement of Vision Language Models (VLMs), refusal mechanisms have become a critical component for ensuring responsible and safe model behavior. However, existing refusal strategies are largely textitone-size-fits-all and fail to adapt to diverse user needs and contextual constraints, leading to either under-refusal or over-refusal. In this work, we firstly explore the challenges mentioned above and develop Configurable Refusal in VLMs (CR-VLM), a robust and efficient approach for em configurable refusal based on activation steering. CR-VLM consists of three integrated components: (1) extracting a configurable refusal vector via a teacher-forced mechanism to amplify the refusal signal; (2) introducing a gating mechanism that mitigates over-refusal by preserving acceptance for in-scope queries; and (3) designing a counterfactual vision enhancement module that aligns visual representations with refusal requirements. Comprehensive experiments across multiple datasets and various VLMs demonstrate that CR-VLM achieves effective, efficient, and robust configurable refusals, offering a scalable path toward user-adaptive safety alignment in VLMs.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 46 references that are in the map. First posted on arXiv 31 Jan 2026.

Builds on

40 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Refusing Safe Prompts for Multi-modal Large Language Models12 Jul
  2. 2024Safety Alignment for Vision Language Models22 May

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  9. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  10. 2023A Survey on Multimodal Large Language Modelscited by 898
  11. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  12. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
26 more
  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  3. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  4. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  5. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  6. 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 303
  7. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  8. 2024Programming Refusal with Conditional Activation Steeringcited by 191
  9. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  10. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  11. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  12. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 105
  13. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidancecited by 78
  14. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Contentcited by 65
  15. 2024Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablationcited by 48
  16. 2024SCANS: Mitigating the Exaggerated Safety for LLMs via Safety-Conscious Activation Steeringcited by 45
  17. 2025Refusal Direction is Universal Across Safety-Aligned Languagescited by 37
  18. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 35
  19. 2025A Survey on Large Language Model Benchmarkscited by 32
  20. 2025Just Enough Shifts: Mitigating Over-Refusal in Aligned Language Models with Targeted Representation Fine-Tuningcited by 22
  21. 2025Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)cited by 19
  22. 2025Logic Jailbreak: Efficiently Unlocking LLM Safety Restrictions Through Formal Logical Expressioncited by 16
  23. 2023Adventures of Trustworthy Vision-Language Models: A Surveycited by 10
  24. 2025SteerVLM: Robust Model Control through Lightweight Activation Steering for Vision Language Modelscited by 8
  25. 2025OffTopicEval: When Large Language Models Enter the Wrong Chat, Almost Always!cited by 4
  26. 2025ManagerBench: Evaluating the Safety-Pragmatism Trade-off in Autonomous LLMscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Safety Geometry Collapse in Multimodal LLMs and Adaptive Drift Correctioncited by 1
  2. 2026When Think-with-Image Meets Safety: What Determines Multimodal Jailbreak Robustness?cited by 0
  3. 2026When Safety Overrides Vision: Exploring Dynamics between Vision Influence and Safety Alignment in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DUAL-Bench: Measuring Over-Refusal and Robustness in Vision-Language Models
  2. 2026Principled Steering via Null-space Projection for Jailbreak Defense in Vision-Language Models
  3. 2026On the Steering Dimensionality of Refusal in Language Models
  4. 2026From Refusal Tokens to Refusal Control: Discovering and Steering Category-Specific Refusal Directions
  5. 2025Activation Steering Meets Preference Optimization: Defense Against Jailbreaks in Vision Language Models
  6. 2025VISOR: Visual Input-based Steering for Output Redirection in Vision-Language Models
  7. 2026Surgical Refusal Ablation: Disentangling Safety from Intelligence via Concept-Guided Spectral Cleaning
  8. 2025SARSteer: Safeguarding Large Audio-Language Models via Safe-Ablated Refusal Steering