mlsanityData to 14 Oct 2026

Automating Steering for Safe Multimodal Large Language Models

arXiv 2507.13255 ↗First posted 17 Jul 2025cs.CL cs.AI cs.IRcited by 17

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Recent progress in Multimodal Large Language Models (MLLMs) has unlocked powerful cross-modal reasoning abilities, but also raised new safety concerns, particularly when faced with adversarial multimodal inputs. To improve the safety of MLLMs during inference, we introduce a modular and adaptive inference-time intervention technology, AutoSteer, without requiring any fine-tuning of the underlying model. AutoSteer incorporates three core components: (1) a novel Safety Awareness Score (SAS) that automatically identifies the most safety-relevant distinctions among the model's internal layers; (2) an adaptive safety prober trained to estimate the likelihood of toxic outputs from intermediate representations; and (3) a lightweight Refusal Head that selectively intervenes to modulate generation when safety risks are detected. Experiments on LLaVA-OV and Chameleon across diverse safety-critical benchmarks demonstrate that AutoSteer significantly reduces the Attack Success Rate (ASR) for textual, visual, and cross-modal threats, while maintaining general abilities. These findings position AutoSteer as a practical, interpretable, and effective framework for safer deployment of multimodal AI systems.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 34 of its 47 references that are in the map. First posted on arXiv 17 Jul 2025.

Builds on

34 of 47 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment27 Nov
  2. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation14 Mar

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  8. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  9. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  10. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  11. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  12. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
20 more
  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  2. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  3. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  4. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  5. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 378
  6. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  7. 2023Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentationcited by 147
  8. 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 113
  9. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 103
  10. 2025Tell me about yourself: LLMs are aware of their learned behaviorscited by 91
  11. 2023Controlled Text Generation with Natural Language Instructionscited by 87
  12. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidancecited by 83
  13. 2023Word Embeddings Are Steers for Language Modelscited by 70
  14. 2020PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generationcited by 35
  15. 2024CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibrationcited by 31
  16. 2024Self-DC: When to Reason and When to Act? Self Divide-and-Conquer for Compositional Unknown Questionscited by 30
  17. 2023ToViLaG: Your Visual-Language Generative Model is Also An Evildoercited by 27
  18. 2021Deep Extrapolation for Attribute-Enhanced Generationcited by 18
  19. 2020Technical Report: Auxiliary Tuning and its Application to Conditional Text Generationcited by 17
  20. 2022Efficient Reinforcement Learning for Unsupervised Controlled Text Generationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 13.

17 papers.

Sort:

By area:

  1. 2026Accurate Failure Prediction in Agents Does Not Imply Effective Failure Preventioncited by 15
  2. 2025MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steeringcited by 10
  3. 2026ICON: Indirect Prompt Injection Defense for Agents based on Inference-Time Correctioncited by 10
  4. 2026EmoShift: Lightweight Activation Steering for Enhanced Emotion-Aware Speech Synthesiscited by 7
  5. 2026Safety Alignment as Continual Learning: Mitigating the Alignment Tax via Orthogonal Gradient Projectioncited by 6
  6. 2025The Anatomy of Alignment: Decomposing Preference Optimization by Steering Sparse Featurescited by 5
  7. 2025Attention Misses Visual Risk: Risk-Adaptive Steering for Multimodal Safety Alignmentcited by 4
  8. 2025Towards Reasoning-Preserving Unlearning in Multimodal Large Language Modelscited by 4
  9. 2026AMPS: Adaptive Modality Preference Steering via Functional Entropycited by 4
  10. 2026Risk Awareness Injection: Calibrating Vision-Language Models for Safety without Compromising Utilitycited by 2
  11. 2026Principled Steering via Null-space Projection for Jailbreak Defense in Vision-Language Modelscited by 2
  12. 2026RoCo-ACE: Rollout-Conditioned Online Distillation for Retention-Aware Knowledge Injectioncited by 0
5 more
  1. 2026Do Unified Multimodal Models Think in One Space? A Lens Through Cross-Branch Steeringcited by 0
  2. 2026ReFrame: Evidence-Guided Test-Time Safety Alignment in Multimodal Large Language Modelscited by 0
  3. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0
  4. 2026Transfer Safety Awareness for Cross-Modal Safety Drift in Multimodal Large Language Modelscited by 0
  5. 2026SPARK: Representation-Level KV Memory Alignment for Safer Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025When Safe Unimodal Inputs Collide: Optimizing Reasoning Chains for Cross-Modal Safety in Multimodal Large Language Models
  2. 2026Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguards
  3. 2026SafeSteer: A Decoding-level Defense Mechanism for Multimodal Large Language Models
  4. 2025DREAM: Disentangling Risks to Enhance Safety Alignment in Multimodal Large Language Models
  5. 2024RapGuard: Safeguarding Multimodal Large Language Models via Rationale-aware Defensive Prompting
  6. 2026Yuvion VL: A Multimodal Foundation Model for Adversarial Content and AI Safety
  7. 2026MME-Safety: A Fine-grained Benchmark for Safety Evaluation of MLLMs
  8. 2026OOD-MMSafe: Advancing MLLM Safety from Harmful Intent to Hidden Consequences