mlsanityData to 14 Oct 2026

Learning to Steer: Input-dependent Steering for Multimodal LLMs

arXiv 2508.12815 ↗First posted 18 Aug 2025cs.LG cs.AI cs.CLcited by 15

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Steering has emerged as a practical approach to enable post-hoc guidance of LLMs towards enforcing a specific behavior. However, it remains largely underexplored for multimodal LLMs (MLLMs); furthermore, existing steering techniques, such as mean steering, rely on a single steering vector, applied independently of the input query. This paradigm faces limitations when the desired behavior is dependent on the example at hand. For example, a safe answer may consist in abstaining from answering when asked for an illegal activity, or may point to external resources or consultation with an expert when asked about medical advice. In this paper, we investigate a fine-grained steering that uses an input-specific linear shift. This shift is computed using contrastive input-specific prompting. However, the input-specific prompts required for this approach are not known at test time. Therefore, we propose to train a small auxiliary module to predict the input-specific steering vector. Our approach, dubbed as L2S (Learn-to-Steer), demonstrates that it reduces hallucinations and enforces safety in MLLMs, outperforming other static baselines. Our code is publicly available at https://jayneelparekh.github.io/learn-to-steer/

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 64 references that are in the map. First posted on arXiv 18 Aug 2025.

Builds on

55 of 64 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Analyzing Finetuning Representation Shift for Multimodal LLMs Steering6 Jan
  2. 2024Skipping Computations in Multimodal LLMs12 Oct
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  4. 2024A Concept-Based Explainability Framework for Large Multimodal Models12 Jun
  5. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  6. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning1 Oct
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep

Other references · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  11. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
36 more
  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  2. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  3. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  4. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  5. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  6. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  8. 2018Object Hallucination in Image Captioningcited by 669
  9. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  10. 2020InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANscited by 394
  11. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  12. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  13. 2024What matters when building vision-language models?cited by 326
  14. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 316
  15. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 274
  16. 2023Detecting and Preventing Hallucinations in Large Vision Language Modelscited by 256
  17. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 226
  18. 2024Programming Refusal with Conditional Activation Steeringcited by 206
  19. 2024ReFT: Representation Finetuning for Language Modelscited by 197
  20. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 167
  21. 2024Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversationscited by 129
  22. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 125
  23. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 124
  24. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinationscited by 104
  25. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revisioncited by 98
  26. 2024Visual Hallucinations of Multi-modal Large Language Modelscited by 68
  27. 2024Red Teaming Visual Language Modelscited by 65
  28. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steeringcited by 59
  29. 2025Scaling Laws for Native Multimodal Modelscited by 50
  30. 2024Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreakscited by 48
  31. 2022MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Promptingcited by 41
  32. 2024Extending Activation Steering to Broad Skills and Multiple Behaviourscited by 40
  33. 2023eP-ALM: Efficient Perceptual Augmentation of Language Modelscited by 34
  34. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 23
  35. 2024Improved Baselines for Data-efficient Perceptual Augmentation of LLMscited by 18
  36. 2025Internal Activation Revision: Safeguarding Vision Language Models Without Parameter Updatecited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 12.

15 papers.

Sort:

By area:

  1. 2025MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steeringcited by 10
  2. 2026CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steeringcited by 9
  3. 2025Hallucination reduction with CASAL: Contrastive Activation Steering For Amortized Learningcited by 6
  4. 2025Towards Reasoning-Preserving Unlearning in Multimodal Large Language Modelscited by 4
  5. 2026AMPS: Adaptive Modality Preference Steering via Functional Entropycited by 4
  6. 2026SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectorscited by 4
  7. 2026Where to Steer: Input-Dependent Layer Selection for Steering Improves LLM Alignmentcited by 3
  8. 2026Deployable Per-Instance Multi-Layer Activation Steering for Large Language Modelscited by 3
  9. 2026When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMscited by 2
  10. 2026One-shot Optimized Steering Vector for Hallucination Mitigation for VLMscited by 1
  11. 2026DiMaS: Distribution Matching for Steering Vision-Language-Action Modelscited by 1
  12. 2026Do Unified Multimodal Models Think in One Space? A Lens Through Cross-Branch Steeringcited by 0
3 more
  1. 2026Absorbed in Inertia: Activation Analysis for Computer-Use Agentscited by 0
  2. 2026TRACESTEER: Trace Driven Activation Steering for Hallucination Mitigation in Diffusion Language Modelscited by 0
  3. 2026On the Limits of Representation-Based Test-Time Adaptation for Multimodal LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Does Steering Break Your Model? A Multi-Dimensional Evaluation Suite for LLM Steering Methods
  2. 2025Guiding Giants: Lightweight Controllers for Weighted Activation Steering in LLMs
  3. 2025Effectively Steer LLM To Follow Preference via Building Confident Directions
  4. 2025One-shot Optimized Steering Vectors Mediate Safety-relevant Behaviors in LLMs
  5. 2026Steered LLM Activations are Non-Surjective
  6. 2025ExpertSteer: Intervening in LLMs through Expert Knowledge
  7. 2026Steer Like the LLM: Activation Steering that Mimics Prompting
  8. 2026How to Tame a Multi-Headed Hydra? Adaptive Multi-Category Safety Steering for Large Language Models