mlsanityData to 7 Oct 2026

SteerVLM: Robust Model Control through Lightweight Activation Steering for Vision Language Models

arXiv 2510.26769 ↗First posted 30 Oct 2025cs.CV cs.LGcited by 8

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

This work introduces SteerVLM, a lightweight steering module designed to guide Vision-Language Models (VLMs) towards outputs that better adhere to desired instructions. Our approach learns from the latent embeddings of paired prompts encoding target and converse behaviors to dynamically adjust activations connecting the language modality with image context. This allows for fine-grained, inference-time control over complex output semantics without modifying model weights while preserving performance on off-target tasks. Our steering module requires learning parameters equal to 0.14% of the original VLM's size. Our steering module gains model control through dimension-wise activation modulation and adaptive steering across layers without requiring pre-extracted static vectors or manual tuning of intervention points. Furthermore, we introduce VNIA (Visual Narrative Intent Alignment), a multimodal dataset specifically created to facilitate the development and evaluation of VLM steering techniques. Our method outperforms existing intervention techniques on steering and hallucination mitigation benchmarks for VLMs and proposes a robust solution for multimodal model control through activation engineering.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 38 references that are in the map. First posted on arXiv 30 Oct 2025.

Builds on

27 of 38 references.

References · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2015VQA: Visual Question Answeringcited by 2,489
  12. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
15 more
  1. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  2. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  3. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  4. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  5. 2024Flex Attention: A Programming Model for Generating Optimized Attention Kernelscited by 180
  6. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
  7. 2024Adaptive Activation Steering: A Tuning-Free LLM Truthfulness Improvement Method for Diverse Hallucinations Categoriescited by 74
  8. 2024Controlling Language and Diffusion Models by Transporting Activationscited by 59
  9. 2025Multi-Attribute Steering of Language Models via Targeted Interventioncited by 44
  10. 2024Erasing Conceptual Knowledge from Language Modelscited by 34
  11. 2024Steering Large Language Models using Conceptors: Improving Addition-Based Activation Engineeringcited by 33
  12. 2025Controlling Large Language Models Through Concept Activation Vectorscited by 27
  13. 2024Style-Specific Neurons for Steering LLMs in Text Style Transfercited by 24
  14. 2024Investigating and Mitigating Object Hallucinations in Pretrained Vision-Language (CLIP) Modelscited by 15
  15. 2025Analyzing Finetuning Representation Shift for Multimodal LLMs Steeringcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Modelscited by 14
  2. 2026Dual-Pathway Circuits of Object Hallucination in Vision-Language Modelscited by 4
  3. 2026Steering to Say No: Configurable Refusal via Activation Steering in Vision Language Modelscited by 3
  4. 2026One-shot Optimized Steering Vector for Hallucination Mitigation for VLMscited by 1
  5. 2026Sparse Visual Thought Circuits in Vision-Language Modelscited by 0
  6. 2026ScAle: Attention Head Scaling as a Minimal Adapter for Spatial Reasoning in Vision Language Modelscited by 0
  7. 2026Multimodal Model Diffing for Feature Discovery and Controlcited by 0
  8. 2026When Safety Overrides Vision: Exploring Dynamics between Vision Influence and Safety Alignment in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VISOR++: Universal Visual Inputs based Steering for Large Vision Language Models
  2. 2025VISOR: Visual Input-based Steering for Output Redirection in Vision-Language Models
  3. 2025Activation Steering Meets Preference Optimization: Defense Against Jailbreaks in Vision Language Models
  4. 2026Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering
  5. 2026EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training
  6. 2026VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models
  7. 2026SteerVLA: Steering Vision-Language-Action Models in Long-Tail Driving Scenarios
  8. 2026Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Control