mlsanityData to 14 Oct 2026

Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Models

arXiv 2505.14071 ↗First posted 20 May 2025cs.LG cs.CL cs.CVcited by 9

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Steering methods have emerged as effective and targeted tools for guiding large language models' (LLMs) behavior without modifying their parameters. Multimodal large language models (MLLMs), however, do not currently enjoy the same suite of techniques, due in part to their recency and architectural diversity. Inspired by this gap, we investigate whether MLLMs can be steered using vectors derived from their text-only LLM backbone, via sparse autoencoders (SAEs), mean shift, and linear probing. We find that text-derived steering consistently enhances multimodal accuracy across diverse MLLM architectures and visual tasks. In particular, mean shift boosts spatial relationship accuracy on CV-Bench by up to +7.3% and counting accuracy by up to +3.3%, outperforming prompting and exhibiting strong generalization to out-of-distribution datasets. These results highlight textual steering vectors as a powerful, efficient mechanism for enhancing grounding in MLLMs with minimal additional data collection and computational overhead.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 67 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

47 of 67 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering16 Dec
  2. 2024TLDR: Token-Level Detective Reward Model for Large Vision Language Models7 Oct
  3. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  4. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 42

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  10. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  11. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  12. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
30 more
  1. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  2. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  3. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  4. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  5. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  6. 2023Steering Language Models With Activation Engineeringcited by 922
  7. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  8. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  9. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  10. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  11. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  12. 2024Scaling and evaluating sparse autoencoderscited by 553
  13. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  14. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  15. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  16. 2024The Platonic Representation Hypothesiscited by 382
  17. 20242 OLMo 2 Furiouscited by 340
  18. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  19. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
  20. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 218
  21. 2024Programming Refusal with Conditional Activation Steeringcited by 206
  22. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 200
  23. 2025Open Problems in Mechanistic Interpretabilitycited by 179
  24. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoderscited by 143
  25. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 141
  26. 2022Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoningcited by 92
  27. 2024DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMscited by 75
  28. 2024TableVQA-Bench: A Visual Question Answering Benchmark on Multiple Table Domainscited by 69
  29. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  30. 2024IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representationscited by 38

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

  1. 2025MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steeringcited by 10
  2. 2025Resa: Transparent Reasoning Models via SAEscited by 6
  3. 2026Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Modelscited by 3
  4. 2026EPSVec: Efficient and Private Synthetic Data Generation via Dataset Vectorscited by 2
  5. 2026One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucinationcited by 2
  6. 2025VISOR++: Universal Visual Inputs based Steering for Large Vision Language Modelscited by 1
  7. 2025VISOR: Visual Input-based Steering for Output Redirection in Vision-Language Modelscited by 0
  8. 2026Do Unified Multimodal Models Think in One Space? A Lens Through Cross-Branch Steeringcited by 0
  9. 2026Sparse Autoencoders for Mechanistic Interpretability in NLP: A Critical Survey of Recoverability, Interpretability, and Utilitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Revolution of Multimodal Large Language Models: A Survey
  2. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model
  3. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  4. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  5. 2024From Text to Pixel: Advancing Long-Context Understanding in MLLMs
  6. 2025Visual Representation Alignment for Multimodal Large Language Models
  7. 2024MileBench: Benchmarking MLLMs in Long Context
  8. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review