# Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models

arXiv: https://arxiv.org/abs/2602.21704 · HTML: https://mlsanity.com/paper/2602.21704/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 19 of its 57 references that are in the map.
- First posted on arXiv 25 Feb 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)
- Mechanistic interpretability / [Hallucination detection from model internals](https://mlsanity.com/mechinterp/hallucination-detection-from-model-internals/)

## Builds on (most cited first, 19 of 19)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [YOLOv11: An Overview of the Key Architectural Enhancements](https://mlsanity.com/paper/2410.17725/) (2024)
- [Hallucination of Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2404.18930/) (2024)
- [A Survey on Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2402.00253/) (2024)
- [AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation](https://mlsanity.com/paper/2311.07397/) (2023)
- [Building and better understanding vision-language models: insights and future directions](https://mlsanity.com/paper/2408.12637/) (2024)
- [RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness](https://mlsanity.com/paper/2405.17220/) (2024)
- [Efficient Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2405.10739/) (2024)
- [MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation](https://mlsanity.com/paper/2410.11779/) (2024)
- [Semantics-Adaptive Activation Intervention for LLMs via Dynamic Steering Vectors](https://mlsanity.com/paper/2410.12299/) (2024)
- [IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Models](https://mlsanity.com/paper/2403.15952/) (2024)

## Built on by (most cited first, 8 of 8)

- [Dual-Pathway Circuits of Object Hallucination in Vision-Language Models](https://mlsanity.com/paper/2605.13156/) (2026)
- [Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating](https://mlsanity.com/paper/2605.24024/) (2026)
- [When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models](https://mlsanity.com/paper/2605.08245/) (2026)
- [Spectral Query-Key Product Weight Steering for Training-Free VLM Hallucination Mitigation](https://mlsanity.com/paper/2606.20419/) (2026)
- [Prior Directions: Why GUI Grounding Gets Locked in the Past](https://mlsanity.com/paper/2607.26913/) (2026)
- [Test-Time Hallucination Control in Large Vision-Language Models](https://mlsanity.com/paper/2608.11474/) (2026)
- [GeoPID: Decomposing and Steering Visual Information in Vision-Language Models](https://mlsanity.com/paper/2610.08401/) (2026)
- [Mitigating Cross-Modal Hallucinations by Correcting Modality Misrouting in Omnimodal LLMs](https://mlsanity.com/paper/2610.18141/) (2026)

## Abstract (arXiv)

Large Vision-Language Models (LVLMs) exhibit outstanding performance on vision-language tasks but struggle with hallucination problems. Through in-depth analysis of LVLM activation patterns, we reveal two key findings: 1) truthfulness and visual perception capabilities predominantly engage different subsets of attention heads within the model architecture; and 2) truthfulness steering vectors vary significantly across different semantic contexts. Based on these observations, we propose Dynamic Multimodal Activation Steering, a training-free approach for hallucination mitigation. Our method constructs a semantic-based truthfulness steering vector database and computes visual perception steering vectors, enabling context-aware interventions during inference by dynamically selecting the most relevant steering vectors based on input semantic similarity and applying them to the most influential attention heads. We conduct comprehensive experiments across multiple models and datasets, demonstrating that our approach significantly enhances model performance, outperforming existing state-of-the-art methods.