mlsanityData to 14 Oct 2026

The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steering

arXiv 2502.03628 ↗First posted 5 Feb 2025cs.CV cs.AI cs.LGcited by 66

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large Vision-Language Models (LVLMs) can reason effectively over both textual and visual inputs, but they tend to hallucinate syntactically coherent yet visually ungrounded contents. In this paper, we investigate the internal dynamics of hallucination by examining the tokens logits ranking throughout the generation process, revealing three key patterns in how LVLMs process information: (1) gradual visual information loss - visually grounded tokens gradually become less favored throughout generation, and (2) early excitation - semantically meaningful tokens achieve peak activation in the layers earlier than the final layer. (3) hidden genuine information - visually grounded tokens though not being eventually decoded still retain relatively high rankings at inference. Based on these insights, we propose VISTA (Visual Information Steering with Token-logit Augmentation), a training-free inference-time intervention framework that reduces hallucination while promoting genuine information. VISTA works by combining two complementary approaches: reinforcing visual information in activation space and leveraging early layer activations to promote semantically meaningful decoding. Compared to existing methods, VISTA requires no external supervision and is applicable to various decoding strategies. Extensive experiments show that VISTA on average reduces hallucination by about 40% on evaluated open-ended generation task, and it consistently outperforms existing methods on four benchmarks across four architectures under three decoding strategies. Code is available at https://github.com/LzVv123456/VISTA.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 43 references that are in the map. First posted on arXiv 5 Feb 2025.

Builds on

35 of 43 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering21 Oct
  2. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  3. 2024Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models28 May
  4. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  5. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  6. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  7. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  8. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  9. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  10. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
  11. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  12. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
1 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 22

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  9. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  10. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  11. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  12. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
10 more
  1. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  2. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  3. 2018Object Hallucination in Image Captioningcited by 669
  4. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  5. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  6. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 327
  7. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 248
  8. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  9. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 139
  10. 2024Implicit In-context Learningcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 51.

66 papers.

Sort:

By area:

  1. 2026Verbalizable Representations Form a Global Workspace in Language Modelscited by 75
  2. 2025Qwen Look Again: Guiding Vision-Language Reasoning Models to Re-attention Visual Informationcited by 28
  3. 2025Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flowcited by 13
  4. 2025MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Modelscited by 12
  5. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 11
  6. 2025Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attentioncited by 9
  7. 2026Hallucination Begins Where Saliency Dropscited by 9
  8. 2025TraceDet: Hallucination Detection from the Decoding Trace of Diffusion Large Language Modelscited by 8
  9. 2026Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steeringcited by 8
  10. 2026From Weights to Activations: Is Steering the Next Frontier of Adaptation?cited by 8
  11. 2025Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correctioncited by 7
  12. 2026Dual-Pathway Circuits of Object Hallucination in Vision-Language Modelscited by 7
38 more
  1. 2025Silence is Golden: Mitigating Hallucinations in Large Audio-Language Models via Layer-Weighted Vector Steeringcited by 6
  2. 2025SAVE: Sparse Autoencoder-Driven Visual Information Enhancement for Mitigating Object Hallucinationcited by 5
  3. 2026Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidancecited by 5
  4. 2026KVSmooth: Mitigating Hallucination in Multi-modal Large Language Models through Key-Value Smoothingcited by 5
  5. 2026Improving Visual Reasoning with Iterative Evidence Refinementcited by 5
  6. 2025CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decodingcited by 4
  7. 2026CRoPS: A Training-Free Hallucination Mitigation Framework for Vision-Language Modelscited by 4
  8. 2026When VLMs 'Fix' Students: Identifying and Penalizing Over-Correction in the Evaluation of Multi-line Handwritten Math OCRcited by 4
  9. 2025VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steeringcited by 3
  10. 2026Aligning What Vision-Language Models See and Perceive with Adaptive Information Flowcited by 3
  11. 2026Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Modelscited by 3
  12. 2026FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Modelscited by 3
  13. 2025Beyond Interpretability: When, Why, and How Sparse Autoencoders Enable Label-Free Visual Steeringcited by 2
  14. 2026One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucinationcited by 2
  15. 2025Large Sign Language Models: Toward 3D American Sign Language Translationcited by 1
  16. 2025Adaptive Residual-Update Steering for Low-Overhead Hallucination Mitigation in Large Vision Language Modelscited by 1
  17. 2026Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMscited by 1
  18. 2026SAKED: Mitigating Hallucination in Large Vision-Language Models via Stability-Aware Knowledge Enhanced Decodingcited by 1
  19. 2026HIME: Mitigating Object Hallucinations in LVLMs via Hallucination Insensitivity Model Editingcited by 1
  20. 2026VisRef: Visual Refocusing while Thinking Improves Test-Time Scaling in Multi-Modal Large Reasoning Modelscited by 1
  21. 2026GroundCount: Grounding Vision-Language Models with Object Detection for Mitigating Counting Hallucinationscited by 1
  22. 2026Two Birds, One Projection: Harmonizing Safety and Utility in LVLMs via Inference-time Feature Projectioncited by 1
  23. 2026ReflectCAP: Detailed Image Captioning with Reflective Memorycited by 1
  24. 2026GEASS: Gated Evidence-Adaptive Selective Caption Trust for Vision-Language Modelscited by 1
  25. 2026When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Modelscited by 1
  26. 2026Vocabulary Hijacking in LVLMs: Unveiling Critical Attention Heads by Excluding Inert Tokens to Mitigate Hallucinationcited by 1
  27. 2026Not Blind but Silenced: Rebalancing Vision and Language via Adversarial Counter-Commonsense Equilibriumcited by 1
  28. 2026Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigationcited by 1
  29. 2026Steering Where to Listen: Instruction-Based Activation Steering Redirects Temporal Attention in Large Audio-Language Modelscited by 1
  30. 2025Conscious Gaze: Adaptive Attention Mechanisms for Hallucination Mitigation in Vision-Language Modelscited by 0
  31. 2026ClueTracer: Question-to-Vision Clue Tracing for Training-Free Hallucination Suppression in Multimodal Reasoningcited by 0
  32. 2026Visual Attention Drifts,but Anchors Hold:Mitigating Hallucination in Multimodal Large Language Models via Cross-Layer Visual Anchorscited by 0
  33. 2026Benchmarking and Mechanistic Analysis of Vision-Language Models for Cross-Depiction Assembly Instruction Alignmentcited by 0
  34. 2026Aligning Recommendations with User Popularity Preferencescited by 0
  35. 2026Revisiting Greedy Decoding for Visual Question Answering: A Calibration Perspectivecited by 0
  36. 2026Referring Multiple Regions with Large Multimodal Models via Contextual Latent Steeringcited by 0
  37. 2026Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Modelscited by 0
  38. 2026Self-Captioning Multimodal Interaction Tuning: Amplifying Exploitable Redundancies for Robust Vision Language Modelscited by 0

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisit What You See: Revealing Visual Semantics in Vision Tokens to Guide LVLM Decoding
  2. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens
  3. 2025Curing Semantic Drift: A Dynamic Approach to Grounding Generation in Large Vision-Language Models
  4. 2026Beyond Visual Enhancement: Adaptive Multi-Context Steering to Mitigate LVLM Hallucinations
  5. 2025Enhancing Visual Reliance in Text Generation: A Bayesian Perspective on Mitigating Hallucination in Large Vision-Language Models
  6. 2026Context-Aware Decoding for Faithful Vision-Language Generation
  7. 2025Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs
  8. 2025PAINT: Paying Attention to INformed Tokens to Mitigate Hallucination in Large Vision-Language Model