mlsanityData to 14 Oct 2026

Conscious Gaze: Adaptive Attention Mechanisms for Hallucination Mitigation in Vision-Language Models

arXiv 2512.05546 ↗First posted 5 Dec 2025cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large Vision-Language Models (VLMs) often exhibit text inertia, where attention drifts from visual evidence toward linguistic priors, resulting in object hallucinations. Existing decoding strategies intervene only at the output logits and thus cannot correct internal reasoning drift, while recent internal-control methods based on heuristic head suppression or global steering vectors lack principled grounding. We introduce Conscious Gaze (CG-VLM), a training-free, inference-time framework that converts game-theoretic interpretability into actionable decoding control. A Cognitive Demand Sensor built on Harsanyi interactions estimates instantaneous vision-text synergy and identifies moments when visual grounding is necessary. Conditioned on this signal, a Focused Consensus Induction module selectively reorients mid-layer attention toward visual tokens before collapse into text priors. CG-VLM achieves state-of-the-art results on POPE and CHAIR across InstructBLIP, LLaVA, Qwen-VL, and mPLUG, while preserving general capabilities, demonstrating that token-level sensing enables precise, context-aware intervention without compromising foundational knowledge.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 29 references that are in the map. First posted on arXiv 5 Dec 2025.

Builds on

27 of 29 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Sampling7 Jul
  2. 2025Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppression22 May
  3. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steering5 Feb
  4. 2025PAINT: Paying Attention to INformed Tokens to Mitigate Hallucination in Large Vision-Language Model21 Jan
  5. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens23 Nov
  6. 2024Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoning30 Aug
  7. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  9. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  10. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
  11. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  12. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 15

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  12. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
3 more
  1. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  2. 2018Object Hallucination in Image Captioningcited by 669
  3. 2023Explaining Generalization Power of a DNN Using Interactive Conceptscited by 25

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Focus Matters: Phase-Aware Suppression for Hallucination in Vision-Language Models
  2. 2025VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steering
  3. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence
  4. 2025Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation
  5. 2026Mitigating Hallucination in Vision-Language Models through Barrier-Regulated Adaptive Closed-form Steering
  6. 2026See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
  7. 2026Beyond Visual Enhancement: Adaptive Multi-Context Steering to Mitigate LVLM Hallucinations
  8. 2026SAGE: Sink-Aware Grounded Decoding for Multimodal Hallucination Mitigation