mlsanityData to 14 Oct 2026

Vocabulary Hijacking in LVLMs: Unveiling Critical Attention Heads by Excluding Inert Tokens to Mitigate Hallucination

arXiv 2605.10622 ↗First posted 11 May 2026cs.MM cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large Vision-Language Models (LVLMs) have achieved remarkable progress in multimodal tasks, yet their reliability is persistently undermined by hallucinations-generating text that contradicts visual input. Recent studies often attribute these errors to inadequate visual attention. In this work, we analyze the attention mechanisms via the logit lens, uncovering a distinct anomaly we term Vocabulary Hijacking. We discover that specific visual tokens, defined as Inert Tokens, disproportionately attract attention. Crucially, when their intermediate hidden states are projected into the vocabulary space, they consistently decode to a fixed set of unrelated words (termed Hijacking Anchors) across layers, revealing a rigid semantic collapse. Leveraging this semantic rigidity, we propose Hijacking Anchor-Based Identification (HABI), a robust strategy to accurately localize these Inert Tokens. To quantify the impact of this phenomenon, we introduce the Non-Hijacked Visual Attention Ratio (NHAR), a novel metric designed to identify attention heads that remain resilient to hijacking and are critical for factual accuracy. Building on these insights, we propose Hijacking-Aware Visual Attention Enhancement (HAVAE), a training-free intervention that selectively strengthens the focus of these identified heads on salient visual content. Extensive experiments across multiple benchmarks demonstrate that HAVAE significantly mitigates hallucinations with no additional computational overhead, while preserving the model's general capabilities. Our code is publicly available at https://github.com/lab-klc/HAVAE.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 29 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

17 of 29 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steering5 Feb
  2. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens23 Nov
  3. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  4. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  5. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  6. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  7. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov

Other references · 10

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  5. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  6. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  7. 2023Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMscited by 91
  8. 2023AtMan: Understanding Transformer Predictions Through Memory Efficient Attention Manipulationcited by 31
  9. 2025Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfercited by 11
  10. 2025Knowledge Fusion of Large Language Models Via Modular SkillPackscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Dynamic Model Merging Made Slimcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Attention Hijackers: Detect and Disentangle Attention Hijacking in LVLMs for Hallucination Mitigation
  2. 2026Visual Attention Drifts,but Anchors Hold:Mitigating Hallucination in Multimodal Large Language Models via Cross-Layer Visual Anchors
  3. 2025Mitigating Hallucination in Large Vision-Language Models through Aligning Attention Distribution to Information Flow
  4. 2026Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinations
  5. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence
  6. 2025Revisit What You See: Revealing Visual Semantics in Vision Tokens to Guide LVLM Decoding
  7. 2026Attention Hijacking: Response Manipulation Across Queries in Vision-Language Models
  8. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens