mlsanityData to 14 Oct 2026

Hallucinatory Image Tokens: A Training-free EAZY Approach on Detecting and Mitigating Object Hallucinations in LVLMs

arXiv 2503.07772 ↗First posted 10 Mar 2025cs.CV cs.LGcited by 15

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Despite their remarkable potential, Large Vision-Language Models (LVLMs) still face challenges with object hallucination, a problem where their generated outputs mistakenly incorporate objects that do not actually exist. Although most works focus on addressing this issue within the language-model backbone, our work shifts the focus to the image input source, investigating how specific image tokens contribute to hallucinations. Our analysis reveals a striking finding: a small subset of image tokens with high attention scores are the primary drivers of object hallucination. By removing these hallucinatory image tokens (only 1.5% of all image tokens), the issue can be effectively mitigated. This finding holds consistently across different models and datasets. Building on this insight, we introduce EAZY, a novel, training-free method that automatically identifies and Eliminates hAllucinations by Zeroing out hallucinatorY image tokens. We utilize EAZY for unsupervised object hallucination detection, achieving 15% improvement compared to previous methods. Additionally, EAZY demonstrates remarkable effectiveness in mitigating hallucinations while preserving model utility and seamlessly adapting to various LVLM architectures.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 33 references that are in the map. First posted on arXiv 10 Mar 2025.

Builds on

30 of 33 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination6 Oct
  2. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations3 Oct
  3. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models4 Aug
  4. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  5. 2024DOPRA: Decoding Over-accumulation Penalization and Re-allocation in Specific Weighting Layer21 Jul
  6. 2024Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models28 May
  7. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  8. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  9. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  10. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  11. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  12. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
5 more
  1. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  2. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision13 Nov
  3. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models23 Oct
  4. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  5. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 13

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  8. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  9. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  10. 2018Object Hallucination in Image Captioningcited by 669
  11. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  12. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
1 more
  1. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 122

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 13.

15 papers.

Sort:

By area:

  1. 2025Backdoor Cleaning without External Guidance in MLLM Fine-tuningcited by 35
  2. 2025On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Modelscited by 5
  3. 2026Counting Circuits: Mechanistic Interpretability of Visual Reasoning in Large Vision-Language Modelscited by 3
  4. 2026SAVAA: Mitigating Hallucinations in LVLMs via Step-wise Adaptive Visual Attention Amplificationcited by 2
  5. 2026MI-Pruner: Crossmodal Mutual Information-guided Token Pruner for Efficient MLLMscited by 2
  6. 2026Mitigating Multimodal Hallucination via Phase-wise Self-rewardcited by 2
  7. 2026Spotlight and Shadow: Attention-Guided Dual-Anchor Introspective Decoding for MLLM Hallucination Mitigationcited by 1
  8. 2026Instruction Lens Score: Your Instruction Contributes a Powerful Object Hallucination Detector for Multimodal Large Language Modelscited by 1
  9. 2026ConsensusDrop: Fusing Visual and Cross-Modal Saliency for Efficient Vision Language Modelscited by 0
  10. 2026HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations in Vision-Language Modelscited by 0
  11. 2026Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Modelscited by 0
  12. 2026VisER: Visual Evidence and Reliance for Object Hallucination Detection in LVLMscited by 0
3 more
  1. 2026RVSD: Retrieval Vision Sparse Decoding for Mitigating Visual Hallucinations in Large Vision-Language Modelscited by 0
  2. 2026(How) Do MLLMs Report Bistable Images Like Humans?cited by 0
  3. 2026Attention Sinks in VLMs Are Borrowed, Not Seencited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens
  2. 2026Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinations
  3. 2025Exploring Causes and Mitigation of Hallucinations in Large Vision Language Models
  4. 2025Diving into Mitigating Hallucinations from a Vision Perspective for Large Vision-Language Models
  5. 2026HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Token
  6. 2026CRoPS: A Training-Free Hallucination Mitigation Framework for Vision-Language Models
  7. 2025Analyzing and Mitigating Object Hallucination: A Training Bias Perspective
  8. 2026Test-Time Hallucination Control in Large Vision-Language Models