mlsanityData to 14 Oct 2026

Energy-Guided Decoding for Object Hallucination Mitigation

arXiv 2507.07731 ↗First posted 10 Jul 2025cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Mitigating object hallucination in large vision-language models (LVLMs) is critical to their safe deployment. Existing methods either are restricted to specific decoding methods, or demand sophisticated modifications to visual inputs, or rely on knowledge from external models. In this work, we first reveal the phenomenon that VLMs exhibit significant imbalance in the ``Yes'' ratio ( ie, the fraction of ``Yes'' answers among the total number of questions) across three different visual question answering (VQA) datasets. Furthermore, we propose an energy-based decoding method, which dynamically selects the hidden states from the layer with minimal energy score. It is simple yet effective in reducing the bias for the yes ratio while boosting performance across three benchmarks (POPE, MME, and MMVP). Our method consistently improves accuracy and F1 score on three VQA datasets across three commonly used VLMs over several baseline methods. The average accuracy improvement is 4.82% compared to greedy decoding. Moreover, the average yes-ratio gap reduction is 8.81%, meaning the proposed method is less biased as shown in Figure 1.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 53 references that are in the map. First posted on arXiv 10 Jul 2025.

Builds on

43 of 53 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens23 Nov
  2. 2024CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs19 Aug
  3. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  4. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  5. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  6. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  7. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  9. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  10. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  11. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  11. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  12. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
20 more
  1. 2020Energy-based Out-of-distribution Detectioncited by 1,299
  2. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  3. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  4. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  5. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  6. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  7. 2018Object Hallucination in Image Captioningcited by 669
  8. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  9. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  10. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  11. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  12. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  13. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 405
  14. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  15. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 320
  16. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
  17. 2022Analyzing Transformers in Embedding Spacecited by 132
  18. 2022Linear Adversarial Concept Erasurecited by 101
  19. 2023Overthinking the Truth: Understanding how Language Models Process False Demonstrationscited by 81
  20. 2024Your Classifier Can Be Secretly a Likelihood-Based OOD Detectorcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Look on Demand: A Cognitive Scheduling Framework for Visual Evidence Acquisition in Multimodal Reasoningcited by 1
  2. 2026Semantic-Spatial Agreement Verification for Mitigating Object Hallucination in Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision-Language Models
  2. 2024Seeing is Believing: Mitigating Hallucination in Large Vision-Language Models via CLIP-Guided Decoding
  3. 2025ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models
  4. 2025Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models
  5. 2026When Looking Is Not Enough: Visual Attention Structure Reveals Hallucination in MLLMs
  6. 2025Watch Closely: Mitigating Object Hallucinations in Large Vision-Language Models with Disentangled Decoding
  7. 2025Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decoding
  8. 2024Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding