mlsanityData to 14 Oct 2026

Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization

arXiv 2405.15356 ↗First posted 24 May 2024cs.CVcited by 34

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Although Large Visual Language Models (LVLMs) have demonstrated exceptional abilities in understanding multimodal data, they invariably suffer from hallucinations, leading to a disconnect between the generated text and the corresponding images. Almost all current visual contrastive decoding methods attempt to mitigate these hallucinations by introducing visual uncertainty information that appropriately widens the contrastive logits gap between hallucinatory and targeted ones. However, due to uncontrollable nature of the global visual uncertainty, they struggle to precisely induce the hallucinatory tokens, which severely limits their effectiveness in mitigating hallucinations and may even lead to the generation of undesired hallucinations. To tackle this issue, we conducted the theoretical analysis to promote the effectiveness of contrast decoding. Building on this insight, we introduce a novel optimization strategy named Hallucination-Induced Optimization (HIO). This strategy seeks to amplify the contrast between hallucinatory and targeted tokens relying on a fine-tuned theoretical preference model (i.e., Contrary Bradley-Terry Model), thereby facilitating efficient contrast decoding to alleviate hallucinations in LVLMs. Extensive experimental research demonstrates that our HIO strategy can effectively reduce hallucinations in LVLMs, outperforming state-of-the-art methods across various benchmarks.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 51 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

42 of 51 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  2. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  3. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  4. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding28 Feb
  5. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  6. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  8. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  9. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  10. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models9 Oct
  11. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  12. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Models29 Aug
3 more
  1. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  2. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  3. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 27

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
15 more
  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  3. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  4. 2018Hierarchical Neural Story Generationcited by 1,193
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  6. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  7. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  8. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  9. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  10. 2018Object Hallucination in Image Captioningcited by 669
  11. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  12. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  13. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
  14. 2020Detecting Hallucinated Content in Conditional Neural Sequence Generationcited by 152
  15. 2022Natural Color Fool: Towards Boosting Black-box Unrestricted Attackscited by 38

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 15; 2026: 16.

34 papers.

Sort:

By area:

  1. 2025Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Modelscited by 44
  2. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 30
  3. 2025Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigationcited by 23
  4. 2025Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigationcited by 20
  5. 2024Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignmentcited by 19
  6. 2025ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Modelscited by 16
  7. 2025LanP: Rethinking the Impact of Language Priors in Large Vision-Language Modelscited by 14
  8. 2025TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Interventioncited by 11
  9. 2025CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Interventioncited by 11
  10. 2024MMRel: Benchmarking Relation Understanding in Multi-Modal Large Language Modelscited by 10
  11. 2025Through the Magnifying Glass: Adaptive Perception Magnification for Hallucination-Free VLM Decodingcited by 10
  12. 2025Attention Hijackers: Detect and Disentangle Attention Hijacking in LVLMs for Hallucination Mitigationcited by 9
22 more
  1. 2025OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucinationcited by 9
  2. 2025Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineeringcited by 8
  3. 2026Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steeringcited by 8
  4. 2026Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigationcited by 7
  5. 2026MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimizationcited by 6
  6. 2026KVSmooth: Mitigating Hallucination in Multi-modal Large Language Models through Key-Value Smoothingcited by 5
  7. 2025HalluRNN: Mitigating Hallucinations via Recurrent Cross-Layer Reasoning in Large Vision-Language Modelscited by 4
  8. 2026Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucinationcited by 4
  9. 2025CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 3
  10. 2026Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbationcited by 3
  11. 2026Mitigating Hallucinations in Large Vision-Language Models without Performance Degradationcited by 3
  12. 2026Anatomy of a Lie: A Multi-Stage Diagnostic Framework for Tracing Hallucinations in Vision-Language Modelscited by 2
  13. 2026ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMscited by 2
  14. 2025Self-Aug: Query and Entropy Adaptive Decoding for Large Vision-Language Modelscited by 1
  15. 2025Toward More Reliable Artificial Intelligence: Reducing Hallucinations in Vision-Language Modelscited by 1
  16. 2026Kestrel: Grounding Self-Refinement for LVLM Hallucination Mitigationcited by 1
  17. 2026The Truth Stays in the Family: Enhancing Contextual Grounding via Inherited Truthful Heads in Model Lineagescited by 1
  18. 2026Beyond Dominant Patches: Spatial Credit Redistribution For Grounded Vision-Language Modelscited by 0
  19. 2026Self-Correction Inside the Model: Leveraging Layer Attention to Mitigate Hallucinations in Large Vision Language Modelscited by 0
  20. 2026CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steeringcited by 0
  21. 2026Test-Time Hallucination Control in Large Vision-Language Modelscited by 0
  22. 2026DARE to Mitigate Hallucination: Dual-path Auto-Regressive-aware Editingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Delve into Visual Contrastive Decoding for Hallucination Mitigation of Large Vision-Language Models
  2. 2025The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?
  3. 2024ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models
  4. 2026SDCD: Structure-Disrupted Contrastive Decoding for Mitigating Hallucinations in Large Vision-Language Models
  5. 2024VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding
  6. 2025Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Models
  7. 2025Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs
  8. 2025SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding