mlsanityData to 14 Oct 2026

Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models

arXiv 2502.06130 ↗First posted 10 Feb 2025cs.CV cs.CLcited by 44

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

While recent Large Vision-Language Models (LVLMs) have shown remarkable performance in multi-modal tasks, they are prone to generating hallucinatory text responses that do not align with the given visual input, which restricts their practical applicability in real-world scenarios. In this work, inspired by the observation that the text-to-image generation process is the inverse of image-conditioned response generation in LVLMs, we explore the potential of leveraging text-to-image generative models to assist in mitigating hallucinations in LVLMs. We discover that generative models can offer valuable self-feedback for mitigating hallucinations at both the response and token levels. Building on this insight, we introduce self-correcting Decoding with Generative Feedback (DeGF), a novel training-free algorithm that incorporates feedback from text-to-image generative models into the decoding process to effectively mitigate hallucinations in LVLMs. Specifically, DeGF generates an image from the initial response produced by LVLMs, which acts as an auxiliary visual reference and provides self-feedback to verify and correct the initial response through complementary or contrastive decoding. Extensive experimental results validate the effectiveness of our approach in mitigating diverse types of hallucinations, consistently surpassing state-of-the-art methods across six benchmarks. Code is available at https://github.com/zhangce01/DeGF.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 59 of its 68 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

59 of 68 references.

Closest ancestors · 19

Same direction, newest first

  1. 2024Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models16 Jul
  2. 2024Multi-Object Hallucination in Vision-Language Models8 Jul
  3. 2024Detecting and Evaluating Medical Hallucinations in Large Vision Language Models14 Jun
  4. 2024CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models4 Jun
  5. 2024Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization24 May
  6. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  7. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  8. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  9. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  10. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  11. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Model12 Dec
  12. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
7 more
  1. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  2. 2023Mitigating Hallucination in Visual Language Models with Visual Supervision27 Nov
  3. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  4. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  5. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  6. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  7. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 40

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  12. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
28 more
  1. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  4. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  7. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  9. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  10. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  11. 2022Diffusion Models in Vision: A Surveycited by 1,015
  12. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  13. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  14. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  15. 2018Object Hallucination in Image Captioningcited by 669
  16. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  17. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  18. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  19. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  20. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  21. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  22. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 306
  23. 2021SegDiff: Image Segmentation with Diffusion Probabilistic Modelscited by 280
  24. 2023StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesiscited by 212
  25. 2021Diffusion Models for Implicit Image Segmentation Ensemblescited by 198
  26. 2023Expressive Text-to-Image Generation with Rich Textcited by 89
  27. 2024Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Modelscited by 27
  28. 2024Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Modelscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 19; 2026: 25.

44 papers.

Sort:

By area:

  1. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Modelscited by 41
  2. 2025ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Modelscited by 16
  3. 2025Visual Representations inside the Language Modelcited by 15
  4. 2026pySpatial: Generating 3D Visual Programs for Zero-Shot Spatial Reasoningcited by 14
  5. 2025When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understandingcited by 12
  6. 2026ApET: Approximation-Error Guided Token Compression for Efficient VLMscited by 12
  7. 2026Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decodingcited by 12
  8. 2025Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimizationcited by 10
  9. 2025MRFD: Multi-Region Fusion Decoding with Self-Consistency for Mitigating Hallucinations in LVLMscited by 9
  10. 2025Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attentioncited by 9
  11. 2026Hallucination Begins Where Saliency Dropscited by 9
  12. 2025Cross-Image Contrastive Decoding: Precise, Lossless Suppression of Language Priors in Large Vision-Language Modelscited by 8
32 more
  1. 2025SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewardscited by 8
  2. 2026Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memorycited by 5
  3. 2025Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Modelscited by 4
  4. 2025HalluRNN: Mitigating Hallucinations via Recurrent Cross-Layer Reasoning in Large Vision-Language Modelscited by 4
  5. 2025CoFi-Dec: Hallucination-Resistant Decoding via Coarse-to-Fine Generative Feedback in Large Vision-Language Modelscited by 4
  6. 2025Taming Hallucinations: Boosting MLLMs' Video Understanding via Counterfactual Video Generationcited by 4
  7. 2026Seeing Farther and Smarter: Value-Guided Multi-Path Reflection for VLM Policy Optimizationcited by 4
  8. 2026Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decodingcited by 4
  9. 2026SDCD: Structure-Disrupted Contrastive Decoding for Mitigating Hallucinations in Large Vision-Language Modelscited by 3
  10. 2026Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbationcited by 3
  11. 2026Temporal Contrastive Decoding: A Training-Free Method for Large Audio-Language Modelscited by 3
  12. 2026Mitigating Hallucinations in Large Vision-Language Models without Performance Degradationcited by 3
  13. 2025Controlling Multimodal LLMs via Reward-guided Decodingcited by 2
  14. 2026One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucinationcited by 2
  15. 2026Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gatingcited by 2
  16. 2026ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMscited by 2
  17. 2026LENS: Adaptive Spatio-Temporal Zooming for Keyframe Sampling in Long-Form Videoscited by 2
  18. 2025Curing Semantic Drift: A Dynamic Approach to Grounding Generation in Large Vision-Language Modelscited by 1
  19. 2025ChainMPQ: Interleaved Text-Image Reasoning Chains for Mitigating Relation Hallucinationscited by 1
  20. 2025Adaptive Residual-Update Steering for Low-Overhead Hallucination Mitigation in Large Vision Language Modelscited by 1
  21. 2026Kestrel: Grounding Self-Refinement for LVLM Hallucination Mitigationcited by 1
  22. 2026Finding Distributed Object-Centric Properties in Self-Supervised Transformerscited by 1
  23. 2026Risk-aware Selective Prompting for Hallucination Mitigation in Large Vision-Language Modelscited by 1
  24. 2025Rationale-Enhanced Decoding for Multi-modal Chain-of-Thoughtcited by 0
  25. 2025Self-Consistency as a Free Lunch: Reducing Hallucinations in Vision-Language Models via Self-Reflectioncited by 0
  26. 2026Self-Correction Inside the Model: Leveraging Layer Attention to Mitigate Hallucinations in Large Vision Language Modelscited by 0
  27. 2026Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Modelscited by 0
  28. 2026Self-Correction Can Amplify Hallucinations: Fact-Level Repair with Graph-Based Evidence Routing in Multimodal Generationcited by 0
  29. 2026Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguardscited by 0
  30. 2026Test-Time Hallucination Control in Large Vision-Language Modelscited by 0
  31. 2026Dynamic Alignment Compensation for Hallucination Mitigation in Large Vision-Language Modelscited by 0
  32. 2026Semantic-Spatial Agreement Verification for Mitigating Object Hallucination in Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs
  2. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling
  3. 2024HELPD: Mitigating Hallucination of LVLMs by Hierarchical Feedback Learning with Vision-enhanced Penalty Decoding
  4. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  5. 2026Context-Aware Decoding for Faithful Vision-Language Generation
  6. 2024Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding
  7. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models
  8. 2026SAGE: Sink-Aware Grounded Decoding for Multimodal Hallucination Mitigation