# Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models

arXiv: https://arxiv.org/abs/2502.06130 · HTML: https://mlsanity.com/paper/2502.06130/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 59 of its 68 references that are in the map.
- First posted on arXiv 10 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)

## Builds on (most cited first, 30 of 59)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [Aligning Large Multimodal Models with Factually Augmented RLHF](https://mlsanity.com/paper/2309.14525/) (2023)
- [A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge](https://mlsanity.com/paper/2206.01718/) (2022)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [Muse: Text-To-Image Generation via Masked Generative Transformers](https://mlsanity.com/paper/2301.00704/) (2023)

## Built on by (most cited first, 30 of 44)

- [VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Models](https://mlsanity.com/paper/2505.22654/) (2025)
- [ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models](https://mlsanity.com/paper/2507.00898/) (2025)
- [Visual Representations inside the Language Model](https://mlsanity.com/paper/2510.04819/) (2025)
- [pySpatial: Generating 3D Visual Programs for Zero-Shot Spatial Reasoning](https://mlsanity.com/paper/2603.00905/) (2026)
- [When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding](https://mlsanity.com/paper/2506.05551/) (2025)
- [ApET: Approximation-Error Guided Token Compression for Efficient VLMs](https://mlsanity.com/paper/2602.19870/) (2026)
- [Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decoding](https://mlsanity.com/paper/2603.13366/) (2026)
- [Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimization](https://mlsanity.com/paper/2506.04039/) (2025)
- [MRFD: Multi-Region Fusion Decoding with Self-Consistency for Mitigating Hallucinations in LVLMs](https://mlsanity.com/paper/2508.10264/) (2025)
- [Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attention](https://mlsanity.com/paper/2511.20032/) (2025)
- [Hallucination Begins Where Saliency Drops](https://mlsanity.com/paper/2601.20279/) (2026)
- [Cross-Image Contrastive Decoding: Precise, Lossless Suppression of Language Priors in Large Vision-Language Models](https://mlsanity.com/paper/2505.10634/) (2025)
- [SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewards](https://mlsanity.com/paper/2506.07963/) (2025)
- [Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memory](https://mlsanity.com/paper/2603.15800/) (2026)
- [Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models](https://mlsanity.com/paper/2505.17061/) (2025)
- [HalluRNN: Mitigating Hallucinations via Recurrent Cross-Layer Reasoning in Large Vision-Language Models](https://mlsanity.com/paper/2506.17587/) (2025)
- [CoFi-Dec: Hallucination-Resistant Decoding via Coarse-to-Fine Generative Feedback in Large Vision-Language Models](https://mlsanity.com/paper/2512.23453/) (2025)
- [Taming Hallucinations: Boosting MLLMs' Video Understanding via Counterfactual Video Generation](https://mlsanity.com/paper/2512.24271/) (2025)
- [Seeing Farther and Smarter: Value-Guided Multi-Path Reflection for VLM Policy Optimization](https://mlsanity.com/paper/2602.19372/) (2026)
- [Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding](https://mlsanity.com/paper/2606.27596/) (2026)
- [SDCD: Structure-Disrupted Contrastive Decoding for Mitigating Hallucinations in Large Vision-Language Models](https://mlsanity.com/paper/2601.03500/) (2026)
- [Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbation](https://mlsanity.com/paper/2604.12424/) (2026)
- [Temporal Contrastive Decoding: A Training-Free Method for Large Audio-Language Models](https://mlsanity.com/paper/2604.15383/) (2026)
- [Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation](https://mlsanity.com/paper/2604.20366/) (2026)
- [Controlling Multimodal LLMs via Reward-guided Decoding](https://mlsanity.com/paper/2508.11616/) (2025)
- [One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination](https://mlsanity.com/paper/2603.10360/) (2026)
- [Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating](https://mlsanity.com/paper/2605.24024/) (2026)
- [ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMs](https://mlsanity.com/paper/2606.31054/) (2026)
- [LENS: Adaptive Spatio-Temporal Zooming for Keyframe Sampling in Long-Form Videos](https://mlsanity.com/paper/2607.25125/) (2026)
- [Curing Semantic Drift: A Dynamic Approach to Grounding Generation in Large Vision-Language Models](https://mlsanity.com/paper/2506.21509/) (2025)

## Abstract (arXiv)

While recent Large Vision-Language Models (LVLMs) have shown remarkable performance in multi-modal tasks, they are prone to generating hallucinatory text responses that do not align with the given visual input, which restricts their practical applicability in real-world scenarios. In this work, inspired by the observation that the text-to-image generation process is the inverse of image-conditioned response generation in LVLMs, we explore the potential of leveraging text-to-image generative models to assist in mitigating hallucinations in LVLMs. We discover that generative models can offer valuable self-feedback for mitigating hallucinations at both the response and token levels. Building on this insight, we introduce self-correcting Decoding with Generative Feedback (DeGF), a novel training-free algorithm that incorporates feedback from text-to-image generative models into the decoding process to effectively mitigate hallucinations in LVLMs. Specifically, DeGF generates an image from the initial response produced by LVLMs, which acts as an auxiliary visual reference and provides self-feedback to verify and correct the initial response through complementary or contrastive decoding. Extensive experimental results validate the effectiveness of our approach in mitigating diverse types of hallucinations, consistently surpassing state-of-the-art methods across six benchmarks. Code is available at https://github.com/zhangce01/DeGF.