mlsanityData to 7 Oct 2026

Woodpecker: Hallucination Correction for Multimodal Large Language Models

arXiv 2310.16045 ↗First posted 24 Oct 2023cs.CV cs.AI cs.CLcited by 223

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Hallucination is a big shadow hanging over the rapidly evolving Multimodal Large Language Models (MLLMs), referring to the phenomenon that the generated text is inconsistent with the image content. In order to mitigate hallucinations, existing studies mainly resort to an instruction-tuning manner that requires retraining the models with specific data. In this paper, we pave a different way, introducing a training-free method named Woodpecker. Like a woodpecker heals trees, it picks out and corrects hallucinations from the generated text. Concretely, Woodpecker consists of five stages: key concept extraction, question formulation, visual knowledge validation, visual claim generation, and hallucination correction. Implemented in a post-remedy manner, Woodpecker can easily serve different MLLMs, while being interpretable by accessing intermediate outputs of the five stages. We evaluate Woodpecker both quantitatively and qualitatively and show the huge potential of this new paradigm. On the POPE benchmark, our method obtains a 30.66%/24.33% improvement in accuracy over the baseline MiniGPT-4/mPLUG-Owl. The source code is released at https://github.com/BradyFU/Woodpecker.

Key facts. Cited by 223 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 63 references that are in the map. First posted on arXiv 24 Oct 2023.

Builds on

51 of 63 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Models29 Aug
  2. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  3. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  4. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 47

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2023Segment Anythingcited by 8,158
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
35 more
  1. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  2. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  3. 2021Grounded Language-Image Pre-trainingcited by 1,147
  4. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  5. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  6. 2023A Survey on Multimodal Large Language Modelscited by 898
  7. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  8. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  9. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  10. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  11. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  12. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  13. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  14. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  15. 2021Image Segmentation Using Text and Image Promptscited by 509
  16. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  17. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  18. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  19. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  20. 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 197
  21. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 186
  22. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  23. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 112
  24. 2023Caption Anything: Interactive Image Description with Diverse Multimodal Controlscited by 98
  25. 2023MindAgent: Emergent Gaming Interactioncited by 87
  26. 2023ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptionscited by 86
  27. 2023VIGC: Visual Instruction Generation and Correctioncited by 80
  28. 2023VPGTrans: Transfer Visual Prompt Generator across LLMscited by 79
  29. 2023Detecting Everything in the Open World: Towards Universal Object Detectioncited by 62
  30. 2021The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpuscited by 60
  31. 2023Towards Language Models That Can See: Computer Vision Through the LENS of Natural Languagecited by 53
  32. 2023IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Modelscited by 45
  33. 2023SAM3D: Zero-Shot 3D Object Detection via Segment Anything Modelcited by 40
  34. 2023Zero-shot Faithful Factual Error Correctioncited by 26
  35. 2022Fewer is More: Efficient Object Detection in Large Aerial Imagescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 80; 2025: 79; 2026: 54.

223 papers.

Sort:

By area:

  1. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  2. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  3. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  4. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  5. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  6. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  7. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 223
  8. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  9. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 174
  10. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 171
  11. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 139
  12. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
38 more
  1. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  2. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Datacited by 124
  3. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  4. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 113
  5. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinationscited by 103
  6. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  7. 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Modelscited by 88
  8. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 87
  9. 2024Calibrated Self-Rewarding Vision Language Modelscited by 77
  10. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74
  11. 2024SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detectioncited by 73
  12. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  13. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  14. 2024Visual Hallucinations of Multi-modal Large Language Modelscited by 68
  15. 2024VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understandingcited by 65
  16. 2024CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoningcited by 64
  17. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steeringcited by 56
  18. 2024EventHallusion: Diagnosing Event Hallucinations in Video LLMscited by 55
  19. 2025Video-Bench: Human-Aligned Video Generation Benchmarkcited by 53
  20. 2023Mitigating Hallucination in Visual Language Models with Visual Supervisioncited by 49
  21. 2024Mitigating Object Hallucination via Concentric Causal Attentioncited by 48
  22. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Promptscited by 47
  23. 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projectioncited by 46
  24. 2024PhD: A ChatGPT-Prompted Visual hallucination Evaluation Datasetcited by 44
  25. 2025VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Modelcited by 44
  26. 2025Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Modelscited by 44
  27. 2024Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptionscited by 43
  28. 2025Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decodingcited by 42
  29. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understandingcited by 37
  30. 2025PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Trainingcited by 37
  31. 2024Cantor: Inspiring Multimodal Chain-of-Thought of MLLMcited by 34
  32. 2024Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimizationcited by 34
  33. 2024Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMscited by 33
  34. 2025Treble Counterfactual VLMs: A Causal Approach to Hallucinationcited by 33
  35. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resamplingcited by 32
  36. 2023Griffon: Spelling out All Object Locations at Any Granularity with Large Language Modelscited by 31
  37. 2025CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMscited by 31
  38. 2024FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedbackcited by 30

50 most cited of 223 All 223 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024NoiseBoost: Alleviating Hallucination with Noise Perturbation for Multimodal Large Language Models
  2. 2024EFUF: Efficient Fine-grained Unlearning Framework for Mitigating Hallucinations in Multimodal Large Language Models
  3. 2025Mitigating Behavioral Hallucination in Multimodal Large Language Models for Sequential Images
  4. 2024Hallu-PI: Evaluating Hallucination in Multi-modal Large Language Models within Perturbed Inputs
  5. 2024Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluations
  6. 2024Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding
  7. 2025ResNetVLLM-2: Addressing ResNetVLLM's Multi-Modal Hallucinations
  8. 2025When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding