mlsanityData to 14 Oct 2026

Analyzing and Mitigating Object Hallucination in Large Vision-Language Models

arXiv 2310.00754 ↗First posted 1 Oct 2023cs.LG cs.CL cs.CVcited by 316

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large vision-language models (LVLMs) have shown remarkable abilities in understanding visual information with human languages. However, LVLMs still suffer from object hallucination, which is the problem of generating descriptions that include objects that do not actually exist in the images. This can negatively impact many vision-language tasks, such as visual summarization and reasoning. To address this issue, we propose a simple yet powerful algorithm, LVLM Hallucination Revisor (LURE), to post-hoc rectify object hallucination in LVLMs by reconstructing less hallucinatory descriptions. LURE is grounded in a rigorous statistical analysis of the key factors underlying object hallucination, including co-occurrence (the frequent appearance of certain objects alongside others in images), uncertainty (objects with higher uncertainty during LVLM decoding), and object position (hallucination often appears in the later part of the generated text). LURE can also be seamlessly integrated with any LVLMs. We evaluate LURE on six open-source LVLMs, achieving a 23% improvement in general object hallucination evaluation metrics over the previous best approach. In both GPT and human evaluations, LURE consistently ranks at the top. Our data and code are available at https://github.com/YiyangZhou/LURE.

Key facts. Cited by 316 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 57 references that are in the map. First posted on arXiv 1 Oct 2023.

Builds on

40 of 57 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Models29 Aug
  2. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  3. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,916
25 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  2. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  5. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  6. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  7. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  8. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  9. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  10. 2018Object Hallucination in Image Captioningcited by 669
  11. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  12. 2019Unlabeled Data Improves Adversarial Robustnesscited by 416
  13. 2023How Language Model Hallucinations Can Snowballcited by 311
  14. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  15. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  16. 2017Beam Search Strategies for Neural Machine Translationcited by 232
  17. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  18. 2023ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Modelscited by 116
  19. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 76
  20. 2021When and How Mixup Improves Calibrationcited by 60
  21. 2023Exposing and Mitigating Spurious Correlations for Cross-Modal Retrievalcited by 36
  22. 2023LLM as A Robotic Brain: Unifying Egocentric Memory and Controlcited by 33
  23. 2023Advancing Medical Imaging with Language Models: A Journey from N-grams to ChatGPTcited by 28
  24. 2023LLM-Based Human-Robot Collaboration Framework for Manipulation Taskscited by 17
  25. 2015Bridging AIC and BIC: a new criterion for autoregressioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 112; 2025: 116; 2026: 76.

316 papers.

Sort:

By area:

  1. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  2. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  3. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  4. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 410
  5. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  6. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  7. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
  8. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 240
  9. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 225
  10. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 208
  11. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  12. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
38 more
  1. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 189
  2. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 179
  3. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 174
  4. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 170
  5. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 143
  6. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 137
  7. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  8. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 131
  9. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Datacited by 126
  10. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  11. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 108
  12. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinationscited by 104
  13. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 103
  14. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 94
  15. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 88
  16. 2025MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigmcited by 87
  17. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 85
  18. 2024Calibrated Self-Rewarding Vision Language Modelscited by 78
  19. 2023Multimodal Representation Learning by Alternating Unimodal Adaptationcited by 75
  20. 2024Unified Hallucination Detection for Multimodal Large Language Modelscited by 75
  21. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvementcited by 74
  22. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74
  23. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  24. 2025MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understandingcited by 67
  25. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 66
  26. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steeringcited by 66
  27. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 65
  28. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
  29. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steeringcited by 59
  30. 2024Detecting and Evaluating Medical Hallucinations in Large Vision Language Modelscited by 57
  31. 2025Video-Bench: Human-Aligned Video Generation Benchmarkcited by 57
  32. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  33. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 53
  34. 2024Mitigating Object Hallucination via Concentric Causal Attentioncited by 49
  35. 2024Position: What Can Large Language Models Tell Us about Time Series Analysiscited by 47
  36. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Promptscited by 47
  37. 2024Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Modelscited by 47
  38. 2024Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidancecited by 46

50 most cited of 316 All 316 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Analyzing and Mitigating Object Hallucination: A Training Bias Perspective
  2. 2025A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models
  3. 2024Multi-Object Hallucination in Vision-Language Models
  4. 2024VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models
  5. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training
  6. 2025What Makes "Good" Distractors for Object Hallucination Evaluation in Large Vision-Language Models?
  7. 2026Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
  8. 2025Watch Closely: Mitigating Object Hallucinations in Large Vision-Language Models with Disentangled Decoding