mlsanityData to 7 Oct 2026

Evaluating Object Hallucination in Large Vision-Language Models

arXiv 2305.10355 ↗First posted 17 May 2023cs.CV cs.CL cs.MMcited by 1,696

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Inspired by the superior language abilities of large language models (LLM), large vision-language models (LVLM) have been recently explored by integrating powerful LLMs for improving the performance on complex multimodal tasks. Despite the promising progress on LVLMs, we find that LVLMs suffer from the hallucination problem, i.e. they tend to generate objects that are inconsistent with the target images in the descriptions. To investigate it, this work presents the first systematic study on object hallucination of LVLMs. We conduct the evaluation experiments on several representative LVLMs, and show that they mostly suffer from severe object hallucination issue. We further discuss that the visual instructions may influence the hallucination, and find that: objects that frequently occur in the visual instructions or co-occur with the image objects, are obviously prone to be hallucinated by LVLMs. Besides, we find that existing evaluation methods might be affected by the input instructions and generation styles of LVLMs. Thus, we further design an improved evaluation method for object hallucination by proposing a polling-based query method called POPE. Experiment results demonstrate that our POPE can evaluate the object hallucination in a more stable and flexible way. Our codes and data are publicly available at https://github.com/RUCAIBox/POPE.

Key facts. Cited by 1,696 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 55 references that are in the map. First posted on arXiv 17 May 2023.

Builds on

39 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training14 Oct

Other references · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
26 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  5. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  7. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  8. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  9. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  10. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  11. 2018Object Hallucination in Image Captioningcited by 661
  12. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  13. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  14. 2023Segment Everything Everywhere All at Oncecited by 546
  15. 2018nocaps: novel object captioning at scalecited by 444
  16. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  17. 2023StructGPT: A General Framework for Large Language Model to Reason over Structured Datacited by 339
  18. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  19. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  20. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  21. 2015Yin and Yang: Balancing and Answering Binary Visual Questionscited by 109
  22. 2018Neural Baby Talkcited by 107
  23. 2021The Factual Inconsistency Problem in Abstractive Text Summarization: A Surveycited by 79
  24. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 74
  25. 2022LAVIS: A Library for Language-Vision Intelligencecited by 55
  26. 2023Language Models are Few-shot Learners for Prognostic Predictioncited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 100; 2024: 438; 2025: 605; 2026: 553.

1,696 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  7. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  8. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  9. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  10. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  11. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  12. 2023A Survey on Multimodal Large Language Modelscited by 898
38 more
  1. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  2. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  3. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  4. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  5. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  6. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  8. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  9. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  10. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  11. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  12. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  13. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  14. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  15. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  16. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  17. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  18. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  19. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  20. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  21. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  22. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  23. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  24. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  25. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  26. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  27. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
  28. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  29. 2023A Survey of Hallucination in Large Foundation Modelscited by 365
  30. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  31. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  32. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  33. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  34. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  35. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 329
  36. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  37. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  38. 2024What matters when building vision-language models?cited by 321

50 most cited of 1,696 All 1,696 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models
  2. 2024VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models
  3. 2024Multi-Object Hallucination in Vision-Language Models
  4. 2025Analyzing and Mitigating Object Hallucination: A Training Bias Perspective
  5. 2024Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluations
  6. 2024Hallu-PI: Evaluating Hallucination in Multi-modal Large Language Models within Perturbed Inputs
  7. 2024Investigating and Mitigating Object Hallucinations in Pretrained Vision-Language (CLIP) Models
  8. 2024Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)