mlsanityData to 7 Oct 2026

Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models

arXiv 2310.05338 ↗First posted 9 Oct 2023cs.CV cs.CLcited by 80

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Object hallucination poses a significant challenge in vision-language (VL) models, often leading to the generation of nonsensical or unfaithful responses with non-existent objects. However, the absence of a general measurement for evaluating object hallucination in VL models has hindered our understanding and ability to mitigate this issue. In this work, we present NOPE (Negative Object Presence Evaluation), a novel benchmark designed to assess object hallucination in VL models through visual question answering (VQA). We propose a cost-effective and scalable approach utilizing large language models to generate 29.5k synthetic negative pronoun (NegP) data of high quality for NOPE. We extensively investigate the performance of 10 state-of-the-art VL models in discerning the non-existence of objects in visual questions, where the ground truth answers are denoted as NegP (e.g., "none"). Additionally, we evaluate their standard performance on visual questions on 9 other VQA datasets. Through our experiments, we demonstrate that no VL model is immune to the vulnerability of object hallucination, as all models achieve accuracy below 10% on NegP. Furthermore, we uncover that lexically diverse visual questions, question types with large scopes, and scene-relevant objects capitalize the risk of object hallucination in VL models.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 71 of its 98 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

71 of 98 references.

References · 71

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2020Learning to summarize from human feedbackcited by 2,562
38 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  6. 2019Towards VQA Models That Can Readcited by 1,696
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  9. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  10. 2018Hierarchical Neural Story Generationcited by 1,182
  11. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  12. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  13. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  14. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  15. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  16. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  17. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  18. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  19. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  20. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  21. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  22. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  23. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  24. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  25. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  26. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  27. 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 202
  28. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  29. 2015Exploring Models and Data for Image Question Answeringcited by 139
  30. 2020Evaluating the Evaluation of Diversity in Natural Language Generationcited by 127
  31. 2016Analyzing the Behavior of Visual Question Answering Modelscited by 119
  32. 2019Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editingcited by 108
  33. 2019Cycle-Consistency for Robust Visual Question Answeringcited by 105
  34. 2021SPICE: Semantic Pseudo-labeling for Image Clusteringcited by 104
  35. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  36. 2020TAP: Text-Aware Pre-training for Text-VQA and Text-Captioncited by 95
  37. 2017An Analysis of Visual Question Answering Algorithmscited by 90
  38. 2021Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Modelscited by 74

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 24; 2025: 32; 2026: 19.

80 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  3. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  4. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 223
  5. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74
  6. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steeringcited by 56
  7. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 50
  8. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 48
  9. 2024Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidancecited by 44
  10. 2024PhD: A ChatGPT-Prompted Visual hallucination Evaluation Datasetcited by 44
  11. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Modelscited by 42
  12. 2025ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Modelscited by 40
38 more
  1. 2025ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLMcited by 36
  2. 2024Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimizationcited by 34
  3. 2024FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedbackcited by 30
  4. 2024Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Modelscited by 30
  5. 2024HaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoningcited by 30
  6. 2023What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuningcited by 28
  7. 2024Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Modelscited by 27
  8. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  9. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  10. 2025Are Multimodal Large Language Models Ready for Omnidirectional Spatial Reasoning?cited by 22
  11. 2025Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigationcited by 20
  12. 2025VASparse: Towards Efficient Visual Hallucination Mitigation via Visual-Aware Token Sparsificationcited by 17
  13. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  14. 2024TLDR: Token-Level Detective Reward Model for Large Vision Language Modelscited by 15
  15. 2024MetaToken: Detecting Hallucination in Image Descriptions by Meta Classificationcited by 13
  16. 2024Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awarenesscited by 13
  17. 2024ESREAL: Exploiting Semantic Reconstruction to Mitigate Hallucinations in Vision-Language Modelscited by 12
  18. 2025Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decodingcited by 12
  19. 2024FactCheXcker: Mitigating Measurement Hallucinations in Chest X-ray Report Generation Modelscited by 11
  20. 2025Through the Magnifying Glass: Adaptive Perception Magnification for Hallucination-Free VLM Decodingcited by 10
  21. 2025Attention Hijackers: Detect and Disentangle Attention Hijacking in LVLMs for Hallucination Mitigationcited by 9
  22. 2025PAINT: Paying Attention to INformed Tokens to Mitigate Hallucination in Large Vision-Language Modelcited by 8
  23. 2025Cross-Image Contrastive Decoding: Precise, Lossless Suppression of Language Priors in Large Vision-Language Modelscited by 8
  24. 2024LongHalQA: Long-Context Hallucination Evaluation for MultiModal Large Language Modelscited by 7
  25. 2025Evaluating Hallucination in Large Vision-Language Models based on Context-Aware Object Similaritiescited by 7
  26. 2025The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?cited by 7
  27. 2025Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Modelscited by 7
  28. 2025SpurLens: Automatic Detection of Spurious Cues in Multimodal LLMscited by 6
  29. 2025Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formatscited by 6
  30. 2024Prompting Medical Large Vision-Language Models to Diagnose Pathologies by Visual Question Answeringcited by 5
  31. 2024Trust but Verify: Programmatic VLM Evaluation in the Wildcited by 5
  32. 2024Who Brings the Frisbee: Probing Hidden Hallucination Factors in Large Vision-Language Model via Causality Analysiscited by 4
  33. 2025Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Modelscited by 4
  34. 2025GHOST: Hallucination-Inducing Image Generation for Multimodal LLMscited by 4
  35. 2026FREAK: A Fine-grained Hallucination Evaluation Benchmark for Advanced MLLMscited by 4
  36. 2026Can Humans Dream of Electric Sheep? Human-Written Samples for Fine-Grained Vision-and-Language Hallucination Benchmarkingcited by 4
  37. 2024FIHA: Autonomous Hallucination Evaluation in Vision-Language Models with Davidson Scene Graphscited by 3
  38. 2025HKD4VLM: A Progressive Hybrid Knowledge Distillation Framework for Robust Multimodal Hallucination and Factuality Detection in VLMscited by 3

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models?
  2. 2024Visual Hallucination: Definition, Quantification, and Prescriptive Remediations
  3. 2024Towards a Systematic Evaluation of Hallucinations in Large-Vision Language Models
  4. 2023Evaluating Object Hallucination in Large Vision-Language Models
  5. 2024TUBench: Benchmarking Large Vision-Language Models on Trustworthiness with Unanswerable Questions
  6. 2025Energy-Guided Decoding for Object Hallucination Mitigation
  7. 2024THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Models
  8. 2026TruthLens: Object Hallucination Detection via Self-Evaluating Truthfulness Scores in LVLMs