mlsanityData to 7 Oct 2026

Object Hallucination in Image Captioning

arXiv 1809.02156 ↗First posted 6 Sep 2018cs.CL cs.CVcited by 661

Not on the map. Why?

Cited by 451 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite continuously improving performance, contemporary image captioning models are prone to "hallucinating" objects that are not actually in a scene. One problem is that standard metrics only measure similarity to ground truth captions and may not fully capture image relevance. In this work, we propose a new image relevance metric to evaluate current models with veridical visual labels and assess their rate of object hallucination. We analyze how captioning model architectures and learning objectives contribute to object hallucination, explore when hallucination is likely due to image misclassification or language priors, and assess how well current sentence metrics capture object hallucination. We investigate these questions on the standard image captioning benchmark, MSCOCO, using a diverse set of models. Our analysis yields several interesting findings, including that models which score best on standard sentence metrics do not always have lower hallucination and that models which hallucinate more tend to make errors driven by language priors.

Key facts. Cited by 661 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Sep 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 27; 2023: 72; 2024: 122; 2025: 192; 2026: 223.

661 papers.

Sort:

By area:

  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  3. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  5. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  6. 2023A Survey on Multimodal Large Language Modelscited by 898
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  9. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  10. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  11. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  12. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
38 more
  1. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  2. 2023A Survey of Hallucination in Large Foundation Modelscited by 365
  3. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  4. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  5. 2023How Language Model Hallucinations Can Snowballcited by 310
  6. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  7. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 293
  8. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  9. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  10. 2023Challenges and Applications of Large Language Modelscited by 251
  11. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  12. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  13. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 223
  14. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  15. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  16. 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 196
  17. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 196
  18. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  19. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  20. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 174
  21. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 171
  22. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  23. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  24. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 139
  25. 2021A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generationcited by 137
  26. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
  27. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  28. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  29. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 127
  30. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Datacited by 124
  31. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  32. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  33. 2023Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answeringcited by 119
  34. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  35. 2023Calibrated Language Models Must Hallucinatecited by 114
  36. 2021Causal Attention for Vision-Language Taskscited by 111
  37. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  38. 2021Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Groundingcited by 106

50 most cited of 661 All 661 (JSON)