mlsanityData to 7 Oct 2026

Let there be a clock on the beach: Reducing Object Hallucination in Image Captioning

arXiv 2110.01705 ↗First posted 4 Oct 2021cs.CVcited by 74

Not on the map. Why?

Cited by 55 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Explaining an image with missing or non-existent objects is known as object bias (hallucination) in image captioning. This behaviour is quite common in the state-of-the-art captioning models which is not desirable by humans. To decrease the object hallucination in captioning, we propose three simple yet efficient training augmentation method for sentences which requires no new training data or increase in the model size. By extensive analysis, we show that the proposed methods can significantly diminish our models' object bias on hallucination metrics. Moreover, we experimentally demonstrate that our methods decrease the dependency on the visual features. All of our code, configuration files and model weights will be made public.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Builds on 33 of its 48 references that are in the map. First posted on arXiv 4 Oct 2021.

Builds on

33 of 48 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  9. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  10. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  11. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  12. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
21 more
  1. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  2. 2018Object Hallucination in Image Captioningcited by 661
  3. 2018Recognition in Terra Incognitacited by 655
  4. 2016Self-critical Sequence Training for Image Captioningcited by 544
  5. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  6. 2016FastText.zip: Compressing text classification modelscited by 331
  7. 2019Attention on Attention for Image Captioningcited by 289
  8. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  9. 2019Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editingcited by 108
  10. 2018Neural Baby Talkcited by 107
  11. 2020Visual Commonsense R-CNNcited by 106
  12. 2020Say As You Wish: Fine-grained Control of Image Caption Generation with Abstract Scene Graphscited by 87
  13. 2019Good News, Everyone! Context driven entity-aware captioning for news imagescited by 66
  14. 2018Show, Control and Tell: A Framework for Generating Controllable and Grounded Captionscited by 65
  15. 2020Deconfounded Image Captioning: A Causal Retrospectcited by 65
  16. 2020Transform and Tell: Entity-Aware News Image Captioningcited by 52
  17. 2020Comprehensive Image Captioning via Scene Graph Decompositioncited by 52
  18. 2016Captioning Images with Diverse Objectscited by 27
  19. 2020Towards Unique and Informative Captioning of Imagescited by 25
  20. 2021Towards Accurate Text-based Image Captioning with Content Diversity Explorationcited by 25
  21. 2017Speaking the Same Language: Matching Machine to Human Captions by Adversarial Trainingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 18; 2024: 18; 2025: 22; 2026: 14.

74 papers.

Sort:

By area:

  1. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  3. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  5. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  6. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 171
  7. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Modelscited by 99
  8. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revisioncited by 96
  9. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  10. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 83
  11. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  12. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74
38 more
  1. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 71
  2. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewritescited by 61
  3. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 48
  4. 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projectioncited by 46
  5. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 44
  6. 2024Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidancecited by 44
  7. 2025ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Modelscited by 40
  8. 2024AI Hallucinations: A Misnomer Worth Clarifyingcited by 34
  9. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resamplingcited by 32
  10. 2023HallE-Control: Controlling Object Hallucination in Large Multimodal Modelscited by 31
  11. 2025CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuningcited by 31
  12. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learningcited by 30
  13. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decodingcited by 26
  14. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 22
  15. 2025MASH-VLM: Mitigating Action-Scene Hallucination in Video-LLMs through Disentangled Spatial-Temporal Representationscited by 22
  16. 2023Zero-Resource Hallucination Prevention for Large Language Modelscited by 19
  17. 2023Mitigating Open-Vocabulary Caption Hallucinationscited by 18
  18. 2024Towards Rationality in Language and Multimodal Agents: A Surveycited by 15
  19. 2024Investigating and Mitigating Object Hallucinations in Pretrained Vision-Language (CLIP) Modelscited by 15
  20. 2025Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decodingcited by 12
  21. 2023Simple Token-Level Confidence Improves Caption Correctnesscited by 11
  22. 2024Visual Hallucination: Definition, Quantification, and Prescriptive Remediationscited by 11
  23. 2024DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioningcited by 11
  24. 2025Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMscited by 9
  25. 2025Mitigating Hallucinations in Large Vision-Language Models by Adaptively Constraining Information Flowcited by 9
  26. 2025Attention Hijackers: Detect and Disentangle Attention Hijacking in LVLMs for Hallucination Mitigationcited by 9
  27. 2025HalLoc: Token-level Localization of Hallucinations for Vision Language Modelscited by 9
  28. 2024Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models?cited by 8
  29. 2023Hallucination Detection for Grounded Instruction Generationcited by 7
  30. 2025SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defensecited by 7
  31. 2023Insights into Classifying and Mitigating LLMs' Hallucinationscited by 6
  32. 2025HalDec-Bench: Benchmarking Hallucination Detector in Image Captioningcited by 5
  33. 2024Who Brings the Frisbee: Probing Hidden Hallucination Factors in Large Vision-Language Model via Causality Analysiscited by 4
  34. 2025HalluRNN: Mitigating Hallucinations via Recurrent Cross-Layer Reasoning in Large Vision-Language Modelscited by 4
  35. 2025Identify, Isolate, and Purge: Mitigating Hallucinations in LVLMs via Self-Evolving Distillationcited by 4
  36. 2026Tone Matters: The Impact of Linguistic Tone on Hallucination in VLMscited by 4
  37. 2025Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddingscited by 3
  38. 2026CRoPS: A Training-Free Hallucination Mitigation Framework for Vision-Language Modelscited by 3

50 most cited of 74 All 74 (JSON)