mlsanityData to 7 Oct 2026

Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training

arXiv 2210.07688 ↗First posted 14 Oct 2022cs.CL cs.CVcited by 71

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large-scale vision-language pre-trained (VLP) models are prone to hallucinate non-existent visual objects when generating text based on visual information. In this paper, we systematically study the object hallucination problem from three aspects. First, we examine recent state-of-the-art VLP models, showing that they still hallucinate frequently, and models achieving better scores on standard metrics (e.g., CIDEr) could be more unfaithful. Second, we investigate how different types of image encoding in VLP influence hallucination, including region-based, grid-based, and patch-based. Surprisingly, we find that patch-based features perform the best and smaller patch resolution yields a non-trivial reduction in object hallucination. Third, we decouple various VLP objectives and demonstrate that token-level image-text alignment and controlled generation are crucial to reducing hallucination. Based on that, we propose a simple yet effective VLP loss named ObjMLM to further mitigate object hallucination. Results show that it reduces object hallucination by up to 17.4% when tested on two benchmarks (COCO Caption for in-domain and NoCaps for out-of-domain evaluation).

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 62 of its 75 references that are in the map. First posted on arXiv 14 Oct 2022.

Builds on

62 of 75 references.

References · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
38 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  2. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  7. 2019Contrastive Multiview Codingcited by 1,297
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  10. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  11. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  12. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  13. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  14. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  15. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  16. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  17. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  18. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  19. 2018Object Hallucination in Image Captioningcited by 661
  20. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  21. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  22. 2020Contrastive Learning with Hard Negative Samplescited by 619
  23. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  24. 2016Self-critical Sequence Training for Image Captioningcited by 544
  25. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  26. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  27. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  28. 2018nocaps: novel object captioning at scalecited by 444
  29. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  30. 2020Hard Negative Mixing for Contrastive Learningcited by 399
  31. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  32. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  33. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  34. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  35. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  36. 2018Neural Baby Talkcited by 107
  37. 2021Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Groundingcited by 106
  38. 2021M6: A Chinese Multimodal Pretrainercited by 91

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 28; 2025: 14; 2026: 11.

71 papers.

Sort:

By area:

  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  2. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  3. 2023A Survey of Hallucination in Large Foundation Modelscited by 365
  4. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  5. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  6. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 171
  7. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  8. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 99
  9. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 98
  10. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  11. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewritescited by 61
  12. 2024Multi-Object Hallucination in Vision-Language Modelscited by 60
38 more
  1. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 56
  2. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  3. 2023Pro-Cap: Leveraging a Frozen Vision-Language Model for Hateful Meme Detectioncited by 45
  4. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 44
  5. 2024Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidancecited by 44
  6. 2024ALOHa: A New Measure for Hallucination in Captioning Modelscited by 44
  7. 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Modelscited by 44
  8. 2023Visual Instruction Tuning with Polite Flamingocited by 38
  9. 20243D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucinationcited by 36
  10. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  11. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learningcited by 30
  12. 2024Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Modelscited by 30
  13. 2024HaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoningcited by 30
  14. 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoningcited by 29
  15. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  16. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  17. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 22
  18. 2025Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigationcited by 20
  19. 2023Mitigating Open-Vocabulary Caption Hallucinationscited by 18
  20. 2024Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Artcited by 17
  21. 2024An Audit on the Perspectives and Challenges of Hallucinations in NLPcited by 17
  22. 2025VASparse: Towards Efficient Visual Hallucination Mitigation via Visual-Aware Token Sparsificationcited by 17
  23. 2025LanP: Rethinking the Impact of Language Priors in Large Vision-Language Modelscited by 14
  24. 2024MetaToken: Detecting Hallucination in Image Descriptions by Meta Classificationcited by 13
  25. 2024Probing Multimodal Large Language Models for Global and Local Semantic Representationscited by 12
  26. 2023Simple Token-Level Confidence Improves Caption Correctnesscited by 11
  27. 2024Visual Hallucination: Definition, Quantification, and Prescriptive Remediationscited by 11
  28. 2023Beyond Generic: Enhancing Image Captioning with Real-World Knowledge using Vision-Language Pre-Training Modelcited by 10
  29. 2023Food-500 Cap: A Fine-Grained Food Caption Benchmark for Evaluating Vision-Language Modelscited by 10
  30. 2024Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models?cited by 8
  31. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  32. 2023Hallucination Detection for Grounded Instruction Generationcited by 7
  33. 2025Evaluating Hallucination in Large Vision-Language Models based on Context-Aware Object Similaritiescited by 7
  34. 2023Open Visual Knowledge Extraction via Relation-Oriented Multimodality Model Promptingcited by 5
  35. 2024ChatGPT Incorrectness Detection in Software Reviewscited by 5
  36. 2023Parameter Efficient Audio Captioning With Faithful Guidance Using Audio-text Shared Latent Representationcited by 4
  37. 2024Transparent and Coherent Procedural Mistake Detectioncited by 4
  38. 2025Retrieval Visual Contrastive Decoding to Mitigate Object Hallucinations in Large Vision-Language Modelscited by 3

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens
  2. 2026NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors
  3. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
  4. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  5. 2025Analyzing and Mitigating Object Hallucination: A Training Bias Perspective
  6. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models
  7. 2024HELPD: Mitigating Hallucination of LVLMs by Hierarchical Feedback Learning with Vision-enhanced Penalty Decoding
  8. 2025SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding